EVALUATION Mid-Term Performance Evaluation of the USAID/SAUTI Project [April 2019] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the USAID Data for Development Program. USAID/TANZANIA MID-TERM PERFORMANCE EVALUATION OF THE USAID/SAUTI PROJECT Submitted: March 29, 2019 Prepared by: Terence Beney, Team Lead Jake Laden, Evaluation Advisor, NORC Daud Siwalaze, Monitoring and Evaluation Specialist, NORC Nasson Konga, Monitoring and Evaluation Specialist, NORC Dominic Mosha, SME, STTA Bahati Tenga, Researcher and Data Collector, STTA Norah Kaaya, Researcher and Data Collector, STTA Submitted by: Brenda L. Pearson, Chief of Party ME&A USAID/Tanzania Data for Development Program Plot 122 Mwai Kibaki Road, Mikocheni A, Dar es Salaam, Tanzania Email: Bpearson@engl.com Cover Photo: Sauti Peer Educators Credit: Sauti Evaluation Team, Data for Development DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government CONTENTS EXECUTIVE SUMMARY............................................................................................................................II Evaluation Purpose and Evaluation Questions............................................................................................. ii Project Background............................................................................................................................................ ii Evaluation Questions, Design, Methods, and Limitations......................................................................... ii Evaluation Design and Methods...................................................................................................................... ii Key Findings and Conclusions......................................................................................................................... ii 1.0 EVALUATION PURPOSE & QUESTIONS ................................................................................. 2 1.1 Evaluation Purpose .............................................................................................................................. 2 1.2 Evaluation Questions.......................................................................................................................... 2 2.0 PROJECT BACKGROUND ........................................................................................................... 1 2.1 Overview ............................................................................................................................................... 1 2.2 Goal, Objectives, and Outcomes..................................................................................................... 1 2.3 Services, Indicators and Geographical Coverage......................................................................... 2 3.0 EVALUATION METHODS & LIMITATIONS............................................................................. 4 3.1 Data Collection Methods................................................................................................................... 4 3.2 Data Analysis Methods....................................................................................................................... 9 3.3 Methodological Limitations............................................................................................................... 9 4.0 FINDINGS & CONCLUSIONS ................................................................................................... 10 4.1 EQ1. How effective is the project in achieving its goals, objectives and performance targets?................................................................................................................................................. 10 4.2 EQ2. What factors explain the achievement or under-achievement of project results?.................................................................................................................................................. 16 4.3 EQ3. How effectively has USAID and its IPs prepared for and/or responded to constraints to implementation?...................................................................................................... 18 4.4 EQ4. What are the SAUTI project's prospects for sustainability? (design, plan, implement, monitor, finance).......................................................................................................... 22 4.5 EQ5. What critical lessons have been learned that USAID should consider if designing a follow-on award to the Sauti Project?..................................................................... 28 5.0 RECOMMENDATIONS.............................................................................................................. 29 ANNEXES.................................................................................................................................................. 31 Annex I: Evaluation Statement of Work ..................................................................................................... 32 Annex II: Bibliography of Documents Reviewed....................................................................................... 51 Annex III: KII and FGD Discussion Guides................................................................................................. 53 Annex IV: List of KII Participants.................................................................................................................. 90 Annex V: Disclosure of any Conflicts of Interest..................................................................................... 94 ACRONYMS Acronym Description ADS Automated Directives System AGWY Adolescent Girls and Young Women AIDS Acquired Immune Deficiency Syndrome ART Antiretroviral Therapy CDC Centers for Disease Control and Prevention CDCS Country Development Cooperation Strategy CDO Community Development Officer CHAC Council HIV and AIDS Coordinator CHMT Council Health Management Team CHSSP Community Health and Social Welfare Systems Strengthening Program CHW Community Health Workers CMAC Council Multi-sectoral AIDS Committee CSO Civil Society Organization CTC Care and Treatment Clinic D4D Data for Development DACC District AIDS Control Coordinator DATIM Data for Accountability, Transparency and Impact Monitoring DEC Development Experience Clearinghouse DHO District Health Officer DMO District Medical Officer DMT District Master Trainer DO Development Objective DREAMS Determined, Resilient, Empowered, AIDS-free, Mentored and Safe EQ Evaluation Question ET Evaluation Team FGD Focus Group Discussions FP Family Planning FSW Female Sex Worker FY Fiscal Year GBV Gender Based Violence GI Group Interview Acronym Description GoT Government of Tanzania HIV Human Immunodeficiency Virus HIVST HIV Self-Testing HTC HIV Testing and Counseling Services HTS JIV Testing Services IP Implementing Partner IR Intermediate Result IRB Institutional Review Board ISW Institute of Social Work JSI John Snow, Inc. KII Key Informant Interviews KVPs Key and Vulnerable Populations LGA Local Government Authority LOE Level of Effort LOP Life of Program MSM Men Who Have Sex with Men MC Municipal Council M&E Monitoring and Evaluation MIS Management Information System MOHCDGEC Ministry of Health, Community Development, Gender, Elderly and Children NACOPHA National Council of People Living with HIV NASW National Association of Social Workers NACP National AIDS Control Programme NIMR National Institute for Medical Research OCA Organizational Capacity Assessment OHSP Other Hotspot Populations OVC Orphans and Vulnerable Children PD Program Description PEPFAR President’s Emergency Plan for AIDS Relief PLHIV People Living with HIV PO-RALG President’s Office for Regional Administration and Local Government PPR Project Performance Review Acronym Description PrEP Pre-Exposure Prophylaxis Q Quarter RHMT Regional Health Management Team RMO Regional Medical Officer SOW Statement of Work STTA Short-Term Technical Assistance SWO Social Welfare Officer TACAIDS Tanzania Commission for AIDS TC Town Council THMSI Tanzania HIV/AIDS and Malaria Indicator Survey USAID U.S. Agency for International Development vAGYW Vulnerable Adolescent Girls and Young Women VAWC Violence against Women and Children Committee ii EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of the Mid-term Performance Evaluation of the Sauti Program is to: 1) assess the effectiveness, efficiency, and quality of the project by reviewing how well it is achieving its goals, objectives, and performance targets, and identifying the factors enabling or constraining performance; 2) make recommendations to improve performance for the remaining implementation period; and 3) document lessons learned that might inform the United States Agency for International Development Tanzania’s (USAID/Tanzania) future key and priority populations human immunodeficiency virus (HIV) programming. PROJECT BACKGROUND The Sauti Program is a five-year, $104.2 million cooperative agreement (Cooperative Agreement AID￾621-A-15-00003), funded by USAID and focused on HIV prevention. It contributes to the U.S. President’s Plan for AIDS Relief (PEPFAR) in Tanzania by providing key and vulnerable populations with access to a variety of service packages that include biomedical and family planning (FP) services, as well as behavioral and structural interventions. Jhpiego Corporation implements the Sauti Program with its partners, EngenderHealth, Inc., Pact, Inc., and the National Institute for Medical Research (NIMR) Mwanza, in support of the Government of the United Republic of Tanzania’s commitment to HIV prevention. The Sauti Program’s main counterpart from the Government of Tanzania (GoT) is the Ministry of Health, Community Development, Gender, Elderly, and Children (MOHCDGEC). Sauti targets the following key and vulnerable populations (KVPs): • Female Sex Workers (FSWs) • Men Who Have Sex with Men (MSM) • Sex-Worker Men Who Have Sex with Men (SW-MSM) • Other Hotspot Populations (OHSP) • Partners of Female Sex Workers (PSFW) • Vulnerable Adolescent Girls and Young Women (vAGYW) [15 – 24yrs] • Pediatric Key and Vulnerable Populations (Ped-KVP) [18 months – 14yrs] • Pediatric non-Key and Vulnerable Populations [(Ped-nonKVP),18 months – 14yrs] Sauti has the following objectives: • Objective 1: Implement a package of core and expanded biomedical HIV prevention and FP interventions, with enhanced linkages to care, treatment, and support services. • Objective 2: Deploy interventions designed to reduce individual risk behaviors and strengthen support for positive social norms and structures at the community level. • Objective 3: Execute a robust research and learning agenda. • Objective 4: Develop and implement capacity and sustainability building interventions. • Objective 5: Build and deploy vigorous monitoring and evaluation (M&E) systems. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS The evaluation is centered on the following evaluation questions (EQs): • EQ1. How effective is the project in achieving its goals, objectives, and performance targets? • EQ2. What factors explain the achievement or under-achievement of project results? • EQ3. How effectively has USAID and its implementing partners (IPs) prepared for and/or responded to constraints to implementation? • EQ4. What are the Sauti project's prospects for sustainability (design, plan, implement, monitor, finance)? ii • EQ5. What critical lessons have been learned that USAID should consider if designing a follow￾on award to the Sauti Project? EVALUATION DESIGN AND METHODS To address the EQs, the evaluation team (ET) employed a mixed-methods approach, which analyzed project-related documentation, project monitoring data, and data collected through key informant interviews (KIIs), group interviews (GIs), and focus group discussions (FGDs) with various stakeholders. These multiple sources of data allowed the ET to triangulate information ensuring sound findings, conclusions, and recommendations. Primary and secondary data was disaggregated by appropriate demographics and target populations and included age, gender/sex, and KVP type, as possible. Key Components of the Evaluation Design: • Mixed-methods design, integrating a thorough analysis of the context by combining secondary performance data, project documentation, Data for Accountability, Transparency, and Impact Monitoring (DATIM),1 USAID’s Implementing Partner Reporting System (IPRS), and the Sauti Management Information System (MIS) with primary qualitative data generated from KIIs and FGDs during fieldwork. • Purposive sampling of limited sites that compare more comprehensive package of services with basic Sauti service package; Determined, Resilient, Empowered, AIDS-free, Mentored and Safe (DREAMS) Initiative2 vs. non-DREAMS activities. DREAMS is a project nested within Sauti’s interventions with adolescent girls and young women (AGYW). • The primary data, generated through the fieldwork serves to check implementation fidelity and performance at sampled sites, identify persistent challenges across sites, generate explanatory findings, and document progress towards results and unmeasured outcomes. Methods included focus groups and individual and group interviews at the district level with KVP beneficiaries, peer educators, civil society organizations (CSOs), and local government authorities (LGAs). Regional and national authorities were also interviewed. *See body of the report for details on methods, sampling and limitations encountered in the course of this study on page 3 below. KEY FINDINGS AND CONCLUSIONS EQ1: How effective is the project in achieving its goals, objectives and performance targets? 1. Sauti is effective in meeting its targets. In addition, the project is responsive to performance data and addresses challenging deficits in performance to meet targets over time. By virtue of sheer volume of numbers reached, it is fair to state that access to services for key populations has been enhanced through the project, and this has been corroborated by focus group and interview data. However, structural challenges in health information systems and the lack of a comprehensive national unique ID system make it difficult for the program to routinely track layering of services due to the possibility of duplicate ID codes or recording errors. For example, it makes it difficult for the project to rapidly adapt to layering-related service delivery issues, and to optimally learn which combinations of services are most effective, for whom, and under which circumstances. 1 DATIM refers to Data for Accountability, Transparency and Impact Monitoring, which is the data system used for USAID’s PEPFAR projects. 2 Sauti are one DREAMs implementer among others. These activities occur in some but not all Sauti activity sites and represented half of those sampled for this study. iii 2. Results for beneficiaries include gains in health outcomes, health seeking behaviors including FP services, and, to a lesser extent, improved economic prospects. There is also evidence suggesting shifts in community and facility personnel attitudes towards key populations where Sauti is active. The latter should be qualified, however, by the possibility that the perceived attitudinal shifts may simply reflect the sense of empowerment that participation in Sauti interventions seems to cultivate in beneficiaries—FSW and DREAMS beneficiaries in particular. Complimentary services can also provide an incentive which may bring KVPs to HIV testing/screening services without a significant change in attitude or prioritization of those services. 3. Sauti has also contributed substantially to the development of local capacity—both in government and civil society—to utilize evidence and implement services. It has accomplished this through intensive technical support, but also through substantially increasing funding to CSOs, augmenting their reach and resources for service delivery. The latter has implications for the discussion on self-reliance and sustainability. EQ2: What factors explain the achievement or under-achievement of project results? 1. Data from focus groups and interviews points to a number of implementation modalities that may enhance project performance, particularly how access is facilitated. Focus group participants remarked on how an escorted referral, a Sauti referral form, or identification card gets one attention at a facility. Escorted referrals or a clear association with the project also appear to negate demands for non-mandated, prohibitive user fees, and alleviate the fear of stigma for new Sauti clients. KVPs reported that, overall, facilitated access strengthens their recognition of services as a right and their confidence to access those services. 2. Many respondents seemed to concur that the association with Sauti will not be required indefinitely, and that improved service provision at facilities represents a permanent change. Others felt the very active role fulfilled by Sauti in facilitating access is considered indispensable by some respondents, and they viewed the current ease of access to be at risk after the Sauti project ends. 3. Implementation modalities that provide Sauti with access to KVP networks also appear to be key elements influencing project performance. Focus group participants described their initial resistance to enrollment being eroded by the efforts of peer educators, in part because the latter enjoyed credibility as members of their groups. In addition, the initial encounters with Sauti at places KVPs frequent, at times convenient to them, were also named by respondents as factors that eased their willingness to access services. 4. The volunteer workforce and peer educators, particularly, are crucial to project performance. However, while peer educators consistently report being intrinsically motivated to work in the Sauti project, they do raise issues that could potentially disincentivize them. The most persistent of these is the inadequacy of the stipend they receive, especially in the light of the workload demanded of them. 5. KVP respondents explained that accepting offers of HIV services such as testing was facilitated by other services that were seen as higher priority and served as incentives for getting them to HIV screening or testing. For example, the provision of free condoms or free screening of sexually transmitted infections (STIs) aligned with the participants’ own priorities. The success of demand creation is not always matched by supply, however. Demand for condoms, pre-exposure prophylaxis (PrEP), self-testing kits and community-based Anti-retroviral therapy (ART) distribution is high, but shortage of commodities is persistently reported. 6. Systemic dependencies represent a set of constraints that pose substantial risk to project performance but are not controlled by the project’s actions, e.g., the politicization of key population and FP issues in Tanzania, discussed in EQ3 below. An example more relevant to iv project implementation is the underperformance against PrEP and HIV self-testing (HIVST) targets. The underachievement was not due to Sauti’s implementation failures, but to delayed roll out of HIVST and PrEP, and, in Quarter (Q)1 Fiscal Year (FY) 2019 (Oct-Dec 2018) a serious shortage of PrEP medications as the national supply was not ready yet to be included in their system. EQ3: How effectively have USAID and its IPs prepared for and/or responded to constraints to implementation? 1. Sauti’s outstanding feature is its intensive, data-driven adaptive management. This allows it to pursue targets effectively, and to be responsive to implementation challenges in real time. It also supplements the project’s ability to ensure quality and implementation fidelity through its sub-recipients. Quality and fidelity depend on training, equipping, and continuous quality improvement through supportive supervision. The project does provide these in a relentless pursuit of its targets. 2. Sustaining facilitated access after Sauti or independent of a follow-on will be problematic. Bringing community-based services to key vulnerable populations (KVPs), at locations they frequent, is an essential first step in cultivating facility access. 3. The demands of implementing PEPFAR-funded activities are substantial, driven by the urgency to achieve epidemic control. Sauti has effectively adapted to the demands imposed by geographic shifts in priorities, target increases, and the required extensions of service offerings. According to Sauti, USAID has been supportive in the project’s efforts to be responsive to PEPFAR’s imperatives and pivots. 4. Sauti has proven exceptionally adept at responding to the difficult operating environment created by the politicization of KVP issues; this includes adapting to a GoT facility focused strategy after the 2017 closure of services in KVP friendly drop-in centers. Similar events with regards to FP demonstrate that the political risks to services remain, but Sauti’s record of response suggests that the project is equipped to navigate the unpredictable environment. 5. Structural challenges such as the legal status of KVPs are a hurdle to effective program implementation. Fear of arrest and mistreatment makes beneficiaries unlikely to report crimes against them and keeps some KVPs from seeking treatment near police stations. This is of particular concern now that KVPs must procure services in centralized facilities rather than in drop-in centers located in their own communities. Sauti has adapted by providing training and capacity building to GoT facilities and staff for providing KVP-friendly services. They also use mobile strategies to reach KVP hotspots through peer-to-peer networks. EQ4: What are the Sauti project's prospects for sustainability (design, plan, implement, monitor, finance)? 1. Bearing in mind the prevalence of HIV nationally and the need for HIV prevention, treatment, and care as well as KVP-targeted services, there is a long road ahead to decreasing the prevalence of HIV across the country; sustaining services to those at greatest risk of contraction is critical. While Sauti has made significant gains among the key populations it has reached, Sauti’s results will not likely be sustained without continued intervention and donor support. 2. Following structural changes occurring in the first years of the program, Sauti was able to adapt course and find new solutions. Adaptive management on the part of the project increases prospects of sustainability amid ongoing risks. 3. CSOs confirm that the knowledge, skills, and modalities of service delivery cultivated through participation in Sauti are factors that strengthen prospects for sustaining services post-activity. However, financing the implementation of activities will be the key risk factor to address. v 4. Despite a challenging political climate, KVP-centered services are institutionalized in guidelines, job aids, and tools such as registers and forms. From the national to the LGA level and GoT facilities, there is a recognition that targeting KVPs is key to controlling the epidemic. 5. Practices for testing, treatment, and care are becoming more institutionalized and PEPFAR-supported activities are integrated into planning, all the way down to the local level. However, dependence on PEPFAR for human resources, technical assistance, and prevention and treatment supply remains a barrier. 6. Financial sustainability is lacking in service provision for KVPs. There is more work to be done to ensure government support for sustained HIV/AIDS services and for organizations providing these services to become financially capable of providing services without continued donor funding in the near and intermediate term. 7. The GoT is dependent on donor funds for continued service delivery, and routine operations such as supportive supervision and data management. There is still a major imbalance between what donors contribute and the proportion committed by the GoT. The strategy that appears to improve the actualization of the GoT’s commitment is decentralization of service provision (provided funding is disbursed from central government). 8. Operational challenges for GoT facilities and CSOs exist. Further resources are needed in facilities to ensure that human resources are on pace for KVP-friendly services. CSOs need financial support to provide the continued incentives for volunteers such as peer educators to continue their work. Stipends, transportation allowances, and other material support incentivize the efforts of volunteer cadres. Without these incentives, sustained implementation is at risk. 9. This evaluation concurs with PEPFAR’s 2018 assessment that Domestic Resource Mobilization and Technical and Allocative Efficiencies are currently unsustainable, meaning that Tanzania does not adequately generate the necessary financial resources for HIV and AIDS to ensure sufficient resource commitments and use data to strategically allocate funding and maximize investments (PEPFAR/Tanzania, April 2018). Bearing in mind other structural challenges, the picture for KVP￾specific interventions is even more vulnerable without ongoing international donor support. EQ5: What critical lessons have been learned that USAID should consider if designing a follow-on award to the Sauti project? 1. Sauti is able to perform against demanding targets at scale because it is designed to be an intensely data-driven, adaptive activity. The data-driven management model consists of the following elements: • A model of performance based on an assessment of the basic resources required to achieve a result. The project’s monthly planning is informed by these basic resources to result ratios. • Sauti devises annual performance plans in advance, based on PEPFAR-assigned targets, and informed by the resources to result ratios. • Daily targets are assigned in the way described and can therefore be tracked on a daily basis. The daily reporting, weekly consolidation, monthly planning, and issuing of guidance cycle to sub-recipients based on their performance against target keep the massive project on track. • Monitoring data is complemented by regular beneficiary engagements to learn what the indicator data cannot reveal about implementation. • Beneficiary engagements and monitoring data are complemented by substantive research projects that fulfill a learning agenda that cannot be satisfied without more substantive data. 2. Sauti is able to serve hard to reach populations because it implements through effective modalities which enable the project to access its target beneficiaries help them to access services with ease. vi • By mobilizing peer educators, Sauti can take advantage of their credibility with target populations and knowledge of key population networks to access their intended beneficiaries. • Escorted referrals by peer educators are an essential effective practice in the arsenal of active facilitated access that substantially eases linkages of key populations to services. • Sauti incentivizes participation by offering packages of services combining offerings that align with client priorities (e.g., STI screening), while incorporating PEPFAR priorities [e.g., HIV testing services (HTS)]. 3. The more comprehensive the package of services (such as in DREAMS districts) the more tacit benefits reported among beneficiaries. In addition to measured behavior change results, there is a substantial empowerment dividend that is more pronounced with more interventions. 4. Mobilizing a creative response to a deteriorating operating environment in which KVP and FP issues are being politicized resulted in adaptations that are proving more effective than previous practices. • SAUTI and partners have adapted to a changing context where formal facilities have taken the place of drop-in centers. Initially there was some lag in access to services due to this change but ultimately Sauti has been able to effectively work with GoT facilities by sensitizing them to KVP￾friendly treatment care practices. • Framing the case for KVP intervention in terms of epidemic control helps defuse resistance in an environment where KVP services are highly politicized. 2 1.0 EVALUATION PURPOSE & QUESTIONS 1.1 EVALUATION PURPOSE The purpose of the mid-term performance evaluation of the Sauti Project is threefold, namely to: 1) assess the effectiveness, efficiency and quality of the Project by reviewing how well it is achieving its goals, objectives, and performance targets, and identifying the factors enabling or constraining performance; 2) make recommendations to improve performance for the remaining implementation period; and 3) document lessons learned that might inform USAID/Tanzania’s future key and priority populations HIV programming. The primary audience for the evaluation is the USAID/Tanzania health team, Sauti Project staff and key national stakeholders, including the National AIDS Control Program (NACP), Tanzania Commission for AIDS (TACAIDS), and relevant personnel in Local Government Authorities (LAGs). 1.2 EVALUATION QUESTIONS The evaluation purpose is operationalized in a set of evaluation questions (EQs) that focus on Project performance, factors enabling or constraining performance, the responsiveness of USAID/Tanzania and implementing partners (IPs) to emerging performance constraints, prospects for sustainability and lessons learned that might inform adaptive management and future program design. Each EQ has been elaborated through several sub-questions that delineate the specific scope of the evaluation. The sub-questions under EQ 1 address each of Sauti’s Project objectives, which reflect the technical aspects of the Project’s combination prevention model, as well as the additional activity areas specified in the scope of its cooperative agreement. The questions are listed below. Project Performance EQ1. How effective is the Project in achieving its goals, objectives and performance targets? 1.1 To what extent has the Project succeeded in meeting performance expectations against PEPFAR targets? 1.2 To what extent has the Project improved access to and use of a core, quality package of services for KVPs? 1.3 To what extent has the Project succeeded in improving positive behaviors and social norms at individual and community levels? 1.4 To what extent has the Project strengthened capacity of local institutions and services 1.5 To what extent has the Project succeeded in executing a robust learning agenda? Factors Enabling or Constraining Performance EQ2. What factors explain the achievement or under-achievement of Project results? 2.1 What are the Projects strengths, weaknesses and gaps in design? 2.2 Has the Project been implemented with fidelity? 2.3 Have interventions been implemented to the required level of quality for achieving results? 2.4 How have any of the Projects capacity/operational (planning, management etc.) strengths, weaknesses and gaps influenced the achievement of results? 3 EQ3. How effectively has USAID and its IPs prepared for and/or responded to constraints to implementation? 3.1 Has the response to legal barriers constraining implementation among FSW and MSM been effectively dealt with? 3.2 What did USAID do that supported the Project effectively? 3.3 What could USAID do to better support the Project to improve its effectiveness? 3.4 What other key constraints have emerged, and have they been effectively dealt with? Prospects for Sustainability EQ4. What are the SAUTI project's prospects for sustainability? (design, plan, implement, monitor, finance) 4.1 What are the risks to the continued implementation of Project practices after Project close? 4.2 To what extent have Project practices been institutionalized by GoT and local organizations? 4.3 What are the funding related constraints to Project practices being sustained after Project close? 4.4 What can be done to improve the prospects for sustainability during the remaining activity period? Lessons Learned and Adaptive Management EQ5. What critical lessons have been learned that USAID should consider if designing a follow-on award to the Sauti Project? 5.1 What are the critical lessons learned in terms of design, implementation, capacity & operational issues, context-based constraints, sustainability and any other emerging factors that should inform the design of follow-on or similar activities? 5.2 How can critical lessons learned inform adaptive management and performance improvement of the current Project? 5.3 What adaptive management has already been done in light of lessons learned and are these adaptations proving effective? 1 2.0 PROJECT BACKGROUND 2.1 OVERVIEW The Sauti Project is a five-year, $104.2 million cooperative agreement (Cooperative Agreement AID-621- A-15-00003), funded by the U.S. Agency for International Development and focused on HIV prevention. It contributes to the US President’s Plan for AIDS Relief (PEPFAR) in Tanzania by providing key and vulnerable populations with access to a variety of service packages that include biomedical and family planning services, as well as behavioral and structural interventions. Jhpiego Corporation implements the Sauti Project with its partners, EngenderHealth, Inc., Pact, Inc., and the National Institute for Medical Research (NIMR) Mwanza, in support of the Government of the United Republic of Tanzania’s commitment to HIV prevention. The Sauti Project’s main counterpart from the Government of Tanzania (GoT) is the Ministry of Health, Community Development, Gender, Elderly and Children (MOHCDGEC). 2.2 GOAL, OBJECTIVES, AND OUTCOMES The goal of the Project is to “ensure that, within five years, all key and vulnerable populations (KVPs) in Project-supported communities are able to actively participate in a core package of vulnerability-tailored, high-quality, client- and community-centered combination prevention services”3. Key and Vulnerable Populations (KVPs) are those at higher risk of being infected or affected by HIV, and typically demonstrate elevated prevalence and higher incidence rates than the general population. Addressing their disproportionate vulnerability through effective programming is therefore vital to achieving epidemic control. Sauti targets the following categories of KVPs: • Female Sex Workers (FSW) • Men Who Have Sex with Men (MSM) • Sex-Worker Men Who Have Sex with Men (SW-MSM) • Other Hotspot Populations (OHSP) • Partners of Female Sex Workers (PSFW) • Vulnerable Adolescent Girls and Young Women (vAGYW) [15 – 24yrs] • Pediatric Key and Vulnerable Populations (Ped-KVP) [18 months – 14yrs] • Pediatric non-Key and Vulnerable Populations (Ped-nonKVP) [18 months – 14yrs] Sauti aims to realize five interrelated objectives: - Objective 1: Implement a package of core and expanded biomedical HIV prevention and family planning interventions, with enhanced linkages to care, treatment, and support services. - Objective 2: Deploy interventions designed to reduce individual risk behaviors and strengthen support for positive social norms and structures at the community level. - Objective 3: Execute a robust research and learning agenda. - Objective 4: Develop and implement capacity and sustainability building interventions. - Objective 5: Build and deploy vigorous monitoring and evaluation systems. 3 Sauti Project Quarterly Progress Report, FY18 Q3 2 As illustrated in the Project’s conceptual framework4, the intention is that realizing these objectives should increase the agency and social capital of KVPs and their communities, and lead to: 1. Increased and timely use of HIV prevention and treatment, as well as FP services; 2. Improved positive behaviors and social norms at the individual and community levels; 3. Reduced vulnerability of vAGYW through novel structural interventions; and 4. Increasingly sustainable comprehensive HIV prevention services for KVP. 2.3 SERVICES, INDICATORS AND GEOGRAPHICAL COVERAGE Outcomes for beneficiaries are pursued by delivering various packages of services, differentiated according to categories of KVPs, and typically including a selection of biomedical, family planning, behavioral and structural interventions. As a prevention focused PEPFAR Project, Sauti prioritizes the delivery of biomedical services. Performance is reported against a selection of biomedical and behavior change focused PEPFAR MER indicators (HTS, KP_Prev, PP_Prev), supplemented by a number of custom indicators that track other key interventions such as family planning, linkages to treatment, violence prevention, linkages to post-violence care and economic strengthening. Table 1: Sauti Services by KVP Category Intervention FSW MSM AGYW PFSW OHSP Peds of KVP 1. Biomedical Risk assessment and counseling X X X X X X HTS / Index testing X X X X X X HIV Self Testing X X - X - - FP counseling and services X - X - - X STI screening X X X X X X STI periodic presumptive treatment X X - - - - Condoms Promotion Provision X X X X X X TB screening X X X X X X GBV screening X X X X X X Alcohol and drug screening X X X X X X Escorted referral Care & Treatment Clinic, GBV services, RCHS X X X X X X Community ART to stable PLHIV X X X X X - Pre-Exposure Prophylaxis X - - - - - 2. Behavior Change Demand creation X X X X X X SBCC group education X - X - - - SBCC individual education X X - - - - 3. Economic Empowerment Saving and Lending and Parenting - - X - - - Cash transfer program - - X - - - 4. PLHIV and Alcohol support groups X X X X X 5. SASA! X X X X X X 4 See Appendix VI Additional Reference Material 3 In addition, Sauti is implementing the Determined, Resilient, Empowered, AIDS-free, Mentored and Safe (DREAMS) Initiative, Elton John AIDS Foundation (EJAF)-supported interventions for MSM and FSW, and several other collaborative operational research activities. Currently, the Sauti Program operates in 14 regions: Arusha, Dar es Salaam, Dodoma, Iringa, Kilimanjaro, Manyara, Mbeya, Songwe, Morogoro, Mtwara and Njombe, Shinyanga, Singida, Songwe, and Tabora, in accordance with the President’s Emergency Plan for AIDS Relief (PEPFAR) council-level prioritization. 4 3.0 EVALUATION METHODS & LIMITATIONS The information below describes the evaluation methodology, including data collection methods, sampling methods, and methodological limitations. Primary fieldwork/interviews for the evaluation took place from January 21-March 8, 2019. 3.1 DATA COLLECTION METHODS To address the EQs, the evaluation team (ET) employed a mixed-method approach, which analyzed Project related documentation, Project monitoring data, data collected through KIIs/GIs, and FGDs with various stakeholders. These multiple sources of data allowed the ET to triangulate information ensuring sound findings, conclusions and recommendations. The evaluation was designed to provide a detailed assessment of the programs various components, the effectiveness of their approaches and implementing mechanisms as well as the prospects for sustaining ongoing activities. Primary and secondary data has been disaggregated by appropriate demographics and target populations and has included age, gender/sex, and KVP type as possible. Key Components of the Evaluation Design • Mixed methods design, integrating a thorough analysis of secondary performance data (Project documentation, DATIM, IPRS and SAUTI MIS) with primary qualitative data generated from KIIs and FGDs during fieldwork • Highly KP and context sensitive • Purposive sampling of limited sites that compare more comprehensive package of services with basic Sauti service package; DREAMS vs non-DREAMS • Secondary data provides the broad evidence base for findings on results achieved, and informs explanatory findings • The primary data, generated through the fieldwork serves to check implementation fidelity and performance at sampled sites, identify persistent challenges across sites, generate explanatory findings, document progress towards results and unmeasured outcomes The evaluation relied on the following data collection and analysis methods described in further detail below: Document Review Documents reviewed included the Sauti Cooperative Agreement and its modifications; relevant sub￾agreements; Project start-up documentation; work plans; annual and quarterly reports; monitoring and internal reports; and other relevant documents from USAID, the GoT, and other stakeholders. Supplementary documents such as published research and indicator data was received from USAID and the Sauti Project. The ET received several documents from the Project. See Annex II, page 55 for the list of documents received and used in the analysis. Performance Monitoring Data Performance monitoring data produced by the Sauti Project was reviewed and organized by EQ and analyzed using descriptive statistics. No advanced statistical analyses were used in writing the report. 5 Key Informant Interviews KIIs are semi-structured individual or small group (1-8 persons) interviews conducted with key informants who are representative of principal Project stakeholder groups. (See Annex III for copies of the KII and FGD discussion guides.) The ET conducted 23 semi-structured interviews to collect data related to the five above EQs. The ET worked with USAID/Tanzania Mission Health office and the Sauti program to identify and purposively select stakeholders who have played a role in the SAUTI implementation as well as beneficiaries who received services from the program. Respondents for national level KIIs were purposively selected and included senior-level staff from the SAUTI consortium partners, National Council of People Living with HIV (NACOPHA), CDC and relevant GOT staff from National AIDS Control Program (NACP), President’s Office for Regional Administration and Local Government (PORALG) and Tanzania Commission for AIDS (TACAIDS) from KVP area to answer the relevant SAUTI EQs. The ET also held KIIs/GIs with the regional and district health officials in the 3 regions covering the 5 selected districts. To ensure the EQs were being properly addressed, the ET used snowball sampling to get some of the informants to suggest others who should participate in the interviews. In facilities, the ET purposively selected clinical staff who worked most closely with Sauti. All interviews were recorded with informed consent obtained prior to the start of the interview5. Group Interviews Using some of the same protocols as with the KIIs, the ET conducted 16 GIs in groups of 2 to 7 informants. In some cases, GIs were held in place of planned KIIs when other relevant parties were inclined to meet as a group or have others accompany them. This was often the case with district and regional government officials and CSOs reach through the process. In each of three sites, groups of informants, including health officers at the regional and district levels, were reached through GIs. At the regional level, the ET conducted GIs with the few members of the regional health management team (RHMT); at the district level, the team had GIs with different members. For example, in Mbeya, the ET had GIs with KIWOHEDE and MHNT CSOs, Kyela CHMT, health facilities service providers, and CHAC & DCDO. The ET used the same guide for GIs and FGDs with the CHMT and RHMTs. Focus Group Discussions The ET held 34 focus groups with clients/beneficiaries of SAUTI, two FGDs with CHMT and one with RHMT. Each focus group consisted of 8 to 20 participants. The purpose of the evaluation was explained at the commencement of all interviews and FGDs. The FGDs conducted with beneficiaries were with MSM, vAGYW, CBHSPs/Empowerment Workers and Community ART providers. The evaluators explained that they would not be using information collected in a way that would disclose the source, and that responses would be aggregated when reported. All interview and focus group participants were informed that they had the right to decline to answer any questions and to end their participation in evaluation activities at any point, without adverse consequence. Verbal consent for participation was sought prior to the commencement of all interviews and FGDs. In each of five sites (three sites that jointly have both Sauti and DREAMS being implemented; and two that only have Sauti) the evaluation team organized groups with key vulnerable populations and a combination of community-based health care providers (CBHSPs) and empowerment workers. SAUTI HQ prepared a sampling frame from the sampling parameters as provided by ET. The sampling parameters were such as age limit (for vAGYW, they had to be 18 years and above), sex, and the geographic locations. For the vAGYW and FSW in the DREAMS districts, the ET preferred them to be those receiving most or all services in the package. 5 Except CHAC-Shinyanga MC, CHMT-Kyela and Nurse from Kyela who did not provide consent to be recorded. 6 From the sampling frame, SAUTI randomized to get 16 participants for each of the FGD aiming to get 8- 12 participants after refusals and no shows. For confidentiality, ET had no access to personal identifying information of the selected clients. The list of randomly selected participants was sent directly to SAUTI regional program managers who worked with the CSOs to identify the clients for FGDs. Despite having randomly selected beneficiaries sent to CSO, for some of the sites, the ET had to change the approach of inviting clients for FGDs after identifying some challenges. In all the sites, CBHSPs were involved in finding FSWs who have been in touch with them and know their locations. Due to the mobile nature of the FSWs, CBHSPs had to use respondent-driven sampling to identify FSWs to participate in the FGDs as majority of the participants in the sampled list had wrong names and moved away from the area. Peer network also worked to identify FGD participants for FSWs, particularly in Kyela DC. For Kyela, vAGYW were hard to find using the sampled list, however, the ET agreed with the CSOs to purposively select one participant from the 24 vAGYW WORTH groups to form 2 FGDs with not less than 12 participants each. In Kyela DC, peer network was used to identify FSW. Due to the challenging legal environment for men who have sex with men (MSM), MSM participants for FGDs were to a large extent purposively selected to ensure that they can provide accurate responses in a group or individual interview in a safe space. Respondents for Empowerment Workers and CBHSPs’ FDGs were selected using the sampled list from Sauti of those who still work with them. Four groups which had less than 8 participants were changed to GIs instead, following the same discussion guide as the FGD. FGDs were conducted by Swahili speaking facilitators. All FGDs were recorded with informed consent obtained prior to the start of the discussion. FGDs transcripts were coded using Dedoose to produce a summary of recurrent themes that emerge in response to each topic. Table 2: KIIs/GIs and FGDs by Stakeholders and Locations Joint DREAMS + SAUTI Location Only SAUTI Location Data Collection Activities Shinyanga MC (Shinyanga Region) Kyela DC (Mbeya Region) Temeke District + National (Dar es Salaam) Kinondoni (Dar es Salaam) Mbarali (Mbeya Region) Total National-Level KIIs/GIs. KII01 –Sauti Consortium members and partners (JHPIEGO, PACT, ENGENDER, NIMR) - - 4 - - 4 GoT: PO-RALG, NACP and TACAIDS6 - - 3 - - 3 Other Donors: Centre for Disease Control and Prevention (CDC) - - 1 - - 1 National CSO representative: NACOPHA - - 1 - - 1 Total National-Level GIs/KIIs - - 9 - - 9 Regional-Level KIIs/GIs. 6 Interviewed Regional Coordinator for TACAIDS 7 Joint DREAMS + SAUTI Location Only SAUTI Location Data Collection Activities Shinyanga MC (Shinyanga Region) Kyela DC (Mbeya Region) Temeke District + National (Dar es Salaam) Kinondoni (Dar es Salaam) Mbarali (Mbeya Region) Total Regional Health Management Team (RHMT) 1 1 1 - - 3 Regional Reproductive and Child Health Coordinator (RRCHCO) - 1 - - - 1 Regional Medical Officer (RMO) - 1 - - - 1 Implementing Partners at Local Level￾CSOs:WASO, PHSRF, MUKIKUTE, KIWOHEDE, RAFIKI SDO 1 1 2 1 1 6 Total Regional-Level GIs/KIIs 2 4 3 1 1 11 District-Level KIIs/GIs. Council Health Management Team (CHMT) 1 1 1 1 - 4 District Reproductive and Child Health Coordinator (DRCHCO) - 1 1 1 1 4 Nurses 1 1 1 1 - 4 Clinicians 1 - 1 1 - 3 Nurses and Clinicians(combined) - - - - 1 1 District AIDS Control Coordinator(DACC) - 1 - 1 1 3 Council HIV/AIDS Coordinator (CHAC) - - 1 - 1 CHAC and District Community Development Officer (CHAC & DCDO) - 1 - - 1 2 Total District-Level GIs/KIIs 3 5 4 6 4 22 Total GIs/KIIs 42 Community-Level/Beneficiaries FGDs Male Sex with Male (MSM) 1 2 1 1 5 Female Sex Workers (FSW) 2 3 1 1 2 9 8 Joint DREAMS + SAUTI Location Only SAUTI Location Data Collection Activities Shinyanga MC (Shinyanga Region) Kyela DC (Mbeya Region) Temeke District + National (Dar es Salaam) Kinondoni (Dar es Salaam) Mbarali (Mbeya Region) Total Vulnerable Adolescent Girls and Young Women(vAGYW) 2 2 2 2 2 10 Peer Educators (Community-Based Health Service Providers￾CBHSPs/Empowerment Workers-EWs) 2 1 2 1 1 7 Total FGDs 6 7 7 5 6 31 Sample selection Field research was conducted in 5 district sites in the original 3 selected regions in Dar es Salaam, 3 sites that have both Sauti and PEPFAR Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe Women (DREAMS) interventions: (Temeke Dar es Salaam), Shinyanga MC (Shinyanga), and Kyela DC (Mbeya); there are two non-Dream Sauti sites in the same regions (Kinondoni Dar es Salaam and Mbarali DC in Mbeya). Having a balance between DREAM and Sauti only sites allowed the team to reach vAGYW in each context. The sites also have a balance among the 5 intervention types and key informant groups. At each site, SAUTI stakeholders were purposively selected and SAUTI beneficiaries to participate in KIIs, FGDs and GIs were recruited using mixed approaches including random selection, purposive sampling and respondent-driven sampling (snowball sampling). Focusing on the regional capitals allowed the ET to conduct GIs with regional- and municipal-level informants in a single site. Interviews with national-level stakeholders were conducted in Dodoma and Dar es Salaam. Site Selection The 5 district level sites were selected purposively to ensure a balance between DREAMS and Non￾DREAMS sites and to be able to reach a diverse set of key informants as well as observe all Sauti activities. The sites were selected to prioritize the presence of all 5 consortium members and DREAMs interventions. One site was selected in Shinyanga region, 2 sites in Mbeya region and 2 sites in Dar es Salaam region. District level selection used the following criteria. 1. Balance between DREAM and Non-DREAM districts 2. Prevalence of 5 intervention types and consortium partners 3. Prevalence of Key Populations and other Key Informant Groups by district 4. Accessibility and time considerations DREAMS is implemented in Kyela DC (Mbeya), Msalala DC, Ushetu DC, Kahama TC and Shinyanga MC (all in Shinyanga) and Temeke MC (Dar es Salaam). Since the performance evaluation aims at comparing services from DREAM and non-DREAM sites in the same region, 1 DREAM site and 1 non-DREAM site were selected from Dar es Salaam and Mbeya regions. Since there is no non-DREAM site in Shinyanga, only one site, Shinyanga MC, was selected without a comparison site. Temeke MC and Kyela DC were selected as DREAM sites from Dar es Salaam and Mbeya region, respectively; Kinondoni MC and Mbarali DC were selected from the same regions, respectively, as non- 9 DREAM sites. Consequently, Kinondoni MC was compared with Temeke MC while Kyela DC will be compared with Mbarali DC and Shinyanga and was treated as a standalone DREAM site. 3.2 DATA ANALYSIS METHODS To analyze the qualitative data collected via document review, KIIs, and FGDs, the ET conducted a content analysis of the relevant documents and interview/discussion notes using the Dedoose (https://www.dedoose.com/) qualitative analysis software. Thematic grounded coding is a technique for analyzing qualitative data that thematically organizes the narrative in the texts across instruments into fewer content categories based on grounded coding. The ET used thematic coding to identify and clarify patterns in the data among the codes thereby allowing it to draw inferences from the qualitative data by objectively and systematically identifying specific themes and sub-themes within the data, and assessing their relative importance in answering the EQs supported by key quotations and examples from individual documents, KIIs, or FGDs. Primary codes and secondary codes were assigned to relevant sections of the text to analyze and tabulate recurring themes across instruments. Themes and findings across instruments were triangulated in relation to the relevant EQs in the process of developing the findings for this report. 3.3 METHODOLOGICAL LIMITATIONS It is important to identify here some limitations inherent to the design of this evaluation:  Data availability and data quality: While the ET collected and generated primary data, they relied on the Sauti consortium providing comprehensive, good quality performance monitoring data. However, the ET was not able to verify the underlying accuracy of this data. Also, it was not possible for the evaluation to collect quantitative data other than that provided by the Project.  Selection bias: As some key informants declined to be interviewed, hence there is a possibility of selection bias, i.e. those respondents who accepted to be interviewed might differ from those who did not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. In addition, the purposive nature of the site selection process introduces additional selection bias. Due to confidentiality issues, the only feasible strategy for recruitment of FGD participants was to seek the assistance of the CSOs. There may have been some selection bias in recruitment by the CSOs despite having the sampled list of beneficiaries from SAUTI. This was addressed by asking CSOs to recruit participants on a random basis.  Recall bias: Since several questions raised during the interviews addressed issues that took place from 2015-2018, informants might not have been able to provide accurate and complete responses.  Halo bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm, called halo bias. The extent to which respondents were prepared to reveal their true opinions might have been varied for some questions that called upon the respondents to assess the attitudes and perceptions of their colleagues or people upon whom they depended for the provision of services. To mitigate this limitation, the ET outlined confidentiality and anonymity statements to all who participated in KIIs, FGDs, and GIs. The ET also conducted the interviews in a neutral setting where respondents felt free and comfortable. Another measure that was taken was to have separate FGDs for men and women when appropriate and to have GIs with relatively homogenous participants with respect to rank.  Additional Data Collection: Upon request by USAID, the ET included additional family planning stakeholders and more FGDs for key populations in the selected sites out of Dar es Salaam; it should be understood that supplemental recruitment was purposive to get respondents who are informed about and have received services from the Project and not randomized which may have had some amount of bias for this subset of the sample selection. 10 4.0 FINDINGS & CONCLUSIONS This section of the report presents the evaluation findings arranged by EQs and sub-questions. The discussion of each question is immediately followed by the conclusions, which the preponderance of evidence points to, and includes a response to the evaluation questions. 4.1 EQ1. HOW EFFECTIVE IS THE PROJECT IN ACHIEVING ITS GOALS, OBJECTIVES AND PERFORMANCE TARGETS? Findings on Project Performance 1.1.1 To what extent has the Project succeeded in meeting performance expectations against PEPFAR targets? Sauti tracks an extensive set of indicators, including PEPFAR Monitoring Evaluation and Reporting (MER) and custom indicators. Table 3 presents a summary of performance against only the select indicators that feature consistently in quarterly reporting and are most relevant for demonstrating Project performance. Achievement of 95 percent or greater against target is highlighted in green, performance from 80 percent to 94 percent against target is highlighted in yellow, and performance below 80 percent against target is highlighted in red. Even a cursory overview of the tabulated results shows that Sauti consistently achieves or overachieves its targets. In fact, the level of overachievement on a number of the indicators suggests that the assigning of targets by PEPFAR and the Mission is not optimized and may need to be refining. Two important exceptions to the achievement trends must be noted. The first exception relates to the initial set of indicators in the table, namely, KVP HTS, KVP HIV+, KVP Enrolled in CTC, and Enrollment Rate. Together, these indicators document the case finding and linkage cascade. KVPs are screened for risk and the eligibility for services before being linked to testing (KVP HTS). Those that test positive (KVP HIV+) are then linked to treatment (KVP Enrolled in CTC). Given Tanzania’s poor progress towards 90-90-90 targets7, performance on this cascade is the highest priority for epidemic control. Sauti has not been as successful in linking newly found positive clients to treatment as it has against other indicators. The numbers do indicate however, that the Project has improved performance on steps of the cascade over the implementation period. It is apparent that screening has become more stringent and more accurate, as demonstrated by the higher case finding yield from FY2017, despite an initial drop in numbers tested that year. Enrollment in CTC has also improved, from an initial poor 12 percent, to steady in the 90 percent to 94 percent range. The linkage rate – the proportion of newly found positives enrolled into CTC – remained low. However, a promising improvement on the linkage rate is reflected in the year to date linkage rate for FY2019, currently at 84 percent. This steady improvement in an intervention area critical to epidemic control is attributable, in part at least, to the Project’s responsiveness to data, an observation to be detailed in findings on EQ2 7 52.2 percent of people living with HIV (ages 15 to 64) in Tanzania report knowing their status, 90.9 percent of those individuals self-report being on ART, and 87.7 percent of that group are virally suppressed - Tanzania HIV Impact Survey (THIS) 2016-2017 11 Table 3: Project Performance against Targets Indicator FY16 (Oct15-Sep16) FY17 (Oct17-Sep17) FY18 (Oct17-Sep18) FY19 (1Oct18-23Mar19) APR Target Achieved APR Target Achieved APR Target Achieved To Date Target Achieved KVP HTS 653482 670467 97% 505274 541682 93% 1212125 1 141054 106% 293544 501295 59% KVP HIV+ 17157 49840 34% 35718 37450 95% 39805 36560 109% 19 212 37176 52% KVP Enrolled CTC 4922 39872 12% 27556 29960 92% 28134 31076 91% 16 226 35317 46% Enrollment rate - - 29% - 77% - - 71% - - 84% PrEP - - - - - - 3065 4192 73% 621 1847 34% HIVST - - - - - - 4298 27645 16% 8089 29574 27% AGYW FP 3060 - - 14168 13503 105% 95 595 14867 643% 42 720 52011 82% FSW FP 4781 - - 30409 24457 124% 195 885 26903 728% 69 467 91183 76% KP Prev (FSW) 41901 30191 139% 38748 40108 97% 51 685 50144 103% 34 789 64995 54% KP Prev (MSM) 7906 5706 139% 3253 5653 58% 7 428 6565 113% 4071 8792 46% PP Prev (AGYW) 67835 58039 117% 88463 71122 124% 111 164 112806 99% 76 412 146105 52% Comb SE (AGYW) 14933 39989 37% 21894 21289 103% 33 956 32030 106% 21 014 31254 67% Parenting - - - 7845 6177 127% 33 481 32030 105% 21 516 31254 69% Social Assets Building (SAB) - - - 11616 11769 99% 16 519 16017 103% - - - Gender norms 87109 122475 71% 34544 17281 200% 16 575 16420 101% 22 824 42704 53% CTP - - - 11216 12144 92% - - - - - - 12 The second exception to the overachievement trend is observed in the indicators tracking the distribution of PrEP and HIV self-testing kits. The factors constraining performance on these indicators are also discussed under EQ2, as are challenges to the utility of self-testing for case finding and linkage to treatment. 1.1.2 To what extent has the Project improved access to and use of a core, quality package of services for KVPs? It is difficult to precisely determine the extent to which access to a core package of quality services has been improved for KVPs. As explained in the Project background section, Sauti offers numerous services, differentiated into a variety of service packages, each targeting distinct categories of beneficiaries. Moreover, guidance from Office of the US Global AIDS Coordinator (OGAC) and the Mission on what constitutes a core package of services is subject to frequent revision. There is therefore no static core package of services to consider. There is also no counterfactual for access: pre-Project quantitative data on KVPs’ access to services is not readily available for comparison with Sauti data. Tracking the layering of service is also problematic. HIV+ clients enrolled in treatment have CTC ID numbers and their accessing of biomedical services is therefore traceable. Sauti is taking advantage of this in an initiative to share datasets with care and treatment IPs, however traceability will only apply to HIV+ clients and biomedical services. Broader tracking relies on Sauti’s assigning a unique identification code (UIC) to each beneficiary, which is recorded in all the service tools along with the beneficiary’s information. The UIC is not strictly unique, as the algorithm used to generate it (employing elements of beneficiary’s name, names of relatives, birth region) does not exclude all possibility of duplication. It is also prone to recording errors, and beneficiaries supplying different information at different service points will end up with multiple UICs. Where UIC errors are eliminated, Sauti can track access to services over time, as well the layering of services. While it is possible, therefore, to determine service layering for Sauti clients, it is not done routinely and requires some effort to execute. The effort to determine layering is reserved for special analyses. Examples include Sauti’s assessment of sero-conversion among FSW, presented at CROI19; the assessment of HIV testing yields among AGYW, correlated with vulnerability index scores, presented at IAS17; and an assessment of the effect of layering services on HIV+ clients and their enrollment in ART is in the pipeline as part of an internal impact evaluation. Despite factors limiting quantitative analyses to substantiate claims that access has been improved, the sheer numbers reached with services over the implementation period support such finding. Figure 1 illustrates the enormous growth in KVPs served to date. In addition, qualitative data from interviews and focus groups persistently confirms that access to services for KVPs was and remains severely limited outside of Sauti (4/5 FGDs with MSM; 6/10 FGDs with FSW; 5/10 FGDs with vAGYW); that access has been substantially improved for Project clients (5/5 FGDs with MSM; 9/10 FGDs with FSW; 7/10 FGDs with vAGYW); that the range of services available has increased and service gaps have been reduced (3/5 FGDs with MSM; 8/10 FGDs with FSW; 8/10 FGDs with vAGYW); and that the quality, utility and friendliness of available services has been enhanced (4/5 FGDs with MSM; 6/10 FGDs with FSW; 6/10 FGDs with vAGYW). The most frequently reported deficit in Project services was the shortage of commodities (2/5 FGDs with MSM; 6/10 FGDs with FSW; 5/10 FGDs with vAGYW), condoms in particular, but also PrEP and HIV self￾testing kits. Stockouts expose a dependency related risk for Sauti, where service project performance is subject to the influence of other actors such as government and suppliers. Similarly, inconsistencies in the delivery of service packages – where clients report not receiving services they are entitled to and which their peers benefitted from - were also observed in the field, possibly reflecting performance issues at particular Sauti sub-recipients. 13 Figure 1: Cumulative Numbers of KVPs Reached Per Service Category - Finally, beneficiaries are aware of differentiated access to services that disadvantage them. For example, only a very limited number of beneficiaries are linked to vocational training offered through the Kizazi Kipya OVC platform, due to narrow eligibility criteria. Sauti has introduced innovations to try and address the difference, offering vAGYW some skills training in a number of economic generating activities. While these enhancements cannot match the benefits derived from a comprehensive Technical and Vocational Training (TVET) course, the Project is required to navigate stringent guidance and eligibility criteria and make the best of hard budget constraints. These realities are reflected in larger numbers, such as the proportion of vAGYW in the Project enrolled in WORTH groups. Although differentiated service packages should be regularly reviewed and optimized, they remain a necessary feature to not only strengthen effectiveness, but direct resources towards priority needs. 9227 662709 1167983 2380108 2673652 0 500000 1000000 1500000 2000000 2500000 3000000 FY15 FY16 FY17 FY18 FY19 to Date KVPs Received HTS 69 4991 32547 60681 76907 0 20000 40000 60000 80000 100000 FY15 FY16 FY17 FY18 FY19 to Date KVP Enrolled to CTC 427 8268 52845 344325 456512 0 100000 200000 300000 400000 500000 FY15 FY16 FY17 FY18 FY19 to Date KVP Received FP 905 118547 249011 419288 645722 0 200000 400000 600000 800000 FY15 FY16 FY17 FY18 FY19 to Date KVP Received BC 0 14933 36827 70783 91797 0 20000 40000 60000 80000 100000 FY15 FY16 FY17 FY18 FY19 to Date KVP Received Comb SE 14 1.1.3 To what extent has the Project succeeded in improving positive behaviors and social norms at individual and community levels? The indicators in Table 3, above, that track delivery of behavior change interventions consistently demonstrate over-achievement against target. The exception is the KP_Prev result for FY17, which is explained by the GoT’s politicization of KVP issues and their subsequent actions, which resulted in MSM in particular avoiding any activities that would put them at risk. Focus group data suggest that behavior change interventions are producing results. Participants emphatically reported changes in health seeking and risk reduction behaviors: there is an increased willingness to test, seek and adhere to treatment (5/5 FGDs with MSM; 10/10 FGDs with FSWs; 7/10 FGDs with vAGYW); the demand for and consistent use of commodities (condoms, PrEP, self-testing) is repeatedly raised (3/5 FGDs with MSM; 9/10 FGDs with FSWs; 6/10 FGDs with vAGYW); and the demand for family planning services (6/10 FGDs with FSWs; 5/10 FGDs with vAGYW). FSWs also relate instances of negotiating the terms of their sexual transactions, insisting, for example, on the use of condoms, or that partners administer an HIV self-test. This assertiveness is an example of the generalized sense of empowerment FGD respondents (especially FSW and DREAMS beneficiaries) consistently present and attribute to their participation in Sauti. The importance of an adequate supplies of commodities is accentuated in the light of these observations. The extent to which self-testing results in a behavior change dividend, however, (reducing risky sexual behavior, going for re-testing, enrolling in treatment) is difficult to verify, as test results are not routinely disclosed (2/6 FGDs with CBHSPs and Empowerment Workers). Focus group participants also reported reduced stigma by health workers when accessing services at facilities (2/5 FGDs with MSM; 7/10 FGDs with FSW; 6/10 FGDs with vAGYW), which they attribute to Sauti, and reduced stigma in their communities, both of which they attribute to Sauti (2/5 FGDs with MSM; 7/10 FGDs with FSW). Participants often associate the latter outcome with improved socio-economic circumstances, also attributed to Sauti interventions. Although FSWs and vAGYW engaging in transactional sex are often compelled by severely limited economic circumstances, the steady improvements in their prospects offered by economic strengthening activities has allowed some the option of exploring economic alternatives (2/10 FGDs with FSW; 2/10 FGDs with vAGYW). 1.1.4 To what extent has the Project strengthened capacity of local institutions and services? Sauti continuously provides technical assistance to its sub-recipients through a cycle of organizational needs assessments, jointly developed improvement plans, training, systems development, supportive supervision and continuous quality improvement (Quarterly Reports, FY15Q1 to FY18Q4). CSO interview respondents report substantial gains from this support, including strengthened financial management, improved governance, adoption M&E and data use practices, better program management and implementation, and more effective personnel performance management. Respondents frequently claim that these gains have allowed them to obtain additional funding from other donors. In addition to enhancing the skills base and organizational competency of civil society organizations, Sauti’s funding has expanded the available workforce for health services to KVPs. The program model is structured around the mobilization of volunteer cadre’s – peer educators and empowerment workers – who are resourced and paid a stipend with Sauti funds. In addition, clinicians are paid to provide services in community outreaches, also with Sauti funds. A number of interview respondents recognize this contribution and remark on the fact that it is this substantial expenditure that has enabled community based, KVP targeted services, including government officials. The entire system is substantially strengthened through the expanded workforce. Sauti also supports government through technical assistance, participating in national efforts to design guidance and tools for services to KVP. A pivotal contribution was their leading role in the development of the 2017 guidelines for service to KVP, which were developed during a critical period in which GoT 15 was marginalizing KVP populations and curtailing services to these populations. Sauti supports GoT at local level, coordinating technical activities with regional and council health management teams. Officials on these teams and others are engaged by Sauti to provide joint supportive supervision to the Sauti’s sub-recipients. In addition to sponsoring these supportive supervision activities, Sauti trains personnel and provides tools for the process. In addition to enhancing the skills of participating officials, an advocacy dividend is realized as GoT employees are exposed to KVP focused public health work. 1.1.5 To what extent has the Project succeeded in executing a robust learning agenda? Between Project launch in 2015 to date, Sauti has had 17 abstracts accepted for presentation at high level forums in the HIV scientific community, including the International Aids Conference, the International Aids Conference on HIV Science, the African Conference on Key Populations in the HIV Epidemic, HIVR4P, and the Sexual Violence Research Initiative Forum. In addition to contributing to the broader body of knowledge, Sauti has systematically executed an operationally focused research agenda in collaboration with research partners, distilling key findings into implications for implementation and integrating these into subsequent Project planning. An overview of research partners and Projects is presented in Table 2. Table 4: Overview of Research Partners and Projects Research Partner Research Projects Sauti NIMR Mwanza • Qualitative study to inform HIV intervention delivery among MSM • Geographical and Virtual Mapping of Key Populations in Tanzania • Cash Transfer to Adolescent Girls and Young Women (AGYW) to Reduce Sexual Risk Behavior • User-centered insights to support HIV testing and linkage to care services for the key and vulnerable populations of adolescents and young adults in Tanzania • Effectiveness of community ART Project SOAR • Family planning among HIV+ FSW in Tanzania Gates Foundation • Improve effectiveness of cash transfers using behavioral economics and human￾centered design In KIIs stakeholders attributed a number of outcomes to Sauti’s active sponsorship of a research agenda, including elevating the primacy of evidence-based decision making within the Project and among GoT stakeholders, improving access to health services and the availability of commodities through the application of research findings, and strengthening the capacity of local institutions to conduct meaningful research (1/2 KIIs with National Stakeholders, and 1/4 KIIs with Consortium Partners). Conclusions on Project Performance Sauti is incontestably effective in meeting its targets. Where there are deficits in achieving targets, such as linking newly found positives to treatment, Sauti is responsive to performance data, utilizing and supplementing the data as necessary to develop solutions and meet targets over time. By virtue of sheer volume of numbers reached, it is fair to conclude that access to services for key populations has been enhanced through the Project, a conclusion corroborated by focus group and interview data. However, the inability to routinely track layering of services makes it difficult for the Project to rapidly adapt to layering related service delivery issues, and to optimally learn which combinations of services are most effective, for whom and under which circumstances. Results for beneficiaries are apparent in terms of gains in health, health seeking behaviors including family planning services and, to a lesser extent, improved economic prospects. There is also evidence suggesting shifts in community and facility personnel attitudes towards key populations where Sauti is active. The latter conclusion should be qualified however: the attitudinal changes perceived by KVPs may simply reflect 16 the sense of empowerment that their participation in Sauti interventions seems to cultivate. Increased health seeking behavior is also incentivized, by offering complimentary services to KVPs, in addition to those focused on epidemic control. Sauti has also contributed substantially to the development of local capacity – both in government and civil society - to utilize evidence and implement services. It has accomplished this through intensive technical support, but also through substantially increasing funding to CSOs, augmenting their reach and resources for service delivery. The latter conclusion has implications for the discussion on self-reliance and sustainability. 4.2 EQ2. WHAT FACTORS EXPLAIN THE ACHIEVEMENT OR UNDER￾ACHIEVEMENT OF PROJECT RESULTS? Both EQs 2 and 3 address the factors that enable or constrain performance. EQ 2 considers Project￾related factors influencing performance, while EQ 3 focusses on specifically identified factors in the implementing environment. Therefore, while findings are presented separately, the relevant conclusions are considered together in a section immediately following the findings discussion. Findings on Factors Enabling or Constraining Performance for EQ2 2.1 What are the Projects strengths, weaknesses and gaps in design? 2.2 Has the Project been implemented with fidelity? 2.3 Have interventions been implemented to the required level of quality for achieving results? 2.4 How have any of the Projects’ capacity/operational (planning, management etc.) strengths, weaknesses and gaps influenced the achievement of results? This section presents findings on management and implementation practices that data suggests are pivotal in explaining Project performance. These practices are designed into the Project, key features of operational capacity and are responsible for preserving implementation fidelity and quality. The findings below address the sub-questions in an integrated way, and are therefore not arranged by sub-questions, but in a single discussion that covers the scope delineated by the sub-questions. The outstanding feature of Sauti implementation that is demonstrably linked to strong performance is the practice of intensive, data-driven adaptive management. Sauti amasses, analyses and uses data to: (i) routinely track performance against targets on a daily basis, and manage the performance of sub-recipients towards achieving those targets; and (ii) identify emerging challenges to performance, diagnose the causes of faltering performance, and develop, implement and track the efficacy of solutions. In terms of routine performance, Sauti has already modeled the required resources to results ratio and mapped out daily performance targets per sub-recipient, which in turn divides targets amongst individual personnel—community-based health services providers, peer educators and empowerment workers— as appropriate. Sub-recipients report daily target achievements in WhatsApp groups. These preliminary numbers are consolidated by the central M&E team and submitted for the senior technical team to review. The numbers are formally validated by the end of each week, and inform the monthly planning done by the central technical team. Monthly plans are then disseminated to regional teams, together with guidance tailored to specific sub-recipients if necessary. Implementation efforts are adjusted by all parties to meet the requirements of the updated monthly plan. Daily routine tracking also allows for immediate responses to urgent issues in the field as they arise. The accumulated data also supports more substantial adaptive management efforts. For example, the persistent performance gap between identifying new positives and successfully linking them to treatment was diagnosed using this data. The causes of under-performance were identified as new found positives self￾referring to treatment (12 percent of those not inked to treatment), untraceable clients (21 percent of 17 those not linked to treatment), and backlogs in the workload of peer educators (54 percent of those not linked to treatment). Diagnoses were differentiated by District and remediation is being implemented. In addition to Sauti’s intensive, data-driven adaptive management practices, data from focus groups and interviews point to a number of implementation modalities that may enhance Project performance. The most prominent of these is descriptions of ways in which access is facilitated. Focus group participants remark on how an escorted referral, a SAUTI referral form or identification card gets you attention at a facility, and “jumps you to the front of the queue” (2/5 FGDs with MSM; 7/10 FGDs with FSW; 6/10 FGDs with vAGYW). Escorted referrals or a clear association with the Project also appear to negate demands for non-mandated, prohibitive user fees (2/5 FGDs with MSM; 7/10 FGDs with FSW; 7/10 FGDs with vAGYW), and alleviate the fear of stigma for new SAUTI clients, a crippling barrier to access for many (3/5 FGDs with MSM; 9/10 FGDs with FSW; 7/10 FGDs with vAGYW). KVPs claim that overall, facilitated access strengthens their recognition of services as a right and their confidence to access those services (2/5 FGDs with MSM; 5/10 FGDs with FSW), and the need for escorted referrals appears to diminish over time. Many respondents seem to concur that the association with Sauti will not be required indefinitely, and that improved service provision at facilities represents a permanent change (2/5 FGDs with MSM; 5/10 FGDs with FSW; 4/10 FGDs with vAGYW). This confidence is not ubiquitously shared, however. The very active role fulfilled by Sauti in facilitating access is considered indispensable by some respondents, and they view the current ease of access to be at risk after the Sauti Project ends (2/5 FGDs with MSM; 5/10 FGDs with FSW; 4/10 FGDs with vAGYW). Implementation modalities that provide Sauti with access to KVP networks also appear to be key elements influencing Project performance. Focus group participants describe their initial resistance to enrolment being eroded by the efforts of peer educators (2/5 FGDs with MSM; 8/10 FGDs with FSW), in part because the latter enjoyed credibility as members of their groups (2/5 FGDs with MSM; 8/10 FGDs with FSW). In addition, the initial encounters with Sauti at places KVPs frequent, at times convenient to them, are also named by respondents as factors that eased their willingness to access services. The volunteer workforce and peer educators particularly, are crucial to project performance. However, while peer educators consistently report being intrinsically motivated to work in the Sauti project (6/6 FGDs with CBHSPs and Empowerment Workers), they do raise issues that could potentially disincentivize them. The most persistent of these is the inadequacy of the stipend they receive (6/6 FGDs with CBHSPs and Empowerment Workers), especially in the light of the workload demanded of them (2/6 FGDs with CBHSPs and Empowerment Workers). At times the demands seems puzzling and potentially open to gaming. For example, one of the sub-recipients incentivizes the finding of new positives (1/6 KIIs with local CSOs), a difficult demand to meet when yields are prioritized and testing volumes discouraged. Emphasizing finding new positives in the pay for performance system could prejudice other crucial priorities, such as linking existing HIV positive clients to care. The mobility of KP populations and the limits to mobility of peer educators also makes it difficult to provide follow-up and sustain services (2/6 FGDs with CBHSPs and Empowerment Workers). Finally, respondents explain that accepting offers of key services was incentivised by offers more meaningful to them. For example, the provision of free condoms or free STI screening aligned with the participants’ own priorities. Accessing these led on to accepting the offers of testing, enrolment into treatment, and participation in other Project activities. The success of demand creation is not always matched by supply, however: demand for condoms, PrEP, self-testing kits and community-based ART distribution is high, but shortage of commodities is persistently reported. In some instances, shortages may have a deleterious effect on demand, while in others demand will be sustained. Systemic dependencies represent a set of constraints that pose substantial risk to Project performance but that are not controlled by the Project’s actions. The outstanding case for Tanzania is the politicization 18 of key population and family planning issues, discussed in the section on EQ 3. An example more relevant to Project implementation is the underperformance against PrEP and HIV self-testing targets. The under￾achievement was not due to Sauti’s implementation failures, but to delayed roll out of HIVST and PrEP, and, in Q1 FY19 (Oct-Dec 2018), a serious shortage of PrEP medications as the national supply was not ready yet to include PrEP in their system. 4.3 EQ3. HOW EFFECTIVELY HAS USAID AND ITS IPS PREPARED FOR AND/OR RESPONDED TO CONSTRAINTS TO IMPLEMENTATION? Findings on Factors Enabling or Constraining Performance for EQ3 4.3.1 Has the response to legal barriers constraining implementation among FSW and MSM been effectively dealt with? The 2014 National Guidelines for Comprehensive Package of HIV Interventions for Key Populations recognized the vulnerability of certain populations to HIV, as well as specific legal and social challenges these at-risk groups face. Under this guidance, Sauti and other programs provided special interventions to reach out to these groups with comprehensive preventive, treatment, care and support services. However, in 2016, after the election of President Magufuli, the Ministry of Health suspended drop in centers for MSM, halted Sauti and other donor activities, and prohibited the import and use of HIV prevention commodities like non-oil-based lubricants. Meanwhile, the Regional Commissioner of Dar es Salaam announced a task force to track down MSM and many were arrested. This politicization significantly undermined efforts of all stakeholders to provide services to KVPs, and this was reflected in the performance of Sauti against target. In response to these legal barriers, Sauti intensified collaboration with local government authorities and organized advocacy meetings to ensure more understanding of the Project’s aims in reaching these populations with HIV services.8 Sauti also played a pivotal role in mobilizing stakeholders to contribute to the development of new KVP national guidelines. In 2017, the Ministry of Health published the National Guidelines for the Management of HIV and AIDS which removed references to KVPs. In KIIs, Sauti consortium members, and national stakeholders were asked to discuss how the political, legal and policy environment enables or constrains the provision of services to KVPs. Additionally, district and regional medical officers were asked about barriers to providing such services. District councils do not prioritize services to KVPs The attitude of the GoT toward homosexuality, especially among MSM, continues to constrain implementation. Officials speak of individuals who are “affected” by homosexuality and they are outspoken against MSM (2 of 3 KIIs with GoT). A couple of stakeholders are concerned that the lack of recognition of MSM by the GoT has led to poor data. Implementers must report on incidence among men overall, diluting the high prevalence of HIV among MSM (1 of 6 KIIs with local CSOs, 1 of 5 KIIs with regional medical officials). As one regional medical official stated in a KII: “The big thing is that the government has already banned MSM; we don’t deal with that. They are KVP, we don’t deal with that, and that is a big challenge that we do encounter. So, if you ask me about MSM data, for sure I cannot tell because we do not collect them.” Various stakeholders report that the GoT is not prioritizing funding for services to KVPs and instead funds other vulnerable populations (1 of 3 KIIs with GoT, 1 of 2 KIIs with national stakeholders, 1 of 5 KIIs with regional medical officials, 1 of 14 KIIs with district medical officials): 8 FY17 Q4 & Annual Report_Sumitted.pdf (page 13) 19 “The government has increased the health budget, but the priority to people living with HIV in its budget has not been that much...a circular says 10 percent of district revenues should go to vulnerable groups and like women, disabled, people living with HIV. And here we are also struggling to say that some of that amount should go to people living with HIV/AIDS, but still we haven’t achieved.” [KII, National stakeholder] “The government invests much on mothers, children and elders which use a lot of money in such a way that we lack funds which supposed to be directed to KVP.” [KII, Regional medical official] “All districts in the country lack money because internal revenues are not enough, we depend much from donors. It is difficult to allocate money for AIDS and KP activities.” [KII, District medical official] A KII with a GoT representative explained that funding for people living with HIV must be a national priority in order to get funding at the district level. Given the GoT’s unease with being associated with homosexuality, it is unlikely to receive prioritization. “Let us say the [national] government says we want schools as a priority. Once the government says that, all will follow that. If a person is being told by his/her boss to make sure that there are more desks are available in schools or laboratories, and you tell him/her to put AIDS activities in the budget... This becomes so difficult. So, the challenge we get is once [national] governmental priorities are set, each level of government will follow them.” [KII, GoT] Shortage of Facility Staff A variety of stakeholders report that facilities treating KVPs face staff shortages (1of 5 KIIs with regional medical officials, 3 of 7 KIIs with facility level staff, 3 KIIs with district medical officials). As several individuals explained, this can lead to long wait times and KVPs deciding to forgo treatment: “One of the key priorities of clients is to get special attention and quick service without the involvement of hospital queue during service provision. Well unfortunately, sometimes there is a shortage of staffs in CTC here, therefore whenever there is shortage, these KVPs are discouraged to wait for the service, because they also have other activities to attend to, they can’t afford to miss their activities, therefore, they may postpone clinical services by deciding to leave or find some other time to visit the clinic which may be late than how their medical schedule is supposed to be considered.” [KII, Facility staff] “When the nurse is occupied by let’s say three cases the client from the testing point has to wait and those client are clients that need immediate attention when they are kept waiting for too long they might change their mind some do not want to be seen around so they might leave.” [KII, Facility staff] Mobility of KVPs Various stakeholders reported the mobile tendencies of KVPs make it difficult to track individuals and provide ongoing services and treatment (1 of 4 KIIs with Sauti Consortium, 2 of 6 KIIs with local CSOs, 1 of 14 KIIs with district medical officials, 1 of 5 KIIs with regional medical officials, 1 of 6 FGDs with CBHSP, 1 of 7 KII with facility-level staff): “These key populations are money-oriented, and they are mobile. One time they might be at a certain place, but when the parliamentary sessions commence you will find them in Dodoma, when lumbering season commence you will find them in Iringa, when minerals are discovered in a particular region, let’s say Tanga, then you will find them in Tanga. So, you might think of a group and go for outreach and never find anyone because they are always moving, they have no boundaries.” [KII, Regional medical official] Most of sex workers works on bars whereby they are very mobile, you may start giving service today but after few days you find is not there. They are always circulating, today they come here, and next time they go to another area, so continuation of service become very difficult and even tracking become difficult because they move without leaving any information.” [KII, District medical official] 20 These KVPs are unlikely to provide real contact information to providers because they may be tracked down by authorities seeking to do them harm. KVPs fear of authorities FSW and MSM were asked in FGDs if they were ever fearful of the authorities such as the police. The majority of FSW stated in FGDs that they were afraid of mistreatment by the police, including being beaten and required to perform sexual acts (5 of 8 FGDs with FSW). One FSW shared an example of police brutality that occurred during Sauti testing: “We were once in one of the brothels being tested. During testing, there was one madam, one brother, and two counselors with their SAUTI car. Suddenly police appeared, people were heavily beaten and harassed but the Madam and those providers showed their IDs to explain them that they were from SAUTI Project for testing the FSW.” MSM also reported being fearful of being arrested by police (5 of 5 FGDs with MSM). As one MSM reported: “We fear in a way that, if you have been arrested and at the same time police and other people inside there come to realize that you are an MSM, you will be harassed very much. It happened to me when I was arrested. Some of the police physically abused me; some ask for sex with you and others ask for money.” This fear of police has negative implications as it has kept some MSM and FSW from reporting crimes, including intimate partner violence (1 of 5 FGDs with MSM, 2 of 8 FGDs with FSW). In one FGD with MSM, a man reported that MSM will not go to Sauti services if they are located near police stations. Sauti’s work with police Local CSOs implementing the Sauti activity reported working directly with police, especially gender desk officers (5 of 6 KIIs with local CSOs). One local CSO explained why their group started interacting with police: “When we started working with FSW there were some arrests and rounded up in Temeke and we went and negotiated with the police and had them released. That’s when we started meeting with the police gender desks.” However, this same CSO noted a continuing challenge in that “police get transferred and then there are new personnel who we have to be sensitize. There’s a lot of turn over and we have to resensitize new staff.” In FGDs, MSM and FSW reported instances of Sauti assisting them in dealing with authorities (3 of 8 FGDs with FSW, 3 of 5 FGDs with MSM): “SAUTI Project has given us the awareness on the legal matters. They have trained us what to do in case you get a problem concerning our business. They have also created awareness to authorities and other stakeholders like police that they must treat us like other people. So, after SAUTI I can say the harassment and harshness from police have decreased compared to before SAUTI.” [FGD, MSM] “I once encountered a GBV. One of the SAUTI peer educators encouraged me to report the incident at police station and how to present my statement. It worked out and I got help promptly.” [FGD, FSW] “Sometimes police have been forcing us to sex without condoms and they don’t pay you for that. They threaten that they are police and they will send us to prison if we don’t consent to their needs. After the coming of SAUTI, we have been reporting those issues, at least now the incidences have decreased.” [FGD, FSW] 4.3.2 What did USAID do that supported the Project effectively? Sauti Consortium members were asked to comment on the effectiveness of USAID’s oversight of the Project, including whether USAID’s oversight assisted or hindered the achievement of the activity’s results. Of the four Sauti Consortium members interviewed in KIIs, two had only positive things to say. One consortium member noted that the program was well designed, while another commented on the good communication between USAID and the consortium. “USAID has smartly designed this program. They crafted this program so that we look for evidence base to make decisions and define the package of services we provide.” [KII, Sauti Consortium] 21 “I would say yes because there are frequent communications between USAID and the leading partner JHPIEGO and this communication is trickled down to other partners. So technically yes, USAID has contributed in ensuring that the goals of SAUTI are being made.” [KII, Sauti Consortium] A third consortium member commented that the geographic changes in the Project’s activities have created challenges with local governments: “It gets political when they don’t understand that we are depending on the donor. It would be better not to be in the middle and have USAID communicate these challenges. In most cases, adaptation is more top down. They get orders from OGAC and then they give justification based on data. They want the partners to start implementing something new without a lot of preparation. Sometimes they request reprogramming money when they don’t see the value added…we have some of these changes reported to us in our coordination meetings and through our COP. Sometimes they provide informational letters.” [KII, Sauti Consortium] 4.3.3 What could USAID do to better support the Project to improve its effectiveness? Sauti Consortium members were asked to comment on how USAID’s contribution to Project effectiveness could be strengthened. Create transition plans for FSWs and other KPs in case support is cut off Given the potential for Project activities to be disrupted at any time by the government, one male consortium member commented on the need for transition planning: “In year 1 we had a focus on AGYW and KP, we started mobilizing beneficiaries, but by the end of year 1 we were told to just focus on AGYW and drop FSW for the economic strengthening. Some continued but some dropped out. Some were transitioned to Kizazi Kipya or became AGYW, but those over age 24 were not reached. Transition planning was not possible. Some of the groups are fragmented because of not having a proper transition plan from USAID. The rapid decision and cut of support was the reason for this…the groups are no longer intact and must be started again.” Communicate programming changes directly to local governments As mentioned in EQ 3.3, one consortium member requested that USAID communicate the reasons for geographic changes in programming directly to local governments. Conclusions on Factors Enabling or Constraining Performance Sauti’s outstanding feature is its intensive, data-driven adaptive management. This allows it to pursue targets effectively, and to be responsive to implementation challenges in real time. It also supplements the Project’s ability to ensure quality and implementation fidelity through its sub-recipients. However, the latter implementation imperatives require more than intensive monitoring. Quality and fidelity depend on training, equipping and continuous quality improvement through supportive supervision. The Project does provide these; nevertheless, there is some evidence of faltering implementation in some instances by some sub-recipients, and gaming of results, as might be expected in a relentless pursuit of targets. Certain implementation modalities - such as the use of peer educators for entry into KVP networks and escorted referrals to alleviate fear of stigma, reaching out to KVPs where they gather, and incentivizing health seeking behavior by packaging biomedical services with offerings that align with KVP priorities - are also proving key to Project success. Sustaining facilitated access after SAUTI or independent of a follow￾on will be problematic. As importantly, the performance of peer educators and other cadres needs to be appropriately incentivized and carefully managed to mitigate any risks to their intrinsic motivation. Implementation modalities are also key. Bringing community-based services to KPs, at locations they frequent, is an essential first step in cultivating facility access. The demands of implementing PEPFAR funded activities are substantial, driven by the urgency to achieve epidemic control. Sauti has proven equal to the burdens imposed by geographic shifts, target increases, and the required extensions of service offerings. According to Sauti, USAID has proven to be supportive 22 in the Project’s efforts to be responsive to PEPFAR’s imperatives and pivots. However, there is justification for USAID to assume a more active role in managing the relationships with PEPFAR stakeholders in￾country, government entities in particular, when shifts in programmatic demands require managing expectations and relational risks. Sauti has proven exceptionally adept at responding to the difficult operating environment created by the politicization of KVP issues. Similar events with regards to family planning demonstrate that the political risks to services remain, but Sauti’s record of response suggests that the Project is equipped to navigate the unpredictable environment. Law enforcement remains a hurdle to effective program implementation. Fear of arrest and mistreatment makes beneficiaries unlikely to report crimes against them and keeps some KVPs from seeking treatment near police stations. This is of particular concern now that KVPs must procure services in centralized facilities rather than in drop in centers located in their own communities. Major constraints to Project implementation stem from anti-MSM attitudes held by GoT, including lack of funding for PLHIV and lack of recognition of MSM. Other constraints to Project effectiveness include difficulty tracking KVPs and limited staff in facilities. 4.4 EQ4. WHAT ARE THE SAUTI PROJECT'S PROSPECTS FOR SUSTAINABILITY? (DESIGN, PLAN, IMPLEMENT, MONITOR, FINANCE) Findings on Prospects for Sustainability Sauti’s sustainability is supported by a long-term strategy for sustainability planning which was planned in from the inception of the program. Sauti has integrated the model for their 5-year sustainability plan embedded in the PEPFAR Country Operational Planning (COP) guidance adapted to local context. A corner stone of the plan has been engagement and memoranda of understanding (MOUs) with regional and district authorities to engage them in the implementation and encourage them to integrate Sauti practices, supply of health commodities and other resources in government facilities. MOUs were signed with district councils in Iringa, Njombe, Mbeya, Shinyanga, and Dar es Salaam. Sauti has been working with LGAs to institutionalize practices that support sustainability of KVP friendly services amidst a changing landscape for reaching KVPs. Planning and working directly with regional and district HIV committees has also led to some strengthening of KVP response. This planning has extended into Joint Implementation Plans with local LGAs which has improved GoT buy-in and commitment in councils with Sauti activities. Capacity building in the form of training of facility workers has translated to some facility staff practices. At the national level, prospects for sustainability include the publication of the second edition of the MOHCDGECs National Guidelines for HIV Interventions for Key Populations published in April of 2017, which builds off of the relevant 2014 guidance. At the national level this guidance has been supported through the NACP and TACAIDs who are reinforcing institutionalization across LGAs and local facilities. While the program’s adaptability to changes and structural strengthening and integration of HIV services over recent years give reason to be hopeful, there are several external structural and financial risks to sustaining KVP friendly services and the operationalization of the guidance at the local level encountered by the program (see timeline of events below). Most of these changes occurred in the first year of the program allowing the program to adapt course and find new solutions. For example, after the closure of resource drop in centers in February 2017, Sauti moved to a facilities-focused approach to capacity building to strengthen KVP services. This in the long￾term may prove to be more sustainable than the previous CSO supported model in that the clinics are financed by the government and may continue to receive government support. Adaptive management on the part of the Project increases prospects of sustainability amidst ongoing risks. Declining GoT Commitment and Reliance on Donor Funding 23 While the national trends and guidance has contributed to institutional and budgetary commitments, gaps remain in the GoT’s contribution to KVP and other HIV services versus those contributed by the donor community. This issue is not unique to KVP services but is a factor for HIV intervention overall. The health sector was allocated Tanzanian Shillings (TSh) 2.22 trillion in Fiscal Year (FY) 2017/2018. This represents a 34 percent nominal increase on FY 2016/2017 or a 28 percent increase adjusted for inflation. The health budget accounts for 7 percent of the national budget and 1.8 percent of gross domestic product (GDP). While accounting for nearly 10 percent of the total budget in FY 2013/2014, it was reduced to 7 percent in FY 2017/2018. According to the UNICEF Health Budget Brief, donor contributions to the approved budget have averaged 65 percent, with a large increase expected in FY 2017/2018 and were expected to contribute 57 percent to the health sector development budget in FY 2017/2018. According to PEPFAR, there has been a decline of HIV and AIDS financial resources since 2009. (NMSF III Page 32) According to PEPFAR’s 2018 assessment, PEPFAR/T and the Global Fund to fight AIDS, Tuberculosis, and Malaria (GFATM) are the two largest donors, contributing 98.8 percent of all public financing, according to the 2013-2014 Public Expenditure Review (2015) for HIV and AIDS. Moreover, the share of health sector (all areas of health) budget as a proportion of the total GoT budget has declined from 10.5 percent in 2010/11 to approximately 7 percent in 2017/18. The downward trend in proportion of the GoT budget going to health is concerning; the lack of funding for service delivery, Human Resources for Health (HRH), and information systems could negatively affect the country’s ability to achieve and sustain epidemic control. PEPFAR/T’s investments in short term solutions, such as HRH remuneration, as well as longer term solutions, will enable Tanzania to overcome these challenges and will ensure continued movement toward epidemic control. The United Nations (UN) and World Bank provide a small proportion of additional funding. There are currently no other bilateral donors anticipated in FY 2019. (PEPFAR/Tanzania, April 2018) Reliance on donor support in HIV prevention, treatment and care also shows a lack of commitment to KVP targeted funding. An official from a national agency describes, “The resources have been likely coming from outside Tanzania…If you take out the infrastructure, human resources I would say over 97 percent the resources are coming from outside Tanzania, where USAID, PEPFAR and other bilateral partners. Now the challenge of sustainability or self￾reliance as USAID are calling it now...That has been part of the advocacy for many years, and it is still more or less the same. Some few interventions are going on see you might have heard about the AIDS Trust Fund, but it is not improving much. More many funds are coming there but the Tanzania Commission for AIDS (TACAIDS), PEPFAR are looking into other means of revitalizing and putting up new avenues through which individuals and other corporations can contribute funds, but that hasn’t changed much really.” According to PEPFAR, the government’s contribution to the HIV and AIDS response declined by 50percent from 22 Billion (Year 2006/7) to 11 Billion (2010/11) or 33 percent of the actual value of 2006, if inflation is factored in. Around 95 percent of all HIV funds come from international donors, whereas 5 percent came from government. The government’s contribution does not factor in in-kind or indirect costs such as infrastructure, human resources, transport and health system at large. This caused the calls for government to undertake the study to determine government contribution. The heavy reliance on external sources presents challenges in terms of sustainability. At the same time, stability in the level of external funding is a compromise to ensure the medium-term continuity of service provision (NMSF III Page 57). In the local LGAs the ET visited, 1 of 5 KIIs with RMOs said that funds that are meant to be directed toward KVP programming are lacking, as the government heavily invests in mothers, children, and the elderly instead. One (1) of 2 KIIs with national stakeholders noted this as well, describing the struggle involved persuading the national and regional government to prioritize people with HIV/AIDS in their budgets. While GoT has contributed to health commodities for treatment and care, there were shortages 24 of preventative commodities such as condoms in some facilities and a lack of awareness or implementation of PREP for prevention among high risk KVPs. PEPFAR and international donors play the central role in funding HIV epidemic control in Tanzania, and it is clear that an increase in GoT commitment is needed. However, it is clear also that ongoing commitment from international donors will be required to support the health system through the transition to self￾reliance; ongoing support for KVP targeted HIV intervention is no exception. Financial Sustainability- Local Resources for KVP Service Delivery Regional and district authorities said that they rely on funds for KVPs from sources such as Sauti and PEPFAR [KII of 5 with RMOs and 2 of 14 KIIs with DMOs]. One (1) of 2 KIIs with national stakeholders noted that there is a recognized need at the government level to transition towards self-reliance and away from dependence on donor funding, but that there has not been much urgency in developing other solutions for providing support to KVPs. From the perspective of the Sauti Project stakeholders, risks to the continued implementation of the Project including the ability to conduct outreach through peer to peer networks, which is compounded by concerns about lack of funds for transportation to provide services or reimburse travel (2 of 7 KIIs with facility staff, 2 of 6 KIIs with local CSOs, 2 of 14 KIIs with DMOs, 1 of 6 FGDs with CHBSPs). One volunteer detailed her concern about being able to provide services for beneficiaries residing further away without the financial support provided by the Project for outreach: If SAUTI ends, then it is clear that I will not be able to reach out to such a long distant village due to lack of transport. For example, I am now being paid little allowance for outreach services at Mdibila, I am afraid these will not sustain after SAUTI Project phases out. [FGD, CHBSP/Empowerment Workers 27] Human Resource Challenges and Incentives for Volunteers in GOT Facilities for CSOs Human resources challenges limit institutional readiness in government facilities and limit long-term sustainability. Staff shortages are mentioned as being a likely issue that would limit service efficacy (1 of 14 KIIs with DMOs), in part because of a common concern that staff retention after Project exit would fall due to funding issues affecting the organization hiring workers such as nurses (1 of 10 FGDs with FSW). Seeing that staff turnover is currently being recognized as a problem already (1 of 4 KIIs with Sauti consortium members), it would be useful to further explore the common reasons staff have for leaving. CSOs also mentioned depending on Sauti for the stipends and travel reimbursements provided to volunteers. They also mentioned that they were under capacity to meet the demand for certain types of service delivery, including approaches to prevention. CSO staff discussed that health commodities and human resources were needed to cover the range of services in the provision of PREP for non-positive KVPs who are at highest risk of contracting the virus. As one CSO staff describes, the supply of PREP does meet the demand created through awareness among these populations. As one implementing CSO staff member revealed, “GOT facilities are not aware of prep. They are given other messages and are discouraged from using prep. There some confusion that ARV was given to not HIV positive clients when it was actually prep. -FSW demand is very high…. We have limited resources. The more they trust us the more demand. I have more than 30 groups of 30 people waiting for PREP, but we can’t start them because there’s no drugs available. We can reach 900 beneficiaries who are waiting for services in Temeke. In Kindoni there are more than 50 groups who are waiting for prep and can’t get it.” More GOT awareness of the value of PREP and HIV treatment options along with the improved supply of these methods in GOT facilities will be required to sustain and bring to scale these efforts. 25 State of Institutionalization of Practices and Procedures by GoT and Local Organizations CSO Organizational Capacity Building There is evidence that improved capacity has been built among CSOs and facility workers trained by Sauti to provide effective service to KVPs. This according to some CSO staff has contributed to organizational development in the form of SOPs, and financial management. As a CSO staff member states, “… when [the organization] started implementing the Project there were no specific policy on the ground, but they significantly developed policies and guideline on KVPs. Furthermore, the capacity building from SAUTI through PACT has improved well the organizational development even in Project documentation. We had no, for example, Child Protection Policy, and other specific policies. We have changed a lot and improved on financial documentation too. We were following government policies” [KII, Local CSO 36] CSO subcontractors’ early signs of organizational development in implementing Sauti may be a contributing factor in the future sustainability of Sauti practices, bearing in mind ongoing needs for linkages to donors and local governments for financial support to sustain operations and human resources. As one CSO partner stated, I have people who I have taught in Katani, the knowledge they have received cannot just disappear in their mind after the Project is gone, it’s something that they will always live with, and since I am still available hence, we will keep helping each other in lifetime...SAUTI must link these groups to the DED or RAS and emphasize these people to encourage the community to continue dealing in these groups since they will be ending the Project. Partnerships Built Between Sauti CSOs and Local Government Authorities CSOs also described satisfactory relationships with their LGAs in some cases, which has improved their ability to link clients to treatment and care services available at GOT facilities. In other cases, Sauti has helped KVPs receive recognition by government. Local group registration is considered important for their sustainability. First, a good linkage that we have with the government... So, we expect that even after SAUTI Project ends, these services will continue to be delivered because all facilities we were using belonged to the government and so they can continue providing these health services… Also, when you look at biomedical, when we are teaching them, we normally provide brochures and as you know these booklets last for long and so they can keep on reading.” [KII Local CSO 30] Another factor of importance for CSO sustainability is their establishment as a registered organization, a step toward recognition and support from their respective LGAs. Several organizations appear to be registered and building relationships with their LGA counterparts. “…Regarding the government, what… [the] government and SAUTI did was to register all groups under the SAUTI Project. This is very important to ensure that all groups are recognized by the government, all groups [have a] certificate which [is] recognized by the district council. Therefore, even if the Project ends, the groups will remain strong as they are recognized by the government and they can be able to get any assistance from the LGA [that] they might need.” [FGD, CBHSPs/Empowerment Workers 3] CSOs appear to be cascading their capacity and learning, which also may improve prospects for sustainability. On a local level, focus group respondents indicated a strong commitment to sharing the education they received through Sauti to others in their communities. Nearly every type of focus group made mention of their intent or ability to disseminate what they had learned via Sauti programming (4 of 10 FGDs with vAGYW, 2 of 10 FGDs with FSW, and 3 of 6 FGDs with CBHSPs). The focus group respondent below emphasized her cohort’s commitment to educating future generations: Yes, the changes will prevail because the changes will remain there. We are people of the same generation/age throughout these five years of the Project’s education. The younger ones who receive this 26 education will be like shield of the SAUTI Project. Others when they give birth to their offspring’s they will educate them with education they receive from SAUTI Project.” [FGD, FSW 31] Challenges to GOT commitment to KVP Services Despite the array of benefits noted by respondents, government commitment to continuing Sauti services to KVP is in question. Lack of confidence in the government to continue support for Sauti programming is evident in the following quote from a staff member in the Sauti consortium; “Sustainability depends on the partners and GOT. Usually the government would be the custodian of our intervention, but the fact that the administration is not in favor with our intervention, I’m not confident the government will take over and implement in the case Sauti is not in the equation. GOT personnel are reluctant to be associated with our programming.” [KII, Sauti Consortium 46] Respondents from 2 of 5 FGDs with MSM noted that they are worried that services will become difficult for them to access once the Project leaves, and 1 of 10 FGDs with FSWs expressed concern that GBV will increase after Project phase out. Another FGD respondent from 1 of 10 FGDs with FSWs worried that the situation will worsen once Sauti’s activity period ends: “For me I think if this Project ends there will be some consequences because there will be no more education and therefore future victims will not have an opportunity to be provided by these services. So, if the Project ends, gender-based violence will increase, early marriages, and unexpected pregnancies, unprotected sex etc. We wish this Project would continue.” [FGD, FSW 5] Bearing in mind the ongoing need for KVP services and the long road to decreasing the prevalence of HIV across the country, sustaining services to those at greatest risk of contraction is critical. Other sustainability challenges When asked about the future of their activities, local institutions and organizations see themselves taking an active role in developing and managing a sustainability plan to ensure continuation of their services. While Sauti appears to be rolling out this process, not all district LGAs in Sauti program sites have begun this process. In KIIs with local CSOs (2 of 6) and DMOs (3 of 14) they expressed a desire for a sustainability plan, including one that covers staff training, to be in place, as they did not seem to be aware of any being currently developed. That said, DMOs (2 out of 14) whose councils are currently involved with sustainability planning for their activities seem to express an appreciation for their involvement, and display confidence that their services will continue after Sauti’s exit. The intention to fully execute this plan in LGAs is critical for long term sustainability. There are issues with the incentive system for facility-based health providers. One DMO/CMO noted that in their council, the “payment for results” approach has led to compliance and data quality issues that do not actually enhance results achieved. The solution to this, in their words, is better procedures, guidelines and check and balances, which may enhance the prospects for sustainable results. In 1 of 3 KIIs with GoT and 1 of 14 KIIs with DMOs they noted that more ownership of the Sauti activities was needed at the facility level. A CMO noted the problems that could arise if facility workers do not feel a sense of commitment to proper practices of the program, which can turn drastically affect how effectively the program serves its beneficiaries: They should engage these facilities-based health providers so that they can have a feeling that they are part of the Project. The sensitivity for targets and payments are created by supervisors saying that they want positive results. If CBHSPs and the HIV testing workers work for purpose of positive results only, it will reach a time when they will do forgery so that they can get positive results and once they are paid that is it. We need to be aware of that. Payments are made for those [who] succeeded [in getting] positive results. This discourages others and that sense of ownership drops. We should not try to influence results. 27 There have been many complaints about them forcing for positive results. We need to work under procedures by doing frequent evaluation meetings to find out why we are not reaching targets and revise the approach to adapt for sustainability and to leave a good legacy behind.” [KII, CMO 32] Other CSO and LGA respondents indicated a need for an improved beneficiary tracking mechanism to better render services and maintain up-to-date information between different service providers (1 of 14 KIIs with DMOs, 1 of 6 KIIs with CSOs, 1 of 3 KIIs with the GoT)9. One DMO said, in regard to tracking clients, that “it is difficult to solve this challenge but if we could have common network as a region, we could track them easily or else if there is cooperation between organizations example DREAMS and SAUTI would help. Finally, if we had an ability of finger prints which is connected in a system we would know if someone has already started a service somewhere or not yet” [1 of 14 KIIs with DMOs-DMO 54]. There is a demonstrated need for increased financial capacity amongst CSOs in order for program activities to be effectively sustained after Sauti’s exit. Respondents reported a need for CSOs to be trained in how to solicit funding, which would include instruction on how to write Project proposals [1 of 6 KIIs with local CSOs and 1 of 3 KIIs with GoT]. Sauti consortium partners noted that CSOs lack other sources of financing and are currently learning how to diversify their funding sources [1 of 4 KIIs with Sauti consortium members]. Conclusions on Prospects for Sustainability 1. Bearing in mind the prevalence of HIV nationally and the need for HIV prevention, treatment and care as well as KVP targeted services, there is a long road to decreasing the prevalence of HIV across the country; sustaining services to those at greatest risk of contraction is critical. Results will not be made without continued intervention and donor support at this time. 2. Following structural changes occurring in the first years of the program, the program was able to adapt course and find new solutions. Adaptive management on the part of the Project increases prospects of sustainability amidst ongoing risks. 3. CSOs confirm that the knowledge, skills and modalities of service delivery cultivated through participation in SAUTI are factors that strengthen prospects for sustaining services post-activity. However, financing the implementation of activities will be the key risk factor to address. 4. Despite a challenging political climate, KVP centered services are institutionalized in guidelines, job aids and tools such as registers and forms. From the national to the LGA levels and GoT facilities, there is a recognition that targeting KPs is key to controlling the epidemic. 5. Practices for testing, treatment and care are becoming more institutionalized and PEPFAR supported activities are integrated into planning, all the way down to local level. However, dependence on PEPFAR for human resources, technical assistance and prevention and treatment supply remains a barrier. 6. Financial sustainability is lacking in service provision for KVPs. There is more work to be done to ensure government support for sustained HIV/AIDS services and for organizations providing these services to become financially capable of providing services without continued donor funding in the near and intermediate term. 7. GoT is dependent on donor funds for continued service delivery, and routine operations such as supportive supervision and data management. There is still a major imbalance between what donor's contribute and the proportion committed by GoT. The strategy that appears to improve 9 1 of 5 KIIs with RMOs, 1 of 7 KIIs with facility staff, 1 of 6 FGDs with CBHSPs/Empowerment workers, and 1 of 14 KIIs with DMOs noted in particular the difficulty of tracking beneficiaries. 28 the actualization of GoT commitment is decentralization of service provision (provided funding is disbursed from central government). 8. Operational challenges for GoT facilities and CSOs: Further resources are needed in facilities to ensure that human resources are in pace for KVP friendly services; CSOs need financial support to provide the continued incentives for volunteers such as peer educators to continue their work. Stipends, transportation allowances and other material support incentivize the efforts of volunteer cadres. Without these incentives sustained implementation is at risk. 9. This evaluation concurs with PEPFAR’s 2018 assessment that Domestic Resource Mobilization and Technical and Allocative Efficiencies, is currently unsustainable, meaning that Tanzania does not adequately generate the necessary financial resources for HIV and AIDS, to ensure sufficient resource commitments, and use data to strategically allocate funding and maximize investments. (PEPFAR/Tanzania, April 2018) Bearing in mind other structural challenges, the picture for KVP specific intervention is even more vulnerable without ongoing international donor support. 4.5 EQ5. WHAT CRITICAL LESSONS HAVE BEEN LEARNED THAT USAID SHOULD CONSIDER IF DESIGNING A FOLLOW-ON AWARD TO THE SAUTI PROJECT? Findings on Lessons Learned and Adaptive Management This section revisits findings discussed under the previous evaluation questions to distil key lessons on factors enabling or constraining project performance and the implications of those lessons for project design and adaptive management. a. What are the critical lessons learned in terms of design, implementation, capacity & operational issues, context-based constraints, sustainability and any other emerging factors that should inform the design of follow-on or similar activities? Sauti is able to perform against demanding targets at scale because it is designed to be an intensely data driven, adaptive activity. The data driven management model consists of the following elements: • A model of performance based on an assessment of the basic resources required to achieve a result. For example, Sauti has estimated how many clients can be served by a volunteer in a day and the associated cost. The project’s monthly planning is informed by these basic resources to result ratios. • Sauti devises annual performance plans in advance, based on PEPFAR assigned targets, and informed by the resources to result ratios. Targets are assigned to sub-recipients based on the long-term advanced planning, matched to required resources, and distributed across the performance days available. • Daily targets are assigned in the way described and can therefore be tracked on a daily basis. The daily reporting, weekly consolidation, monthly planning and issuing of guidance cycle to sub-recipients based on their performance against target keep the massive project on track. • Monitoring data is complemented by regular beneficiary engagements to learn what the indicator data cannot reveal about implementation. • Beneficiary engagements and monitoring data are complemented by substantive research projects that fulfil a learning agenda that cannot be satisfied without more substantive data. • Performance deficits are rapidly identified and diagnosed by utilizing the gathered data, hypotheses generated and tested, solutions devised, implemented, monitored and adjusted as necessary, in short learning loops focused on continuous performance improvement. 29 Sauti is able to serve hard to reach populations because it implements through a handful of effective modalities. These modalities enable the project to access its target beneficiaries and its beneficiaries to access services with ease. • By mobilizing peer educators, Sauti can take advantage of their credibility with target populations and knowledge of key population networks to access their intended beneficiaries. • Having referrals escorted by peer educators is an essential, effective practice representative of active facilitated access that substantially eases linkages of key populations to services. • Sauti incentivizes participation by offering packages of services combining offerings that align with client priorities (e.g. STI screening), while incorporating PEPFAR priorities (e.g. HTS). The more comprehensive the package of services (such as in DREAMS districts) the more tacit benefits reported among beneficiaries. In addition to measured behavior change results, there is a substantial empowerment dividend that is more pronounced with more interventions. Examples include reported improvements in parenting, reconciliation in families, a sense of self-worth and esteem, the assumption of leadership roles. 5.3 What adaptive management has already been done in light of lessons learned and are these adaptations proving effective? Mobilising a creative response to a deteriorating operating environment in which KVP and FP issues are being politicized resulted in adaptations that are proving more effective than previous practices. • Sauti and partners have adapted to a changing context where formal facilities have taken the place of drop-in centers. Initially there was some lag in access to services due to this change but ultimately Sauti has been able to effectively work with GOT facilities by sensitizing them to KP friendly treatment care practices. This also may improve prospects for sustainability as facilities take on these roles. • Framing the case for KVP intervention in terms of epidemic control helps defuse resistance in an environment where KVP services are highly politicized. NIMR partnership has helped legitimize this. There is a spectrum of awareness across national, regional and local stakeholders. 5.0 RECOMMENDATIONS Project Performance Refine the Projects’ tracking of service layering. Intensive, data driven adaptive management is the key practice to which high or over-achievement of targets is attributable. However, the data quality issues associated with an ineffective unique identifier undermines the Project’s ability to manage implementation fidelity and quality inherent in the provision of multiple services to each beneficiary. It is also challenging to ascertain what combination of services work for which category of KVP and under what circumstances. The correcting of this deficit is the next and crucial step in strengthening the Projects capacity to track all aspects of its performance. Devise a strategy to improve disclosures of HIV self-testing results. Sauti is not expected to utilize self-testing as a case finding modality. However, given the urgent need to address the testing/treatment gap, devising ways to exploit the credibility of peer educators to encourage follow up, respectful disclosure, escorted re-testing and linkage to treatment seems to offer a feasible supplement to current modalities. 30 Workforce Management Intensify continuous quality improvement efforts to ensure sub-recipients are implementing with fidelity. Sauti already manages sub-recipient performance very closely when it comes to achieving targets. There is some evidence from the field that intervention quality, however, is not consistent across all sub-recipients. Revisiting and investing more in the QI strategy, with a view to strengthening Project fidelity, will serve to strengthen outcomes, beyond target achievement. Review the work load and incentives structure for peer educators to eliminate the testing/treatment gap. The analysis to diagnose the underperformance on linkage rate revealed that the leading cause of the performance deficit is directly linked to the productivity of peer educators. However, there may be hidden causes, such as gaming of the system by peer educators who in some instances are incentivized to find new positives. Reviewing the workload, the dynamics of the incentive structure, and redesigning work for this cadre is likely to prove crucial in addressing the underperformance on this process, central to epidemic control. USAID Support Explore a more substantive communication and relationship management role for USAID supporting implementing partners in a challenging implementation environment. For Projects such as Sauti effective performance in challenging environments depends significantly on preserving good relationships with GoT and other stakeholders. Complying with PEPFAR directives frequently jeopardizes these relationships, as the implementing partner is required to withdraw efforts from geographic locations or reduce services abruptly and redirect resources. There is justification for USAID to assume responsibility with stakeholders for the related fallout and preserve the essential social capital of the IP. Actively pursue data sharing between care & treatment and prevention IPs to improve case finding and client tracking. Data sharing between IPs is a critical support to realizing epidemic control goals. However, the difficulties in securing these arrangements require authority and diplomatic credibility, USAID can assume a more active role in intervening to secure the necessary concessions to realize this important outcome. Sustainability USAID should facilitate the development of joint self-reliance strategies across complementary implementing partners and technical offices. While Sauti’s activities integrate self-reliance objectives through institutions support and capacity building, the ability to fund interventions without donor support is the key deficit. USAID should advocate for greater financial commitment from GoT and the international donor community towards HIV and KVP focused interventions. Also closer collaboration with other activities, such as HP+ in Tanzania, has the potential to strengthen sustainability efforts that address the funding gap, in terms of advocacy or more concrete solutions at LGA level. Project Design Identify and require the inclusion of effective practices in the development of Project design documents for key populations focused activities. Sauti has provided invaluable lessons for strong Project performance. Expectations on intensive data driven adaptive management, workforce management and successful implementation modalities need to be explicit in requests for applications. 31 ANNEXES 32 ANNEX I: EVALUATION STATEMENT OF WORK 33 Background of Project/program/intervention: The Sauti Project, awarded by USAID to Jhpiego an affiliate of Johns Hopkins University and partners EngenderHealth, Pact and the National Institute for Medical Research (NIMR) Mwanza, seeks to contribute to the improved health status of all Tanzanians through a sustained reduction in new HIV infections in support of the Government of Tanzania’s (GOT) commitment to HIV prevention. The Sauti Project aims to introduce new innovations and enhance existing strategies for combination HIV prevention, positive health, dignity and prevention (PHDP), and family planning (FP) services for key and vulnerable populations (KVP). At the end of five years, Sauti Project’s goal is to have all KVP in Project communities able to actively access a core package of vulnerability-tailored, high quality, client- and community-centered prevention services, combining biomedical, behavioral and structural interventions. These include strong and traceable linkages to care, treatment and other referral services, that are being developed with the active support and participation of KVP, their partners, families, and health providers, as well as the wider community, GOT agencies, and the private sector. In FY 2018, Sauti integrated into its service delivery platforms Community antiretroviral therapy (ART), Pre￾Exposure Prophylaxis (PrEP), and HIV self-testing (HIVST) demonstration pilots in select regions. Sauti Project directly contribute to the actualization of the National Guideline for a Comprehensive Package of HIV Interventions for Key and Vulnerable Populations (2017) as well as the GOT’s policies and guidelines for FP and other relevant health areas. HIV prevalence in the United Republic of Tanzania (URT) has decreased from 7 percent in 2003/4 to 5.1 percent in 2011/12 (Tanzania Commission for AIDS/Zanzibar AIDS Commission [TACAIDS/ZAC], Tanzania National HIV and Malaria Indicator Survey, 2011/2012). The ongoing scale-up of the national Voluntary Medical Male Circumcision (VMMC) program, and an increasing modern contraceptive prevalence rate (CPR) from 17 percent in 1999 to 32 percent in 2015 (Demographic and Health Survey [DHS]), are just a few of the successes contributing to this decrease. However, in hotspots across the country, HIV incidence and prevalence remain unacceptably high, with the achievements made in the general population not translating to progress for all, particularly key and vulnerable groups. Tanzania is facing a heterogeneous HIV epidemic in which key populations (FSW and MSM) and vulnerable groups (vAGYW and PFSWs) are disproportionally affected and underserved by HIV and FP programs. When seeking health services, KVPs frequently report high levels of stigma and discrimination by health providers. Prevention among these populations is an important step to sustain and scale the gains Tanzania has achieved thus far. Definitions: Sauti Key and Vulnerable Populations (KVP) Key Populations (KP): female sex workers (FSW) and men who have sex with men (MSM). Vulnerable Populations: vulnerable adolescent girls and young women (vAGYW) aged 15-24 years, and partners of female sex workers (PFSW). Other vulnerable populations at increased risk of HIV acquisition and transmission (e.g. mobile men, men and women in transient work places, and high risk children aged 18 months – 14 years). Objectives The Sauti Project aims to achieve five interrelated objectives: Objective 1: Implement a package of core and expanded biomedical HIV prevention and FP interventions, with enhanced linkages to care, treatment, and support services. Objective 2: Deploy interventions designed to reduce individual risk behaviors and strengthen support for positive social norms and structures at the community level. Objective 3: Execute a robust research and learning agenda. Objective 4: Develop and implement capacity and sustainability building interventions. Objective 5: Build and deploy vigorous monitoring and evaluation systems. 34 The anticipated results over the five-year Sauti Project include: • Increased and timely use of HIV prevention and FP services • Improved positive behaviors and social norms at the individual and community levels • Reduced vulnerability of vAGYW through novel structural interventions • Increasingly sustainable comprehensive HIV prevention services for KVPs Guiding Principles Underlying the Project objectives are the following guiding principles, which the Sauti Project is incorporating into its programming approaches: • Meaningful engagement of local government authorities (LGAs), local civil society organizations (CSOs) and most importantly, the KVP community and people living with HIV (PLHIV) in the design and implementation of the Project • Providing high quality, client centered, and differentiated services that meet the needs of KVPs • Utilization of data and creation of new evidence and learning to inform effective programming • Fostering government ownership, accountability, and sustainability • Strengthened treatment cascades • Remaining nimble and responsive to the service delivery environment • Performance –based management and monitoring of the Project at all levels • Use of affordable technology to increase efficiencies Geographic Scope Sauti Project operate in 14 regions, including Arusha, Dar es Salaam, Dodoma, Iringa, Kilimanjaro, Manyara, Mbeya, Songwe, Morogoro, Mtwara and Njombe, Shinyanga, Singida, Songwe, and Tabora in accordance with the President’s Emergency Plan for AIDS Relief (PEPFAR) council level prioritization (see figure 1). 35 Figure 1: Map of Tanzania showing Sauti Geographical Coverage Partnerships and Collaborations Investing in synergistic and strategic partnerships is critical to amplify and sustain efforts in achieving epidemic control. From Project inception, and under the leadership of USAID and Sauti consortium partners, several of these partnerships have been forged to leverage additional funding and expertise, and to institutionalize long-term Project investments. These wide-ranging collaborations, directly linked to the Project’s objectives, are described in detail in subsequent sections and summarized in Table 1. Table 2: Strategic Partnerships/Special Initiatives Partner Description Current status Population Council (Project SOAR) Sauti Project and NIMR/Mwanza is collaborating with Population Council in implementing a community ART study for FSWs at CBHTC+ sites in Njombe. Sauti is receiving funding to support operations. This study is underway. Within 32 days of commencement, 220 FSW clients were initiated on ART. Sauti aims to utilize study results to advocate with the MOHCDGEC/ NACP to include this model in current service delivery approaches. 36 Partner Description Current status Population Council (FSW FP Study) In tandem with the above ongoing study, Population Council has been awarded another grant and is in the final stages of starting another study aimed at examining safer conception options for HIV positive FSWs. Sauti will receive funding to support operations. This study will contribute to refinement of Sauti core package of HIV-FP integrated services for KVPs focusing on FSWs. Bill & Melinda Gates Foundation (BMGF), through University of North Carolina (UNC)/Final Mile With support from BMGF, UNC and Final Mile will conduct qualitative research to gain a behavioral economics-based understanding of the short and long-term impact of cash transfer interventions (including reduction in compensated sex and intergenerational sex over time) among vAGWY. This qualitative work will build on Sauti Project’s “CARE” study, with no exchange of funds. Current ongoing. Findings will be used to inform Sauti’s cash transfer program, implemented using DREAMS funding (no new funding; this is a continuation of COP16 plan). Funding invested by BMGF will contribute to Sauti’s cost-share Bill & Melinda Gates Foundation (BMGF) through M4ID Under this collaboration, M4ID – a Finnish organization - will collaborate with Sauti Project to implement Human Centered Design aimed at developing strategies to increase vulnerable adolescent and young adult populations’ engagement with HIV testing and care. Sauti will receive funds for implementation. This collaboration is in planning phase; it is expected that the results of this operational research will help Sauti and other KVP stakeholders to refine strategies for reaching at risk AGYWs, and successfully linking HIV positive AGYWs to care and treatment. Funding invested by BMGF will contribute to Sauti’s cost-share Bill & Melinda Gates Foundation (BMGF) through support to College of William & Mary Sauti’s cash transfer platform is being used as part of a Behavioral Economics study. The study will seek to understand whether incentivizing mobile money as a financial saving instrument among vAGYW receiving cash transfers under Sauti improves savings amongst beneficiaries. This collaboration involves no exchange of funds. This study has begun; similar to the UNC/Final Mile study, these findings will inform the Sauti/DREAMS supported cash transfer program. Funding invested by BMGF will contribute to Sauti’s cost-share 37 Partner Description Current status TIGO (MIC) Sauti has established a partnership with TIGO, whereby TIGO is providing in￾kind donation of 12,000 feature phones for vAGYWs participating in cash transfer initiatives. Additionally, TIGO will field test a new mobile wallet application designed especially for savings groups, with Sauti Project’s WORTH+ groups. TIGO will push 3 million SMS messages, and the Project is in discussions about adding interactive voice recordings (IVR) in the near future. Enrolled vAGYWs have already received their phones. Currently, Sauti is waiting for the next consignment of 3,500 phones for use by WORTH+ loaning and saving groups. This initiative complements Sauti/DREAMS cash transfer program and creates an excellent cost-share opportunity for the Sauti Project Elton John AIDS Foundation (EJAF) The EJAF public private partnership with PEPFAR enhances access to sexually transmitted infections (STIs) treatment including periodic presumptive treatment (PPT) and strengthens KP-focused CSO capacity. EJAF program will continue until December 2018 with reprogramming of some activities to align with the new KVP guidelines. TOMS Shoes A public private partnership with TOMs Shoes - who will donate up to 90,000 pairs of shoes, worth more than US $2.2 million – emphasizes distribution to vAGYW as critical piece of our incentivized peer education program. Sauti continues to receive and distribute TOMS shoes. The shoe incetive is contributing to uptake in SBCC services and retention. This partnership provides a good cost￾share opportunity for the Project. International Labor Organization (ILO) Sauti Project and the ILO joined forces in Kyela district council for the delivery of economic strengthening activities. vAGYW in WORTH+ groups were trained in the ILO’s “Start Your Business” series. Revolving loan funds totaling $32,697 were disbursed in FY16. Sauti is advocating with ILO to extend this support to other non-DREAMS councils. This partnership provides a good cost-share opportunity for the Project. National Council of People Living with HIV (NACOPHA) This partnership is for “empowerment groups” for HIV-infected beneficiaries, to reduce internalized and anticipated stigma and increase ART retention. Sauti continues to refer PLHIVs to NACOPHA led support groups EQUIP Sauti partnered with the USAID funded EQUIP Project to developed standard operating procedures (SOPs) for the delivery of community-based ART. The SOAR study in Njombe are currently using the developed SOPs. The SOPs will be adapted for the COP- supported community ART demonstration Project, to be implemented under Sauti Project FY 18. 38 Partner Description Current status UCONNECT UCONNECT donated 100 refurbished Dell OptiPlex 740 Desktop systems, fully loaded with educational software, for distribution to the 24 Sauti Project KVP Drop-in Centers (DICs) in FY17. Our CSO partners will secure and maintain the computers. Since DICs were closed, these computers are being repurposed for use in safe spaces and other CSO spaces Hewlett Packard Enterprise Sauti Project’s partner Pact has signed an award with Hewlett Packard Enterprise. HP Enterprise will digitize community savings groups to build household and community resilience through E-Ledgers, and will partner with Sauti Project WORTH+ groups Design of activities under this partnership is still on going. SCOPE OF WORK A. Purpose: The main purpose of the performance evaluation is to: 1) Assess the effectiveness, efficiency and quality of the Sauti activity at the national, regional, and community-based service levels; identify implementation gaps/challenges; determine how well the Project is achieving its goals, objectives, and performance targets/results. 2) Propose key recommendations for improvement and direction for the remaining activity period. To document lessons learned and provide recommendations that will inform future programming directions for USAID’s key and priority populations HIV support. B. Audience? ● USAID health team ● National AIDS Control Program and Tanzania Commission for AIDS ● Local Governments and beneficiaries ● Sauti Project staff C. Applications and use: ● Findings and recommendations from this performance evaluation will be used for further improvement and direction for the remaining activity period. ● Conclusions from this evaluation will assist USAID in shaping the direction of future Project(s). D. Evaluation/Analytic Questions & Matrix: # Evaluation Question Evaluation Methods Application or Data Use 1 How effective is the project in achieving its goals, objectives, and performance targets? ● Document & data review ● Key informant interviews ● Secondary data analysis ● Focus Group Discussions ● Survey ● Feedback for course correction ● Recommendations for future project(s) 39 # Evaluation Question Evaluation Methods Application or Data Use 2 What are the project’s strengths, weaknesses, and gaps in planning, management, service delivery, and sustainability? ● Document & data review ● Key informant interviews ● Secondary data analysis ● Focus Group Discussions ● Survey ● Feedback for course correction ● Recommendations for future project(s) 3 What are the constraints to successful implementation of this program? ● Document & data review ● Key informant interviews ● Secondary data analysis ● Survey ● Feedback for course correction ● Recommendations for future project(s) 4 How well does the project align with USAID and PEPFAR global priorities and approaches? ● Document & data review ● Key informant interviews ● Survey ● Feedback for course correction ● Recommendations for future project(s) At the end of the evaluation, it is expected that the following recommendations will be provided to USAID/Tanzania Team: 1) Recommendations to build on strengths, correct weaknesses and improve implementation to enable USAID and implementing partner staff to develop a course of action for the remainder of the project. 2) Recommendations for best practices in KVP programming to address the epidemic that can be integrated into future programming. E. Methods: This evaluation will collect information about the implementation of Sauti to date, in providing quality and comprehensive HIV/AIDS prevention services, establishing linkages and referrals between facility and community services, technical assistance to local and national partners and CSOs, and challenges. This performance evaluation will assess the contribution of Jhpiego/Sauti ability to scale up its technical assistance in districts and regions to improve the quality of HIV/AIDS prevention alongside the clinical cascade, at the health center and community levels and capacity building at the national and district levels. Whenever possible, the evaluation should mention gaps in programming as well as innovations and successes, both of which could inform the design future project(s). The evaluation will also consider Sauti contributions to national technical leadership, including work with technical advisory groups, stakeholders, multilaterals, and presence at conferences, publications, and technical fora. Data Quality The qualitative and quantitative data used in this evaluation should meet the following five data quality standards in accordance to USAID’s Automated Directive System (ADS) 203: 1) Validity; 2) Integrity; 3) Precision; 4) Reliability; and 5) Timeliness. Limitations This is a performance evaluation conducted prior to the conclusion of a USAID-funded project; it is not intended to be a rigorous quasi-experimental or experimental design outcome or impact evaluation with predetermined counterfactual groups. It does not attempt to attribute change in health outcome or impact to the project itself. Document and Data Review This desk review will be used to provide background information on the project/program and will also provide data for analysis for this evaluation. Documents and data to be reviewed include: • Sauti Cooperative Agreement, including modification documents • Annual Reports 40 • Quarterly Reports • Information about initial country scoping visits as the project started up • Work plans, and sub-agreements as appropriate • Sauti monitoring and other internal reports • Sauti Monitoring and Evaluation data • HIV Cascade Assessments • National HIV/AIDS Strategy • UNAIDS Atlas • PEPFAR 3.0 “Controlling the Epidemic: Delivering on the Promise of an AIDS-free Generation” • Other relevant documents that may assist the evaluators, • Other sources, as needed Secondary analysis of existing data A thorough review of existing data, and descriptive statistical analysis (including the construction of the clinical cascade from Sauti program data), with the possibility for more advanced statistical analysis, of existing quantitative data will also be conducted. Possible datasets for re-analysis are listed below. Data Source (existing dataset) Description of data Recommended analysis Sauti project monitoring data routinely collected by Jhpiego Data routinely collected as part of the project, primarily for indicator reporting and management purposes. ● Comparison of results against targets. ● Crosstabulation by type of key populations ● Confirm findings as reported in Quarterly and Annual reports. ● Trends over time since the beginning of the project ● Cross tabulations of key indicators by key demographics (e.g., location, sex, age) Key Informant Interviews Interviews will be conducted using a semi-structured question guide. Key informants will include, but not limited to: ● USAID/Tanzania staff working on Key Populations ● Department of Defense (DOD) and Center for Disease Control and Prevention (CDC) staff working on Key Populations ● Sauti staff and sub-partners’ staff, as appropriate ● Government representatives, as appropriate in country ● Beneficiaries (health center staff, SW, MSM, TG, AGYW, Men, etc.) ● Other donor and implementing partners (ICAP, Pact, Engender-Health, NIMR, UNAIDS etc.) Discussions with representatives of CSOs, especially key population leaders and those who advocate for key populations. Focus Group Discussions The purpose is of the focus group discussions (FGDs) are to investigate strengths, weaknesses, successes and challenges as seen by the beneficiaries of Sauti. The FGDs will be a semi-structured gathering of key and priority population beneficiaries, service users, and others who may not utilize services in the districts program but knows about services. The team members will take informal notes, with or without audio recording device, and the notes will be collated and discussed with team members at the end of the day. 41 Group Interviews The purpose is to cluster some Key Informants (see above) into groups for interviews so that information on programs can be shared in a team environment. Categories of groups could include KVP and/or HIV teams at Sauti field and headquarter offices and sub-grantee organizations including community and biomedical service delivery sites. The Evaluation Team will be cognizant to avoid any power differentials within a group, to ensure that all participants in a group feel comfortable sharing their opinions. Survey The purpose of the survey would be to obtain information from the mission and Sauti team. The questionnaire would be short and developed in collaboration with the evaluation team, as appropriate. Observations The purpose of observations is to observe Sauti project intervention activities using a semi-structured observation checklist during site visits. Sites include, but not limited to, resource centers, community and biomedical service delivery sites, community-based ART distribution sites, public health sites, and others. HUMAN SUBJECT PROTECTION The Evaluation Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation (documentation can be shared from previous cascade assessments, to be used, as appropriate), the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community attendance. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. ANALYTIC PLAN The evaluation will: 1. Review information related to the relevant HIV/AIDS and health issues being addressed at the global, national, and community level, and determine the extent of current initiatives of Jhpiego/Sauti and the government (national and local), and their contribution to the overall national responses. 2. Analyze data within the context of PEPFAR initiatives (depending on initiatives may include DREAMS, Elton John Foundation, Test and Start etc.), focus on the provision of technical assistance, capacity building and advocacy. 3. Ascertain program effectiveness in reaching key and priority populations, referring those HIV+ into care and treatment, and project impact on HIV acquisition and/or onward transmission 4. Assess the various current and potential areas for intervention, as described in the program implementation guide, capacity building of local implementing agencies and government counterparts, and gender-based violence reduction activities. All analyses will be geared to answer the evaluation questions. Additionally, the evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Quantitative data will be analyzed primarily using descriptive statistics. Data will be stratified by demographic characteristics, such as sex, age, and location, whenever feasible. Other statistical test of association (i.e., odds ratio) and correlations will be run as appropriate. 42 Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist. Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data behavior surveillance survey data, etc.) will allow the Team to triangulate findings to produce more robust evaluation results. The Evaluation Report will describe analytic methods and statistical tests employed in this evaluation. ACTIVITIES Background reading – Several documents are available for review for this analytic activity. These include Sauti award documents, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, as well as survey data reports (as available). This desk review will provide background information for the Evaluation Team and will also be used as data input and evidence for the evaluation. Team Planning Meeting (TPM) – A team planning meeting (TPM) will be held at the initiation of this assignment and before the data collection begins in Dar es Salaam: ● Review and clarify any questions on the evaluation SOW ● Clarify team members’ roles and responsibilities ● Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion ● Review and finalize evaluation questions ● Review and finalize the assignment timeline ● Develop data collection methods, instruments, tools and guidelines ● Review and clarify any logistical and administrative procedures for the assignment ● Develop a data collection plan ● Draft the evaluation work plan for USAID’s approval ● Develop a preliminary draft outline of the team’s report ● Assign drafting/writing responsibilities for the final report Briefing and Debriefing Meetings – Throughout the evaluation, the Team Lead will provide briefings to USAID. The In-Brief and Debrief are likely to include the all Evaluation Team experts, at USAID/Tanzania, as well as individual in-country briefings. Evaluation launch, a call/meeting among the USAID, Team Lead to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH Pro will introduce the Team Lead and review the initial schedule and review other management issues. ● A final debrief between the Evaluation Team and USAID will be held at the end of the evaluation to present preliminary findings to USAID. During this meeting a summary of the data will be presented, along with high-level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the evaluation report. (Note: preliminary findings are not final and as more data sources are developed and analyzed these finding may change.) ● Stakeholders’ debrief/workshop will be held with the project staff and other stakeholders identified by USAID. This will occur following the final debrief with the Mission and will not include any information that may be deemed sensitive by USAID. 43 Fieldwork, Site Visits and Data Collection – The evaluation team will conduct site visits for data collection. Selection of sites to be visited will be finalized during TPM in consultation with USAID. The evaluation team will outline and schedule key meetings and site visits prior to departing to the field. Evaluation/Analytic Report – The Evaluation/Analytic Team under the leadership of the Team Lead will develop a report with findings and recommendations. Report writing and submission will include the following steps: 1. Team Lead will submit draft evaluation report to USAID for review and formatting. 2. The team lead will provide a separate internal USAID memo that provides information about management, the program, or other sensitive issues that will remain internal to USAID. 3. USAID/Tanzania will review the draft report and the memo and send their comments and edits back to Team Lead. 4. USAID will then share this report with Sauti to review the report and provide a statement of difference, if they choose. 5. Team Lead will review and reformat the final Evaluation/Analytic Report, as needed, and resubmit to USAID for approval. 6. Once Evaluation Report is approved USAID post it to the DEC. The Evaluation Report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. This information will be submitted in a memo to USAID separate from the Evaluation Report. DELIVERABLES AND PRODUCTS Deliverable / Product Timelines & Deadlines (estimated) Attend in-brief with USAID/Tanzania - Attend in-brief with Sauti - Submit workplan with timeline - Submit evaluation protocol with data collection tools - Routine briefings Weekly Conduct desk review Prior to country travel Documentation of results from desk review - Present in-country in-brief / debriefs with USAID Mission with Power Point presentation The in briefs should occur after arrival in country, close to the week-----------2018. Teams will be required to provide a preliminary debrief for USAID mission and stakeholders prior to leaving the country. Debrief with Jhpiego/Sauti with Power Point presentation - Submit draft report to USAID - Final report - Report Posted to the DEC - Estimated USAID review time Average number of business days USAID will need to review deliverables requiring USAID review and/or approval? Minimum of 10 Business days TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) Evaluation/Analytic team: When planning this analytic activity, consider: ● Key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc. ● Team leaders for evaluations/analytics must be an external expert with appropriate skills and 44 experience. ● Additional team members can include research assistants, enumerators, translators, logisticians, etc. ● Teams should include a collective mix of appropriate methodological and subject matter expertise. ● Evaluations require an Evaluation Specialist, who should have evaluation methodological expertise needed for this activity. Similarly, other analytic activities should have a specialist with methodological expertise related to the ● Note that all team members will be required to provide a signed statement attesting that they have no conflict of interest or describing the conflict of interest if applicable. Team Qualifications: All Evaluation Team Members will: ● Work with Team Lead to finalize and negotiate the team work plan for the assignment; ● Collectively establish assignment roles, responsibilities, and tasks for each team member; ● Work with Team Lead to ensure that the logistics arrangements in the field are complete; ● Participate in Team Planning Meetings to set the agenda and other elements of the performance evaluation; ● Responsible for preparing specific sections of report, providing input, developing sections of presentation, and revising the assignment report; ● Contribute to the process of report writing; ● Assist Team Lead with the workflow and tasks to ensure that the evaluation adheres to schedule; and ● Assist Team Lead with any supporting logistics, i.e., ensure that team field logistics are arranged (e.g., administrative/clerical support is engaged, ensuring that payment is made for services, car/driver hire, or other travel and transport is arranged, etc.). Members of the evaluation team should be thoroughly familiar with the document entitled “PEPFAR 3.0 Controlling the Epidemic: Delivering on the Promise of an AIDS-free Generation”. Team Lead/Evaluation Specialist: This person will be selected from among the key staff and will meet the requirements of both this and the other position. The team lead will have extensive experience conducting health project evaluations, including evaluation of HIV/AIDS projects. S/he will ensure quality assurance on evaluation issues, including design methods and the development of data collection instruments. Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation/analytic team, and (5) leading briefings and presentations. Qualifications: ● Advanced degree in Public Health, Public Policy/Administration, or a related field ● Minimum of 10 years of experience in public health, which includes experience in implementation of health activities in resource limiting settings ● Demonstrated experience leading health sector project/performance evaluation/analytics, utilizing both quantitative and qualitative s methods ● Demonstrated ability in designing and implementing development programs on a nation-wide or region-wide basis. ● Excellent skills in planning, facilitation, and consensus building ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent skills in project management, leadership, teamwork and teambuilding ● Excellent organizational skills and ability to keep to a timeline ● Good writing skills, with extensive report writing experience 45 ● Experience with key populations is desirable. ● Familiarity with USAID and PEPFAR project implementation ● Familiarity with USAID and PEPFAR policies and practices − Evaluation policies − Results frameworks − Performance monitoring plans Key Staff 1&2 Title: Key Populations Specialist (FSW and for MSM/TG)- Local Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in HIV, in prevention, treatment, care and support services, particularly for high risk groups. S/He will participate in planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Qualifications: ● At least 8 years’ experience with HIV/AIDS projects/programming; USAID project implementation experience preferred ● Expertise in supply and demand for HIV services at the community and clinical level ● Knowledgeable about HIV/AIDS prevention, biomedical services, health systems strengthening, policy, and other issues related to targeted interventions for HIV service delivery for key populations ● Firm understanding of working with key populations, including dealing with stigma and discrimination ● Biomedical experience would be considered a plus ● Familiar with PEPFAR guidelines and policies, including − PEPFAR Next Generation Indicators Reference Guidance − PEPFAR Monitoring, Evaluation, and Reporting Indicator Reference Guide − PEPFAR Evaluation Standards of Practice − Capacity Building and Strengthening Framework − Gender Strategy − Country Operational Plans (COP) − Site Improvement through Monitoring System (SIMS) ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Proficient in English ● Good writing skills, specifically technical and evaluation report writing experience ● Experience in conducting USAID evaluations of health programs/activities Key Staff 3 Title: Strategic Information Specialist- Local Roles & Responsibilities: Serve as a member of the evaluation team, assist the lead evaluator in providing quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. S/He will oversee the training of all engaged in data collection, ensuring highest level of reliability and validity of data being collected. S/He is the lead analyst, responsible for all data analysis, and will coordinate the analysis of all data, assuring all quantitative and qualitative data analyses are done to meet the needs for this evaluation. S/He will participate in all aspects of the evaluation, from planning, data collection, and data analysis to report writing. Qualifications: ● At least 10 years of experience in USAID M&E procedures and implementation ● At least 5 years managing M&E, including evaluations ● Experience in design and implementation of evaluations ● Strong knowledge, skills, and experience in qualitative and quantitative evaluation tools 46 ● Experience implementing and coordinating other to implements surveys, key informant interviews, focus groups, observations and other evaluation methods that assure reliability and validity of the data. ● Experience in data management ● Able to analyze quantitative, which will be primarily descriptive statistics ● Able to analyze qualitative data ● Experience using analytic software ● Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data ● Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation ● Strong data interpretation and presentation skills ● Significant experience in developing and implementing monitoring systems and conducting evaluations for HIV/AIDS prevention and/or impact mitigation and service delivery programs ● An advanced degree in public health, evaluation or research or related field ● Proficient in English ● Good writing skills, including extensive report writing experience ● Familiarity with USAID health programs/projects, primary health care or health systems strengthening preferred ● Familiarity with USAID and PEPFAR M&E policies and practices − Evaluation policies − Results frameworks − Performance monitoring plans − PEPFAR Next Generation Indicators Reference Guidance − PEPFAR Monitoring, Evaluation, and Reporting 2.0 Indicator Reference Guide − PEPFAR Evaluation Standards of Practice − Site Improvement through Monitoring System (SIMS) Staffing Level of Effort (LOE) Matrix (Optional): Level of Effort in days for each Evaluation/Analytic Team member Activity/ Deliverable Evaluation Team Team Lead/ Evaluation Specialist Strategic Information Specialist KP Specialist (FSW) KP Specialist (MSM) Number of persons → 1 1 1 1 1 Launch Briefing 1 1 1 1 2 Desk review 5 5 5 5 3 Preparation for Team convening in-country 5 5 5 5 4 Travel to country 8 5 Team Planning Meeting 5 5 5 5 6 In-brief with Mission 4 4 4 4 7 In-brief with project 2 2 2 2 8 Prep / Logistics for Site Visits 2 2 2 2 9 Data collection / Site Visits (including travel to sites) 30 30 30 30 10 Data analysis 5 5 5 5 11 Debrief with Mission with prep 2 2 2 2 47 Activity/ Deliverable Evaluation Team Team Lead/ Evaluation Specialist Strategic Information Specialist KP Specialist (FSW) KP Specialist (MSM) 12 Stakeholder debrief workshop with prep 2 2 2 2 13 Depart country 8 8 8 8 14 Draft report(s) 10 - - - 15 GH Pro Report QC Review & Formatting - - - - 16 Submission of draft report(s) to Mission - - - - 17 USAID Report Review - - - - 18 Revise report(s) per USAID comments 3 2 2 1 19 Finalize and submit report to USAID - - - - 20 508 Compliance Review - - - - 21 Upload Eval Report(s) to the DEC - - - - Total LOE per person 53 50 50 50 Total LOE 53 50 50 50 ANALYTIC REPORT The product of this evaluation will be a final report that evaluates the successes, shortcomings, and lessons learned of Jhpiego Sauti activity that will inform a new project design for USAID. The report should also include an evaluation of the sustainability of the project. The report should include recommendations for improving USAID’s assistance delivery for key and priority populations and highlight comparative advantages in areas not addressed by other initiatives. The Evaluation/Analytic Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report. a. The report must not exceed 30 pages (excluding executive summary, table of contents, acronym list and annexes). b. The structure of the report should follow the Evaluation Report template, including branding found here or here. c. Draft reports must be provided electronically, in English d. For additional Guidance, please see the Evaluation Reports to the How-To Note on preparing Evaluation Draft Reports found here. e. It should also adhere to the PEPFAR Evaluation Standards of Practice. Reporting Guidelines: The draft report should be a comprehensive analytical evidence-based evaluation/analytic report. It should detail and describe results, effects, constraints, and lessons learned, and provide recommendations and identify key questions for future consideration. The report shall follow USAID branding procedures. The report will be edited/formatted and made 508 compliant as required by USAID for public reports and will be posted to the USAID/DEC. The findings from the evaluation/analytic will be presented in a draft report at a full briefing with USAID and at a follow-up meeting with key stakeholders. The report should use the following format: ● Executive Summary: concisely state the most salient findings, conclusions, and recommendations (not more than 4 pages); 48 ● Table of Contents (1 page); ● Acronyms ● Evaluation/Analytic Purpose and Evaluation/Analytic Questions (1-2 pages) ● Project [or Program] Background (1-3 pages) ● Evaluation/Analytic Methods and Limitations (1-3 pages) ● Findings ● Conclusions ● Recommendations ● Annexes - Annex I: Evaluation/Analytic Statement of Work - Annex II: Evaluation/Analytic Methods and Limitations - Annex III: Data Collection Instruments - Annex IV: Sources of Information o List of Persons Interviews o Bibliography of Documents Reviewed o Databases o [etc] - Annex V: Disclosure of Any Conflicts of Interest - Annex VI: Statement of Differences (if applicable) The evaluation methodology and report will be compliant with the USAID Evaluation Policy and Checklist for Assessing USAID Evaluation Reports, and with PEPFAR Evaluation Standards of Practice. -------------------------------- The Evaluation Report should exclude any potentially procurement-sensitive information. As needed, any procurement sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USAID separate from the Evaluation Report. The memo will also include other sensitive issues or concerns that will not be included in the overall evaluation report. -------------------------------- All data instruments, data sets (if appropriate), presentations, meeting notes and report for this evaluation/analysis will be provided to GH Pro and presented to USAID electronically to the Program Manager. All data will be in an unlocked, editable format. USAID CONTACTS Identifying Category Primary Contact Alternate Contact 1 Alternate Contact 2 Name: Erick Mlanga Anike Akridge Lilian Benjamin Title: Project Management Specialist Project Management Specialist Health Promotion Unit Lead USAID Mission USAID/Tanzania Health Office USAID/Tanzania Health Office USAID/Tanzania Health Office Email: emlanga@usaid.gov aakridge@usaid.gov lbenjamin@usaid.gov Telephone: 255713768441 - - Annex A: Interview guide questions To answer the four main evaluation questions and to provide the appropriate recommendations at the end of the evaluation, the following list provides a set of sample questions under four major themes that the evaluation team must consider: Program Management, Program Accomplishments and Results, Monitoring and Evaluation, and Lessons Learned. However, additional questions can be added during the evaluation when deemed 49 appropriate. We would like the evaluation team to lead to help further develop and refine these questions. One area should also be in achievement towards 90-90-90 in Key and Priority Population programming at the country level. 1. Program Management ● Has the project established constructive working relationships with key stakeholders to improve program outcomes (partners, government, NGOs, others)? ● How the sub-contractors of Jhpiego/Sauti adapted to the continuously changing demands and new data collecting initiatives coming from PEPFAR? ● Is USAID satisfied with communications with project staff, both that of Jhpiego/Sauti and those of their sub-partners? Is the staff of Jhpiego/Sauti and their sub-partners satisfied with USAID? What improvements could be made for more effective communication? ● Do work plans and budgets reflect project priorities? How are they used as project management tools? ● Have the staff in the sites received supportive supervision throughout the life of the project? What are their main points of satisfaction and main grievances with the project? ● Were members of the evaluation team given an opportunity to assess the quality of services? If yes, what are their impressions? 2. Program Accomplishments and Results: ● Does the project have well-reasoned strategies to achieve its goals, objectives and indicators within the life of project? Is the vision and direction still valid or should there be changes or new approaches? ● Is the project truly reaching the populations they are targeting? Are the target populations reached effectively? ● Has the project successfully engaged and retained key and priority populations in the HIV cascade? ● Is the project likely to have had impact on HIV acquisition and/or onward transmission among key and priority populations? ● If the project had changed and/or modified its strategy or priorities during the implementation phase, were the changes/modifications justified? ● What has been the effectiveness and quality of program inputs (training, systems developed, guidelines, services, etc.)? Are they timely and appropriate to achieving program goals and objectives? ● Is adequate technical leadership available among local project staff or from partner HQ staff, consultants, etc.? Is HQ staff or consultants used appropriately and judiciously? How is the management structure with USAID/Tanzania working? ● How effective is Jhipiego/Sauti approach to capacity building for local organizations? Has the program achieved its goals and objectives in this area? ● What has the project done to implement sustainable activities? If project activities were to stop at the end of the project, what activities would/would not continue to operate? Would the government have the systems and/or capacity to carry forward Jhpiego/Sauti activities? ● Is the project creating parallel service delivery systems from the government’s established system? Are the services implemented or supported by the project well integrated into the government’s established system? What has the project done to build health systems and human resources capacity at different levels within the government or other local partners? Are the activities implemented by Jhpiego/Sauti in alignment with the National HIV/AIDS strategy? Have the project’s collaborations and partnerships been successful? Does project partnership work to effectively use available skills of all partners? How successful has the project been in leveraging resources from other donors and partners to maximize project’s impact? How has this project complemented work done under other projects (e.g. CDC and/or Global Fund projects) and avoided duplication? ● To what degree are the country governments and other partner’s replicating/scaling-up project best practices and models? 50 ● Is there evidence that Jhipiego/Sauti approach has: - Introduced and improved the quality and availability of biomedical services in the community? - Improved the quality of services at the community level? - Improved referral linkages between community and facility level and within the facility level. ● What kind of model has Sauti implemented in country? Is there evidence that Jhpiego-supported sites are providing quality HIV/AIDS services? What are the gaps in service provision, if any? ● Is the geographic scope of the program still relevant given the latest data and understanding of where other donors are working? ● When Sauti ends, will they have left enough of a ‘capacity building footprint’ such that the current sites can provide an acceptable standard of HIV prevention, Linkage and other biomedical services? ● How has Sauti demonstrated national technical leadership? How were interventions or innovations adopted by other organizations? 3. Monitoring and Evaluation (M&E) ● Is the M&E plan being implemented and kept up to date? How are data being used by project management to make strategic and management decisions? With whom are the data shared? How are project achievements reported to the national government systematically and on a regular basis? ● How might M&E systems be improved? ● Is the project creating parallel reporting system? How is their routine program monitoring aligned with the existing government reporting system? ● What M&E systems are in place to monitor and follow the progress and trends of their achievements? ● How successful has Sauti been in providing valid and reliable strategic information? Are data collected by Sauti disseminated-widely, regularly, and utilized by key stakeholders? ● Has the project improved the Government’s capacity to produce and manage strategic information? 4. Lessons Learned ● Does the service delivery model used by Sauti meet the needs of key and priority populations? ● What are the lessons learned from successful interventions that merit continuation or replication, better practices, significant products and tools for possible dissemination and replication? ● What are the gaps in services and how can this be corrected or mitigated for the remaining lifespan of this project or in the follow-on activity? ● Is current focus and methodology used by Jhpiego/Sauti appropriate for the country’s epidemic? If not, what adjustments need to be made for the remainder of the project and for a follow-on activity? ● How well do Sauti activities in country collaborate and leverage activities in country funded by other donors? 51 ANNEX II: BIBLIOGRAPHY OF DOCUMENTS REVIEWED 52 Fourth Health Sector HIV and AIDS Strategic Plan (HSHSP IV) 2017 – 2022 Ministry of Health, Community Development, Gender, Elderly and Children, National AIDS Control Programme, (2017), Fourth Health Sector HIV and AIDS Strategic Plan (HSHSP IV) 2017 – 2022. Dar es Salaam: Government of Tanzania. NACP, (2017). National Guideline for Comprehensive Package of HIV Interventions for Key and Vulnerable Populations, MOHCDGEC, 2017. Pact.2018. Government Performance Index(GPI): A Pact Handbook. Washington, DC: Pact Sauti Project, (April 2015), Annual Workplan FY15 Sauti Project, (April 2015). Quarterly Progress Report FY15 Q2 Sauti Project, (April 2016). Quarterly Progress Report FY16 Q2 Sauti Project, (April 2017). Quarterly Progress Report FY17 Q2 Sauti Project, (April 2018), Unapproved Annual Workplan FY19 Sauti Project, (August 2017), Annual Workplan FY18 Sauti Project, (February 2016). Quarterly Progress Report FY16 Q1 Sauti Project, (February 2017). Quarterly Progress Report FY17 Q1 Sauti Project, (July 2015). Quarterly Progress Report FY15 Q3 Sauti Project, (July 2016). Quarterly Progress Report FY16 Q3 Sauti Project, (July 2017). Quarterly Progress Report FY17 Q3 Sauti Project, (June 2016). Standard Operating Procedures for Social and Behavior Change Communication Sauti Project, (March 2016). Sauti Cash Transfer Program (CTP) SOP for Vulnerable Adolescent Girls and Young Women Sauti Project, (November 2015), Annual Workplan FY16 Sauti Project, (November 2016), Annual Workplan FY17 Sauti Project, (October 2015). Quarterly Progress Report and Annual FY15 Q4 and FY15 Annual Results Sauti Project, (October 2016). Quarterly Progress Report and Annual FY16 Q4 and FY16 Annual Results Sauti Project, (October 2016). Standard Operating Procedures for WORTH+ Groups Sauti Project, (October 2017). Quarterly Progress Report and Annual FY17 Q4 and FY17 Annual Results Sauti Project, (Revised Dec 2017). Monitoring, Evaluation and Learning Plan Sauti Project, (September 2016). Standard Operating Procedures for SASA! Approaches in SAUTI DREAMS Districts Sauti Project, (September 2016). Standard Operating Procedures for Biomedical Services TACAIDS, (2016). Gender Operational Plan for HIV Response in Tanzania Mainland, MOHCDGEC, 2016. Tarimo, B. L. (2018, January). Health Policy Plus. Retrieved from Health Policy Plus website:http://www.healthpolicyplus.com/pubs.cfm?get=7183 UNICEF. (2018, October). Research and Reports: UNICEF. Retrieved from UNICEF Website https://www.unicef.org/esaro/UNICEF-Tanzania-2018-Health-Budget-Brief-Mainland.pdf USAID, (2015). Environmental Monitoring and Mitigation Plan USAID, (2015). Sauti AID-621-A-15-00003 Original Award_Jhpiego_CP2 53 ANNEX III: KII AND FGD DISCUSSION GUIDES 54 Focus Group Discussion Protocol: KVPs Female The following protocol is to be used to facilitate focus groups with Key vulnerable population female group. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping you, and how it can do better. In this discussion, we will ask you questions about yourself and your friends. When I say “friends”, “colleagues”, “peers” I’m talking about women passing in similar situation you are passing. Description of Study Procedures If you agree to be in this focus group discussion, you will be asked to share your experiences with and views about the SAUTI project; you will also be asked to share information regarding your health care needs and about any ways in which your behavior might have changed as a result of the SAUTI project. I will be asking the questions to the groups. Let’s allow each other to speak uninterrupted and respect each other’s comments. There are no right or wrong answers here. What we discuss during our group talk should remain here with us. Our team will take notes during our conversation and we will also be recording our session. The recording and the notes are to make sure that the evaluators have heard and understood what you all share with us today and in other similar discussions we will be conducting at other sites. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 60-90 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the focus group. Once again, we ask that what we discuss during our group talk remains here with us. Risks/Discomforts of Participating in this Study Some of the questions I will be asking are about personal issues and you may feel emotional or upset when answering some of the questions. Tell me at any time if you wish to take a break or stop the discussion. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this focus group, but we will reimburse any transportation expenses you have incurred to attend this meeting. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. Your decision to refuse will not result in any loss of benefits like project services that you normally receive. You have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from the analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have question for me at this time? If you have any further questions about the study feel free to contact [Tanzania-based Evaluation Team POC] at [EMAIL] or by telephone at [PHONE NUMBER]. If you have any other concerns about your rights as a research participant that have not been answered by the investigators, you may contact M&E Specialist for Data for Development, Daud Siwalaze at dsiwalaze@engl.com or by telephone at 0758 067 700. We will provide this information to you on a piece of paper. 55 Consent Do you agree to participate in this study? If you disagree, please feel free to leave at this time. Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Site location: - Focus group identifier: - Number of participants: - Ages of participants: (Enter each participant’s age separated by a comma) - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission to turn on the tape recorder and begin the focus group? If you do not wish to be recorded, please feel free to leave at this time. Access to Services I would like to start the conversation by asking about the assistance you receive from SAUTI. A.1 How has the SAUTI project helped you? What services are you receiving through the SAUTI project? Probe for services across the core package offered by SAUTI Continue until respondents provide an exhaustive list of services; check that it is exhaustive by suggesting services they have not named A.2 Do all of you receive all this help that we’ve listed here? What do you each receive, and what do you not receive? Allow respondents to indicate which of the services in the core package they each receive A.3 Let’s talk about each of these services. How do you receive them? A.3.1 Do you receive any directly from SAUTI? A.3.2 Which services does SAUTI link you to? How do they link you to the services? Go through the services listed to obtain data on how SAUTI facilitates access. A.4 Do you ever carry forms to a service or from one service to other, so that they know who sent you? A.4.1 Can you describe how you carry these forms between services? A.4.2 What happens if you don’t have your form? Do you still get the service? A.4.3 Do you have to fetch the form before receiving the service? Do you return with the form after receiving the service? 56 Changes in Access and Quality Were you getting any assistance or services that we’ve been talking about before SAUTI started helping you? B.1 Which services were you already receiving before SAUTI began assisting you? Prompt respondents by referring back to the list of services they nominated earlier B.2 Which services do you now get that you were not getting before SAUTI began assisting you? B.3 When it comes to services you were receiving before, what has changed since SAUTI started assisting? For example, is it easier to get the services now than it was before? Why do you say so? Prompt for clear examples of improved access with reference to knowledge about service offerings, obstacles to access such as distance and cost, and whether the project assists in addressing these B.4 Has anything else changed about the services you get? For example, are they better in any way than they were before SAUTI started helping? Why do you say so? Prompt for clear examples of improved quality and service experience, including friendlier services in terms of hours, location, staff attitudes Continued Challenges to Access and Quality What remains difficult about getting this assistance or these services? C.1 Do you feel uncomfortable going to the service provider because you may be seen as someone who exchanges sex for money? Are you treated differently than other people getting the same service? C.2 Are you, or your friends, colleagues or peers afraid of the authorities such as the police? Does the project help them in any way to deal with the police, for example, if they need to report anyone who has acted violently? Be careful to manage this line of questioning sensitively, keep the discussion in the 3rd person as much as possible Prompt for stories of engagement with police of justice system C.3 Are there any other difficulties? Prompt for difficulties including distance to services and cost C.4 Have there been times during the project when you could not get the services promised? Tell me more about that? C.5 Have there been times during the project when services got worse than they were in the beginning? Tell me about that? C.6 What services do you need that the project is not providing? C.7 Could the services already being provided be better? How? Finding Clients SAUTI is supposed to help people that find it difficult to get the health services they need because of how their sexual preferences/behavior is seen by others. D.1 Can you please describe how you were enrolled in the project? Allow a respondent to describe enrolment and ask for similar or different experiences from the first example Look for examples of clients connecting other clients to the project Continue to ask for examples until you’ve exhausted all modalities of enrolment D.2 Is the project reaching all your friends, colleagues and peers? D.2.1 Who is the project not reaching? Where do they work? D.2.2. Why do you think the project cannot reach these women? D.2.3 How do you think the project could reach them? 57 Behavior Change and Community Norms Change Has being enrolled in the project led to you changing the way you behave in your work or daily life? E.1 Are you being safe when having sex? Can you explain how? Prompt for consistent condom use, asking transactional partners if they’ve been tested, other interactions with sexual partners E.2 Are you accessing health services more regularly? Prompt for regularity of testing, accessing ARTS E.3 Are there other ways in which you have changed how you behave? Prompt for adherence, encouraging partners to go for testing, testing children E.4 Has SAUTI succeeded in changing how the community sees women, girls or women who exchange sex for money? Prompt for changes in gender norms, perspectives on GBV, girls attending school Ask for specific stories or example Other Issues I would like to finish our discussion with 3 more questions. F.1 Do you think that the positive changes we’ve talked about when we spoke of services and other things will remain when the SAUTI project finishes? Why do you say so? F.2 How has being assisted by SAUTI impacted your life? I would like to hear stories from you about that. F.3 Are any of you enrolled in the DREAMS program? Do you have a DREAMS passport? May I see it? Conclusion We would like to thank you all for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. 58 Focus Group Discussion Guide: Vulnerable Adolescent Girls and Young Women (vAGYW) The following protocol is to be used to facilitate focus groups with vAGYW respondents. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping you, and how it can do better. In this discussion, we will ask you questions about yourself and your friends. When I say “friends”, “colleagues”, “peers” I’m talking about young and adolescent women you know who are experiencing same challenges you are experiencing. Description of Study Procedures If you agree to be in this focus group discussion, you will be asked to share your experiences with and views about the SAUTI project; you will also be asked to share information regarding your health care needs and about any ways in which your behavior might have changed because of the SAUTI project. I will be asking the questions to the groups. Let’s allow each other to speak uninterrupted and respect each other’s comments. There are no right or wrong answers here. What we discuss during our group talk should remain here with us. Our team will take notes during our conversation and we will also be recording our session. The recording and the notes are to make sure that the evaluators have heard and understood what you all share with us today and in other similar discussions we will be conducting at other sites. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 60-90 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the focus group. Once again, we ask that what we discuss during our group talk remains here with us. Risks/Discomforts of Participating in this Study Some of the questions I will be asking are about personal issues and you may feel emotional or upset when answering some of the questions. Tell me at any time if you wish to take a break or stop the discussion. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this focus group, but we will reimburse any transportation expenses you have incurred to attend this meeting. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. Your decision to refuse will not result in any loss of benefits like project services that you normally receive. You have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the 59 information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from the analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have question for me at this time? If you have any further questions about the study, feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. Consent Do you agree to participate in this study? If you disagree, please feel free to leave at this time. Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Site location: - Focus group identifier: - Number of participants: - Age of participants: (Enter each participant’s age separated by a comma) - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission to turn on the tape recorder and begin the focus group? 60 Access to Services I would like to start the conversation by asking about the assistance you receive from SAUTI. A.1 How has the SAUTI project helped you? What services are you receiving through the SAUTI project? Probe for services across the core package offered by SAUTI Continue until respondents provide an exhaustive list of services; check that it is exhaustive by suggesting services they have not named A.2 Do all of you receive all this help that we’ve listed here? What do you each receive, and what do you not receive? Allow respondents to indicate which of the services in the core package they each receive A.3 Let’s talk about each of these services. How do you receive them? A.3.1 Do receive any directly from SAUTI? A.3.2 Which services does SAUTI link you to? How do they link you to the services? Go through the services listed to obtain data on how SAUTI facilitates access. A.4 Do you ever carry forms to a service or from one service to other, so that they know who sent you? A.4.1 Can you describe how you carry these forms between services? A.4.2 What happens if you don’t have your form? Do you still get the service? A.4.3 Do you have to fetch the form before receiving the service? Do you return with the form after receiving the service? Changes in Access and Quality Were you getting any assistance or services that we’ve been talking about before SAUTI started helping you? B.1 Which services were you already receiving before SAUTI began assisting you? Prompt respondents by referring back to the list of services they nominated earlier B.2 Which services do you now get that you were not getting before SAUTI began assisting you? B.3 When it comes to services you were receiving before, what has changed since SAUTI started assisting? For example, is it easier to get the services now than it was before? Why do you say so? Prompt for clear examples of improved access with reference to knowledge about service offerings, obstacles to access such as distance and cost, and whether the project assists in addressing these B.4 Has anything else changed about the services you get? For example, are they better in any way than they were before SAUTI started helping? Why do you say so? Prompt for clear examples of improved quality and service experience, including friendlier services in terms of hours, location, staff attitudes Continued Challenges to Access and Quality What remains difficult about getting this assistance or these services? C.1 Do you feel uncomfortable going to the service provider because you are a young woman? Are you treated differently than other people getting the same service? C.2 Are there any other difficulties? Prompt for difficulties including distance to services and cost C.3 Have there been times during the project when you could not get the services promised? Tell me more about that? C.4 Have there been times during the project when services got worse than they were in the beginning? Tell me about that? 61 C.5 What services do you need that the project is not providing? C.6 Could the services already being provided be better? How? Finding Clients SAUTI is supposed to help adolescent girls and young woman who are in very difficult circumstances and at risk of being exploited or harmed. The project is supposed to get them the services they need. D.1 Can you please describe how you were enrolled in the project? Allow a respondent to describe enrolment and ask for similar or different experiences from the first example Look for examples of clients connecting other clients to the project Continue to ask for examples until you’ve exhausted all modalities of enrolment D.2 Is the project reaching all your friends, colleagues, peers that are facing challenging circumstances like you are or even worse? Are there groups it is not reaching? D.2.1 Who is the project not reaching? D.2.2 Why do you think the project cannot reach these girls or young women? D.2.3 How do you think the project could reach them? Behavior Change and Community Norms Change Has being enrolled in the project led to you changing the way you behave in your work or daily life? E.1 Are you accessing health services more regularly? Prompt for regularity of testing, accessing ART E.2 Are there other ways in which you have changed how you behave? Prompt for avoiding risky sex, consistent condom use, asking partners if they’ve been tested, other interactions Prompt for adherence, encouraging partners to go for testing, testing children E.3 Has SAUTI succeeded in changing how the community sees women and girls? Prompt for changes in gender norms, perspectives on GBV, girls attending school Ask for specific stories or example Other Issues I would like to finish our discussion with 3 more questions. F.1 Do you think that the positive changes we’ve talked about when we spoke of services and other things will remain when the SAUTI project finishes? Why do you say so? F.2 Are any of you enrolled in the DREAMS program? How has DREAMS impacted your life? F.3 Do you have a DREAMS passport? May I see it? Conclusion We would like to thank you all for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you have given us. We have learned a lot from you, and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. 62 Key Informant Interview Protocol: Key Vulnerable population (KVPs)_Men The following protocol is to be used to conduct interviews with KVPs under the male group respondents. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping you, and how it can do better. In this discussion, we will ask you questions about yourself and your friends. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project; you will also be asked to share information regarding your health care needs and about any ways in which your behavior might have changed as a result of the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Risks/Discomforts of Participating in this Study Some of the questions I will be asking are about sensitive topics and you may feel uncomfortable when answering them. Tell me at any time if you wish to take a break or stop the discussion. You can also decline to answer any of my questions and I will move on to a new question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview, but we will reimburse any transportation expenses you have incurred to attend this meeting. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned, you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Do you have questions for me at this time? 63 If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Site location: - Interview identifier: - Age of respondent: - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission turn on the tape recorder and begin the interview? 64 Access to Services I would like to start the conversation by asking about the assistance you receive from SAUTI. A.1 How has the SAUTI project helped you? What services are you receiving through the SAUTI project? Probe for services offered to clients in this area/in this client category by SAUTI Continue until respondent provides an exhaustive list; check that it is exhaustive by suggesting services they have not named A.2 Let’s talk about each of these services. How do you receive them? A.2.1 Do receive any directly from SAUTI? A.2.2 Which services does SAUTI link you to? How do they link you to the services? Go through the services listed to obtain data on how SAUTI facilitates access. A.3 Do you ever carry forms to a service or from one service to other, so that they know who sent you? A.3.1 Can you describe how you carry these forms between services? A.3.2 What happens if you don’t have your form? Do you still get the service? A.3.3 Do you have to fetch the form before receiving the service? Do you return with the form after receiving the service? Changes in Access and Quality Were you getting any assistance or services that we’ve been talking about before SAUTI started helping you? B.1 Which services were you already receiving before SAUTI began assisting you? Prompt respondent by referring back to the list of services they nominated earlier B.2 Which services do you now get that you were not getting before SAUTI began assisting you? B.3 When it comes to services you were receiving before, what has changed since SAUTI started assisting? For example, is it easier to get the services now than it was before? Why do you say so? Prompt for clear examples of improved access with reference to knowledge about service offerings, obstacles to access such as distance and cost, and whether the project assists in addressing these B.4 Has anything else changed about the services you get? For example are they better in any way than they were before SAUTI started helping? Why do you say so? Prompt for clear examples of improved quality and service experience, including friendlier services in terms of hours, location, staff attitudes Continued Challenges to Access and Quality What remains difficult about getting this assistance or these services? C.1 Do you feel uncomfortable going to the service provider for any reason? Are you treated differently than other people getting the same service? C.2 Are you ever fearful of the authorities such as the police? Do you know of any instances when SAUTI has assisted people in dealing with the authorities? Can you describe that instance? C.2 Are there any other difficulties with services SAUTI helps you with? Prompt for difficulties including distance to services and cost C.3 Have there been times during the project when you could not get the services promised? Tell me more about that? C.4 Have there been times during the project when services got worse than they were in the beginning? Tell me about that? 65 C.5 What services do you need that the project is not providing? C.6 Could the services already being provided be better? How? Finding Clients SAUTI is supposed to help people that find it difficult to get the health services they need because of how their sexual preferences/behavior is seen by others. D.1 Can you please describe how you were enrolled in the project? Allow interviewee to describe enrolment D.2 Is the project reaching all you friends, colleagues, peers that find it difficult to get the health services they need because of how they are seen by others? Are there groups it is not reaching? D.2.1 Who is the project not reaching? D.2.2. Why do you think the project cannot reach these people? D.2.3 How do you think the project could reach them? Behavior Change and Community Norms Change Has being enrolled in the project led to you changing the way you behave in your work or daily life? E.1 Are you accessing health services more regularly? Prompt for regularity of testing, accessing ARTS E.2 Are there other ways in which you have changed how you behave? Prompt for multiple sexual partners, consistent condom use, asking partners if they’ve been tested, other interactions Prompt for adherence, encouraging partners to go for testing E.3 Is SAUTI doing anything to improve how your sexual preferences/behaviors are perceived in the community or how you treated as a result? Can you explain how? E.4 Has SAUTI succeeded in changing how the community sees women and girls? Prompt for changes in gender norms, perspectives on GBV, girls attending school Ask for specific stories or example Other Issues I would like to finish our discussion with 2 more questions. F.1 Do you think that the positive changes we’ve talked about when we spoke of services and other things will remain when the SAUTI project finishes? Why do you say so? F.2 How has being assisted by SAUTI impacted your life? I would like to hear stories from you about that. Conclusion We would like to thank you for participating in this conversation. Sometimes these conversations can be difficult and sometimes you participate when it is difficult or risky for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. 66 Group Interview/KII Protocol: Local Government Officials The following protocol is to be used with government officials at local level INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping beneficiaries, and how it can do better. In this discussion we will ask you questions about the implementation of the project, its effectiveness, how it is managed, and your relationship with the development and implementing partners. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Risks/Discomforts of Participating in this Study We have not identified any risks/discomforts, but you can also decline to answer any of my questions should they make you uncomfortable. If you do so, I will move on to the next question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. Any of you may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned, you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have questions for me at this time? 67 If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Government Department/Agency and location: - Number of participants: - Designations10 of participants (Enter each participant’s designation separated by a comma): - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission turn on the tape recorder and begin the interview? 10 This is the role or job title of each individual participating. Do not document participant names in this table. 68 About SAUTI Project. A.1 How familiar are you with the SAUTI project? Elaborations and probes: Could you describe its goals and objectives as you understand them? Could you describe how it is implemented locally? A.2 From your perspective, how well does the project address local KVP related priorities? Elaborations and probes: What are the KVP related priorities in this area, and is SAUTI addressing those? How well does it align with your implementation of the National Guideline for a Comprehensive Package of HIV Interventions for Key and Vulnerable Populations (2017)? What additional policies, priorities and guidelines are crucial for local government that the project must also adhere to? SAUTI Collaboration and Strengthening of Local Structures B.1 Describe the role of local government structures in the project. Elaborations and probes: For example, how involved have you or other local structures (such as RHMTs and CHMTs) been in SAUTI’s design, its planning, its budgeting, its implementation, or its oversight? To what extent are you consulted on project related matters? B.2 Describe how local community structures and CSOs are involved in SAUTI. Elaborations and probes: Which local community structures and CSOs are involved and how? How are they involved in design, planning, budgeting, implementation, or oversight? Was local government involved in identifying local CSOs and community structures for involvement in the project? Were you consulted in any way? B.3 How effective has SAUTI support to local structures been? Elaborations and probes: Has SAUTI support to strengthen RHMTs and CHMTs been effective? Has the project done enough to support strengthening other local government entities been sufficient? What about its support to community and civil society? B.4 Has SAUTI been hindered by any negativity towards KVPs in community structures or local organizations? Elaborations and probes: How prevalent are negative perspectives towards KVPs in communities and has this hindered delivery of services to these groups in any way? Are these prevalent in local government structures too, such as facilities? How has the project and its partners tried to ensure delivery of quality services under these conditions? Have these efforts been successful? B.5 Are there other organizations other than the SAUTI project implementing KVP-focused activities in this area? Elaborations and probes: Which other projects or organizations? In what ways and how well are there activities coordinated? How do they complement one another? How is SAUTI doing differently? What is its niche? Effectiveness of Services Provision to KVPs. C.1 How has the SAUTI project helped achieve local objectives for providing quality health services to the KVPs? Elaborations and probes: For example, would you say that SAUTI has improved access to services for KVPs? Has the project provided the necessary support to strengthen local government structures and initiatives? Why do you say so? Has the project provided the necessary support to strengthen civil society’s or community structures and initiatives? Why do you say so? C.2 How effective would you say SAUTI’s behavior and norms change interventions are proving to be? Elaborations and probes: Can you describe any such activities? Do you know of any examples where these have led to change? What would make these more effective? Which is the most effective SBCC strategy in your view? C.3 What are the persistent challenges relating to provision of quality services to KVPs in your area? 69 Elaborations and probes: Please describe these challenges (in addition to potentially negative attitudes towards KVPs we’ve already discussed) and what solutions are being considered? How is or how can SAUTI contribute to solutions to these challenges? C.4 What are the factors enabling SAUTI’s successful achievement of its objectives and targets, and what factors are holding it back? Elaborations and probes: For example, what are the strengths and weaknesses in the project’s design? Are there challenges in how it is implemented that undermine its effectiveness or quality? Is the technical and management capacity within the project equal to the challenge from your perspective? Does it engage with other local stakeholders optimally? C.5 How effective has the referral system implemented by the project been in improving access to services for KVPs? Elaborations and probes: What systems have SAUTI established for referrals from one service point to another? To what extent are volunteer cadres such as Peer Educators proving effective? What are the challenges limiting the effectiveness of the referral system? Sustainability D.1 To what extent is SAUTI project setting the stage for sustainability? Elaborations and probes: For example, has SAUTI developed and implement the Sustainability Plan? To what extent are the practices the project has put in place to improve access and quality of services likely to be continued by stakeholders after SAUTI closes? Has enough capacity been built in local community and civil society structures? Do you have specific funding concerns that you can describe for us? D.2 Are there any challenges within the local government structures to sustaining service provision to KVPs? Elaborations and probes: Can you describe these challenges? How can these challenges be overcome? D.3 What should SAUTI do differently to ensure the sustained delivery of quality services to KVPs after it closes? Elaborations and probes: What efforts should the project should be making now? Are there efforts that other stakeholders could be making to improve prospects for sustainability? D.4 In your opinion, do you see continuity of the KP interventions if there is no donor led support? Lessons Learned E.1 What lessons have been learned so far that can be used to improve SAUTI for the remainder of the implementation period? Elaborations and probes: For example, what have been the key lessons learned in implementation of SBCC and Gender component of this project? E.2 What lesson have been learned that can be adopted by future initiatives to improve services to KVPs going forward? Elaborations and probes: What are the key elements you would like to see in a follow-on USAID KP project? Conclusion We would like to thank you for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. 70 Group Interview Protocol: National Government Officials The following protocol is to be used with government officials at national level. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping beneficiaries, and how it can do better. In this discussion we will ask you questions about the implementation of the project, its effectiveness, how it is managed, and your relationship with the development and implementing partners. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Risks/Discomforts of Participating in this Study We have not identified any risks/discomforts, but you can also decline to answer any of my questions should they make you uncomfortable. If you do so, I will move on to the next question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. Any of you may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have questions for me at this time? If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. 71 Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Government Department/Agency and location: - Number of participants: - Designations11 of participants (Enter each participant’s designation separated by a comma): - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission turn on the tape recorder and begin the interview? 11 This is the role or job title of each individual participating. Do not document participant names in this table. 72 About Sauti Project. A.1 How familiar are you with the SAUTI project? Elaborations and probes: Could you describe its goals and objectives as you understand them? Could you describe how it is implemented ‘on the ground’? A.2 Describe the role of your department in the project. Elaborations and probes: For example, how involved have you been in its design, its implementation, or its oversight? To what extent are you consulted on any project related matters? A.3 From your perspective, how well does the project align with the National guidelines, priorities and approaches? Elaborations and probes: In particular, how well does it align with the National Guideline for a Comprehensive Package of HIV Interventions for Key and Vulnerable Populations (2017)? What additional policies, priorities and guidelines are crucial for the project to adhere to? A.4 What is the level of engagement of the GoT in planning different interventions under SAUTI? Elaborations and probes: How involved is your department in SAUTI planning? Do you know how involved other departments might be in SAUTI planning? Effectiveness of Services Provision to KVPs. B.1 How has the SAUTI project helped the Government of Tanzania achieve its objective of providing quality health services to the KVPs? Elaborations and probes: For example, would you say that SAUTI has improved access to services for KVPs? Has the project provided the necessary support to strengthen government structures and initiatives? Why do you say so? Has the project provided the necessary support to strengthen civil society’s or community structures and initiatives? Why do you say so? B.2 How effective would you say SAUTI’s behavior and norms change interventions are proving to be? Elaborations and probes: Can you describe any such activities you may be aware of? Do you know of any examples where these have led to change? What would make these more effective? B.3 What are the persistent challenges relating to provision of quality services to KVPs across the country? Elaborations and probes: Please describe these challenges and what solutions are being considered? How is or how can SAUTI contribute to solutions to these challenges? B4. What are the factors enabling SAUTI’s successful achievement of its objectives and targets, and what factors are holding it back? Elaborations and probes: For example, what are the strengths and weaknesses in the project’s design? Are there challenges in how it is implemented that undermine its effectiveness or quality? Is the technical and management capacity within the project equal to the challenge from your perspective? Does it engage with other stakeholders optimally? Sustainability C.1 To what extent is SAUTI project setting the stage for sustainability? Elaborations and probes: For example, to what extent are the practices the project has put in place to improve access and quality of services likely to be continued by stakeholders after SAUTI closes? Are there any institutions that could play the role the project has taken on after SAUTI exits? Do you specific funding concerns that you can describe for us? C.2 Are there any challenges within the government to sustaining service provision to KVPs? Elaborations and probes: Can you describe these challenges? How can these challenges be overcome? 73 C.3 What should SAUTI do differently to ensure the sustained delivery of quality services to KVPs after it closes? Elaborations and probes: What efforts should the project should be making now? Are there efforts that other stakeholders could be making to improve prospects for sustainability? Lessons Learned D.1 What lessons have been learned so far that can be used to improve SAUTI for the remainder of the implementation period? D.2 What lessons have been learned that can be adopted by the public health community to improve services to KVPs going forward? Conclusion We would like to thank you for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. Group Interview/KII Protocol: Implementing Partner or Sub-Grantee at Local Level The following protocol is to be used with technical implementing staff at local level, whether the respondent(s) is (are) working directly for an implementing partner in the consortium or a local sub-grantee. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping beneficiaries, and how it can do better. In this discussion, we will ask you questions about the implementation of the project, its effectiveness, how it is managed, and your relationship with the development and other implementing partners. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. 74 Risks/Discomforts of Participating in this Study We have not identified any risks/discomforts, but you can also decline to answer any of my questions should they make you uncomfortable. If you do so, I will move on to the next question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. Any of you may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have questions for me at this time? If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Implementing partner organization and location: - Number of participants: - Designations12 of participants (Enter each participant’s designation separated by a comma): - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission turn on the tape recorder and begin the interview? 12 This is the role or job title of each individual participating. Do not document participant names in this table. 75 The Implementer’s Role and Relationship with SAUTI A.1 There are a number of partners implementing the SAUTI project. Describe the work your CSO is doing in SAUTI project? Elaborations and probes: For example, are you is your organization responsible for coordinating activities, delivering services directly, focusing on technical assistance etc.? How is the relationship between you and other SAUTI partners managed? A.2 How did this CSO became a partner to SAUTI? Elaborations and probes: For example, were you approached by a SAUTI implementer or a local government authority to join the project? Did you have to submit a proposal or go through a selection process? How is the relationship between you and other SAUTI partners managed? A.3 Can you describe how your relationship with SAUTI is managed? Elaborations and probes: For example, is there a contract or MOU in place? What and how frequently do you report to SAUTI? What kind of support does SAUTI provide you with? A.4 Has working with SAUTI led to changes that have strengthened your CSO? Elaborations and probes: For example, have you had to introduce policies and practices which recognize the health and related needs of KVPs? What other changes have been introduced and would you say these have improved the CSO? In what way? Working with Others B.1 How do you work with local government and community structures to do your work? Elaborations and probes: For example, do you coordinate with them to identify clients and/or deliver services? How do you do so? Are there risks associated with this coordination for clients? How do you manage such risks? Do you have examples of how coordinating with local government and community structures has improved project performance? B.2 How do you work with other CSOs and NGOs delivering related or similar services in this area? Elaborations and probes: For example, do you coordinate with them to identify clients and/or deliver services? How do you do so? Are there risks associated with this coordination for clients? How do you manage such risks? Do you have examples of how coordinating with other CSOs and projects has improved project performance? B.3 Do you ever engage KVPs in the design, planning or implementation of your KP-focused activities/interventions? Elaborations and probes: For example, do you consult KVPs on planning activities? Do KVPs assist in reaching more clients? Provision of Services to KVPs C.1 What challenges do you face in provide services to KVPs and other key populations? Elaborations and probes: For example, how do you go about finding key population members? Which key population members are most difficult to find? Is there resistance to delivering services to KVPs from local government or the community? If so, how has that effected your work? Is there resistance from KVPs to participating in the project? If so, how has that effected your work? C.2. How does the CSO manage client confidentiality? Elaborations and probes: Are there any SOPs or guidelines for ensuring the confidentiality of the clients? What are the challenges of maintaining confidentiality in close communities such as the one you work in? How successful are you at maintaining confidentiality? Can you describe any issues that have arisen and that you’ve had to manage related to the confidentiality of clients? C.3 We are very interested in how your volunteers manage to deliver services. Would you describe the work the volunteers are doing and how they are supported? 76 Elaborations and probes: Describe a typical working day for a volunteer? Give us an overview of their roles and responsibilities. Do you believe volunteers are equipped with appropriate skills, knowledge and resources to provide services to KVPs? How is the quality of their work reviewed and improved? What are the key challenges volunteers face in performing to the level required? C.4 What strategies have worked particularly well in the implementation of project activities? Elaborations and probes: What strategies have worked well to identify the right implementing sub-partners at local level? For example, are RHMTs/CHMTs engaged in selection of the CSOs? What strategies have worked for identifying and enrolling KVPs into the program? For example, has Hot Spot Mapping been effective? What strategies have worked for providing services to KVPs? For example, how well have outreaches worked? Are any SBCC approaches working well? How are referrals managed? Would you say the current processes are effective? How are QA/QI activities being coordinated? Are these activities proving effective? How are outreach cadres managed? Is there any system SAUTI has in place to assess and measure the quality of the outreach workers and addressing issues related? Successes and Lesson Learned D.1 What have been your organizations key achievements in implementing the Project? Elaborations and probes: Has the project improved access to services for KVPs? Has the project succeeded in changing behavior among KVPs (health seeking behaviors, testing, adherence or reducing risky behaviors)? Have you achieved the required targets for your various activities? What are the key areas where you have not achieved what you hoped to achieve in SAUTI KPs interventions? D.2 What lessons have been learned so far that can be used to improve SAUTI and improve activities like this in the future? Sustainability E.1 Do you think the provision of services to KVP will continue after the SAUTI project ends? Elaborations and probes: For example, to what extent are the practices the project has put in place to improve access and quality of services likely to be continued by stakeholders after SAUTI closes? Are there any institutions that could play the role the project has taken on after SAUTI exits? Would there be funding from other sources for such activities? E.2 What should SAUTI do differently to ensure the sustained delivery of quality services to KVPs after it closes? Elaborations and probes: What efforts should the project be making now? Are there efforts that other stakeholders could be making to improve prospects for sustainability? Conclusion We would like to thank you all for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. 77 Group Interview/KII Protocol: Implementing Partner National Level The following protocol is to be used with implementing partner staff at national level. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping beneficiaries, and how it can do better. In this discussion, we will ask you questions about the implementation of the project, its effectiveness, how it is managed, and your relationship with the development and other implementing partners. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Risks/Discomforts of Participating in this Study We have not identified any risks/discomforts, but you can also decline to answer any of my questions should they make you uncomfortable. If you do so, I will move on to the next question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. Any of you may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have questions for me at this time? If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. 78 Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Implementing partner organization and location: - Number of participants: - Designations13 of participants (Enter each participant’s designation separated by a comma): - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission turn on the tape recorder and begin the interview? 13 This is the role or job title of each individual participating. Do not document participant names in this table. 79 The Implementing Partner’s Role in SAUTI A.1 There are a number of partners implementing the SAUTI project. Could you please describe the distinct role you play as an implementing partner, and the roles played by the other IPs? Elaborations and probes: For example, are you the prime or a sub-partner? To what extent is your organization responsible for coordinating activities, delivering services directly, focusing on technical assistance etc.? How are these roles distributed across members of the consortium? Importantly, how is the relationship between partners managed? A.2 How has SAUTI been working with other national stakeholders with an interest in addressing the needs of KVPs? Elaborations and probes: Can you offer examples of SAUTI’s coordinating with other stakeholders, such as participation in common forums or joint activities? Can you recount any success stories involving collaboration between SAUTI and other stakeholders? Could the project be engaging with other stakeholders more effectively, and how? The Implementation Context B.1 Can you describe the political, legal and policy environment in which SAUTI and projects like it are attempting to provide services to KVPs? Elaborations and probes: How do the political, legal and policy environments enable or constrain the provision of services to KVPs? How does the environment impact on government’s readiness to provide services to KVPs? For example, is budget assigned for such purposes? In addition to the policy environment, what are the key issues for government delivery of services to KVPs? B.2 Could you describe how the National Guideline for Comprehensive Package of HIV Interventions for Key Populations (2017) came to be developed and its anticipated impact on services to KVPs? Elaborations and probes: What factors prompted the revision of the 2014 National KVP guideline? Has the implementation of the 2017 guideline been successful? Why or why not? What role did SAUTI play in the development and rollout of the 2017 guideline? Are any SAUTI components being integrated into the National KVP guideline? Effectiveness of Services Provision to KVPs C.1 From your perspective, how well does the project address KVP related priorities? Elaborations and probes: What are the KVP related priorities in Tanzania (supporting and obstructing factors for accessing health care), and is SAUTI addressing those? What are the persistent challenges relating to provision of quality services to KVPs across the country (stigma, commodity shortages, etc.) and is SAUTI addressing those? How well does the project align with the implementation of the National Guideline for a Comprehensive Package of HIV Interventions for Key and Vulnerable Populations (2017)? C.2 To what extent has the SAUTI project progressed towards achieving its objectives? Elaborations and probes: Has the project improved access to services for KVPs? Why do you say so? Has the project succeeded in changing behavior among KVPs (health seeking behaviors, testing, adherence or reducing risky behaviors)? Has the project provided the necessary support to strengthen government structures and initiatives? Why do you say so? Has the project provided the necessary support to strengthen civil society or community structures and initiatives? Why do you say so? Note any references to sources of evidence and request access to these at end of interview. C.3 What would you say have been the crucial factors facilitating successful progress towards achieving the projects objectives? Elaborations and probes: 80 Is it design? Was the original design of SAUTI appropriate for achieving objectives, given the context in which you’ve been implementing the project? Is it implementation fidelity? Has the project been implemented according to plan, or have you had to deviate in any significant way, and how has that influenced the achievement of results? Is it organizational capacity? How important a role has the organizational design (multiple IPs, subs and local CSOs) and organizational capacity of the SAUTI consortium played in the project’s achievements (planning, management, coordination, budgeting)? Is it implementation quality? Have you had any issues with the quality of implementation and how has that influenced achieving results? Is it multi-stakeholder coordination? How important has coordination and/or collaboration with stakeholders such as government (NACP, NIMR), development partners, other projects proven to be? C.4 What strategies have worked particularly well in the implementation of project activities? Elaborations and probes: What strategies have worked well to identify the right implementing sub-partners at local level? For example, are RHMTs/CHMTs engaged in selection of the CSOs? What strategies have worked for identifying and enrolling KVPs into the program? For example, has Hot Spot Mapping been effective? What strategies have worked for providing services to KVPs? For example, how well have outreaches worked? Are any SBCC approaches working well? How are referrals managed? Would you say the current processes are effective? How are QA/QI activities being coordinated? Are these activities proving effective? How are outreach cadres managed? Is there any system SAUTI has in place to assess and measure the quality of the outreach workers and addressing issues related? Other Factors Impacting Implementation and Results D.1 How has integrating DREAMS activities into implementation impacted on SAUTI? Elaborations and probes: For example, has it improved the achievement of project objectives? Has it introduced administrative burdens or implementation demands that have hindered the achievement of project objectives? D.2 How effective has USAID’s oversight of the project proven to be? Elaborations and probes: For example, has USAID contributed technically to improving the project and achieving its objectives? Have USAID demands hindered project implementation and the achievement of objectives? How can USAID’s contribution to project effectiveness be strengthened? Sustainability E.1 To what extent is SAUTI project setting the stage for sustainability? Elaborations and probes: For example, to what extent are the practices the project has put in place to improve access and quality of services likely to be continued by stakeholders after SAUTI closes? Are there any institutions that could play the role the project has taken on after SAUTI exits? Do you have specific funding concerns that you can describe for us? E.2 Are there any challenges within the government to sustaining service provision to KVPs? Elaborations and probes: Can you describe these challenges? How can these challenges be overcome? E.3 What should SAUTI do differently to ensure the sustained delivery of quality services to KVPs after it closes? Elaborations and probes: What efforts should the project should be making now? Are there efforts that other stakeholders could be making to improve prospects for sustainability? Lessons Learned F.1 What lessons have been learned so far that can be used to improve SAUTI for the remainder of the implementation period? 81 F.2 What lessons have been learned that can be adopted by the public health community to improve services to KVPs going forward? Conclusion We would like to thank you all for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. Group Interview/KII Protocol: National Level Stakeholder The following protocol is to be used with stakeholders14 at national level. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping beneficiaries, and how it can do better. In this discussion we will ask you questions about the implementation of the project, its effectiveness, how it is managed, and your relationship with the development and implementing partners. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Risks/Discomforts of Participating in this Study We have not identified any risks/discomforts, but you can also decline to answer any of my questions should they make you uncomfortable. If you do so, I will move on to the next question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw 14 Non-government entities with interest in but not directly involved in SAUTI, such as development donors. 82 The decision to participate in this study is entirely up to you. Any of you may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have questions for me at this time? If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Stakeholder organization and location: - Number of participants: - Designations15 of participants (Enter each participant’s designation separated by a comma): - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - Do I have your permission turn on the tape recorder and begin the interview? 15 This is the role or job title of the individual participating. Do not document participant name in this table. 83 SAUTI and its Relationship with the Stakeholder A.1 How familiar are you with the SAUTI project? Elaborations and probes: Could you describe its goals and objectives as you understand them? Could you describe what you know about how it’s implemented nationally? A.2 Does your organization play a role in delivering services to KVPs? Elaborations and probes: Could you describe any direct service delivery efforts your organization is involved in? Could you describe any other efforts, such as advocacy or oversight activities? Could you explain any interest your organization has in the delivery of services to KVPs? A.3 Can you describe the nature of the relationship between SAUTI and your organization? Elaborations and probes: To what extent have you been consulted on any SAUTI related matters? Has your organization ever had a role in planning or implementing SAUTI activities? Have you ever played an oversight role for the project? A.4 How effectively has SAUTI been in working with other national stakeholders with an interest in addressing the needs of KVPs? Elaborations and probes: Can you offer examples of SAUTI’s coordinating with other stakeholders, such as participation in common forums or joint activities? Can you recount any success stories involving collaboration between SAUTI and other stakeholders? Could the project be engaging with other stakeholders more effectively, and how? The Implementation Context B.1 Can you describe the political, legal and policy environment in which SAUTI and projects like it are attempting to provide services to KVPs? Elaborations and probes: How do the political, legal and policy environments enable or constrain the provision of services to KVPs? How does the environment impact on government’s readiness to provide services to KVPs? For example, is budget assigned for such purposes? B.2 How are the challenges of the legal and policy environment being addressed by those trying to provide services to KVPs? Elaborations and probes: For example, are there or have there been advocacy efforts underway to improve the environment to enable better service for KVPs? Are there innovative ways in which providers are working to deliver services within the current constraints? B.3 How has the prevailing political, legal and policy environment impacted on government’s provision of services to KVPs? Elaborations and probes: For example, have services been halted or suspended? Have KVPs been discouraged from seeking services? Have KVPs been prioritized as a client group for government services? In addition to the policy environment, what are the key issues for government delivery of services to KVPs? B.4 Could you describe how the National Guideline for Comprehensive Package of HIV Interventions for Key Populations (2017) came to be developed and its anticipated impact on services to KVPs? Elaborations and probes: What factors prompted the revision of the 2014 National KVP guideline? Has the implementation of the 2017 guideline been successful? Why or why not? What role did SAUTI play in the development and rollout of the 2017 guideline? Are any SAUTI components being integrated into the National KVP guideline? Effectiveness of Services Provision to KVPs C.1 From your perspective, how well does the project address KVP related priorities? Elaborations and probes: What are the KVP related priorities in Tanzania, and is SAUTI addressing those? What are the persistent challenges relating to provision of quality services to KVPs across the country, and is SAUTI addressing those? How well does it align with your implementation of the National Guideline for a Comprehensive Package of HIV Interventions for Key and Vulnerable Populations (2017)? 84 C.2 To what extent has the SAUTI project helped the Government of Tanzania achieve its objective of providing quality health services to the KVPs? Elaborations and probes: For example, would you say that SAUTI has improved access to services for KVPs? Why do you say so? Has the project provided the necessary support to strengthen government structures and initiatives? Why do you say so? Has the project provided the necessary support to strengthen civil society’s or community structures and initiatives? Why do you say so? Sustainability D.1 To what extent is SAUTI project setting the stage for sustainability? Elaborations and probes: For example, to what extent are the practices the project has put in place to improve access and quality of services likely to be continued by stakeholders after SAUTI closes? Are there any institutions that could play the role the project has taken on after SAUTI exits? Do you have specific funding concerns that you can describe for us? D.2 Are there any challenges within the government to sustaining service provision to KVPs? Elaborations and probes: Can you describe these challenges? How can these challenges be overcome? D.3 What should SAUTI do differently to ensure the sustained delivery of quality services to KVPs after it closes? Elaborations and probes: What efforts should the project should be making now? Are there efforts that other stakeholders could be making to improve prospects for sustainability? Lessons Learned E.1 What lessons have been learned so far that can be used to improve SAUTI for the remainder of the implementation period? E.2 What lesson have been learned that can be adopted by the public health community to improve services to KVPs going forward? Conclusion We would like to thank you for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask. Group Interview Protocol: Regional/Council Health Management Teams The following protocol is to be used with staff of regional and council health management teams. INTRODUCTION AND CONSENT Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the SAUTI program. We are here to find out whether the project is helping beneficiaries, and how it can do better. In this discussion, we will ask you questions about the project and project beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the SAUTI project. During our conversation we would like to take notes and to record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you share with us today. Your ideas, together with other data, will be used to produce a report. Findings from this evaluation will be used to improve the program as far as possible. Your answers will not impact any aid you or your area will receive in the future. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. 85 Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the project is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Risks/Discomforts of Participating in this Study We have not identified any risks/discomforts, but you can also decline to answer any of my questions should they make you uncomfortable. If you do so, I will move on to the next question. Benefits of Participating in this Study There is no direct benefit to you for being in this study, but we hope that the results of our study will help improve the SAUTI project and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. Any of you may refuse to take part in the study at any time without affecting your relationship with the evaluation team or the SAUTI project. As mentioned you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. You will indicate if the information you have provided up to the time of withdrawal can be retained and analyzed, or if you prefer that this information is destroyed or removed from analysis. Right to Ask Questions and Report Concerns You have the right to ask questions about this research study and to have those questions answered by me before, during or after the research. Does anyone have questions for me at this time? If you have any further questions about the study feel free to contact Daud Siwalaze at [dsiwalaze@engl.com] or by telephone at 0758 067 700. Consent Do you agree to participate in this study? Please fill out the metadata table below. Date (dd/mm/yyyy): - Interviewer’s name: - Government Department/Agency and location: - Number of participants: - Designations16 of participants (Enter each participant’s designation separated by a comma): - Start time (24 h clock, hh:mm): - End time (24 h clock, hh:mm): - 16 This is the role or job title of each individual participating. Do not document participant names in this table. 86 Do I have your permission turn on the tape recorder and begin the interview? 87 Local Service to KVPs A.1 Can you describe any health services being provided by government facilities that specifically targeted at KVPs in this area? Elaborations and probes: Do you have programs targeting KPs in your Council comprehensive plan for health (CCHP)? If so, what are the specific interventions? Are there any issues with the government delivering services to KVPs? Could you describe any challenges with policies and laws in creating enabling environment for service provision to KPs, and if these are being addressed? A.2 Are there particular barriers or obstacles that hinder KVP access to health and HIV services? Elaborations and probes: For example, is there resistance in facilities to delivering services to KVPs? Is there reluctance on the part of KVPs to seek services? Are any KVPs particularly hard to reach? What is the current situation of the KVPs in treatment lost to follow up? What other challenges are impeding service provision to KPs? How are these barriers being addressed? A.3 What essential strategies for an enabling environment to servicing key populations is Government implementing? Elaborations and probes: For example, is the implementation of the KVP National Guideline 2017 proving successful? What other approaches are being adopted by government? SAUTI Collaboration and Strengthening of Local Structures B.1 Describe the role of RHMTs/CHMTs in the project. Elaborations and probes: For example, how involved have you or other local structures (such as RHMTs and CHMTs) been in SAUTI’s design, its planning, its budgeting, its implementation, or its oversight? To what extent are you consulted on project related matters? For example, have RHMTs/CHMTs been in selection of the CSOs who work with SAUTI? B.2 How effective has SAUTI support to local structures been? Elaborations and probes: Has SAUTI support to strengthen RHMTs and CHMTs been effective? Has the project done enough to support strengthening other local government entities been sufficient? What about its support to community and civil society? B.3 Has SAUTI been hindered by any negativity towards KVPs in community structures or local organizations? Elaborations and probes: How prevalent are negative perspectives towards KVPs in communities and has this hindered delivery of services to these groups in any way? Are these prevalent in local government structures too, such as facilities? How has the project and its partners tried to ensure delivery of quality services under these conditions? Have these efforts been successful? B.4 Are there organizations other than the SAUTI project implementing KVP-focused activities in this area? Elaborations and probes: Which other projects or organizations? In what ways and how well are there activities coordinated? How do they complement one another? How is SAUTI doing things differently? What is its niche? Effectiveness of Services Provision to KVPs C.1 How has the SAUTI project helped achieve local objectives for providing quality health services to the KVPs? Elaborations and probes: For example, would you say that SAUTI has improved access to services for KVPs? Has the project provided the necessary support to strengthen local government structures and initiatives? Why do you say so? Has the project provided the necessary support to strengthen civil society’s or community structures and initiatives? Why do you say so? 88 C.2 What strategies have worked particularly well in the implementation of project activities? Elaborations and probes: What strategies have worked well to identify the right implementing sub-partners at local level? For example, are RHMTs/CHMTs engaged in selection of the CSOs? What strategies have worked for identifying and enrolling KVPs into the program? For example, has Hot Spot Mapping been effective? What strategies have worked for providing services to KVPs? For example, how well have outreaches worked? Are any SBCC approaches working well? How are referrals managed? Would you say the current processes are effective? How are QA/QI activities being coordinated? Are these activities proving effective? Is SAUTI supporting RHMTs/CHMTs and health services in other areas such as commodity supply? What is working well? C.3 We are very interested in how your volunteers manage to deliver services. Would you describe the work the volunteers are doing and how they are supported? Elaborations and probes: Do you believe volunteers are equipped with appropriate skills, knowledge and resources to provide services to KVPs? How is the quality of their work reviewed and improved? What are the key challenges volunteers face in performing to the level required? C.4 Is client confidentiality effectively protected? Elaborations and probes: Are there any SOPs or guidelines for ensuring the confidentiality of KVP clients? What are the challenges of maintaining confidentiality in close communities such as the one you work in? How successful is SAUTI at maintaining confidentiality? Can you describe any issues that have arisen and that you’ve had to manage related to the confidentiality of clients? Sustainability D.1 To what extent is SAUTI project setting the stage for sustainability? Elaborations and probes: For example, has SAUTI developed and implement the Sustainability Plan? To what extent are the practices the project has put in place to improve access and quality of services likely to be continued by stakeholders after SAUTI closes? Has enough capacity been built in local community and civil society structures? Do you have specific funding concerns that you can describe for us? D.2 Are there any challenges within the local government structures to sustaining service provision to KVPs? Elaborations and probes: Can you describe these challenges? How can these challenges be overcome? D.3 What should SAUTI do differently to ensure the sustained delivery of quality services to KVPs after it closes? Elaborations and probes: What efforts should the project should be making now? Are there efforts that other stakeholders could be making to improve prospects for sustainability? D.4 In your opinion, do you see continuity of the KP interventions if there is no donor led support? Lessons Learned E.1 What lessons have been learned so far that can be used to improve SAUTI for the remainder of the implementation period? Elaborations and probes: For example, what have been the key lessons learned in implementation of SBCC and Gender component of this project? E.2 What lesson have been learned that can be adopted by future initiatives to improve services to KVPs going forward? Elaborations and probes: What are the key elements you would like to see in a follow-on USAID KP project? 89 Conclusion We would like to thank you all for participating in this conversation. Sometimes these conversations can be difficult and sometimes you attend when it is difficult or inconvenient for you. We truly appreciate your effort and the valuable information you’ve given us. We’ve learned a lot from you and we will use this information to help SAUTI improve what they do. If you have any questions for us, please feel free to ask 90 ANNEX IV: LIST OF KII PARTICIPANTS 91 KEY INFORMANTS INTERVIEWED Name Title District Gender Sr. Mboga District Reproductive and Child Health Coordinator(DRCHCO) Kinondoni MC F Catherine Mbeyela District Reproductive and Child Health Coordinator(DRCHCO) Temeke MC F Prisca Butuyuyu Regional Reproductive and Child Health Coordinator(RRCHCO) Mbeya F Tupoke G. Mwakalobo District Reproductive and Child Health Coordinator(DRCHCO) Kyela DC F Atu Nyondo Nurse Kyela DC F Not Disclosed District AIDS Control Coordinator(DACC) Kinondoni MC F Hawa Kilasi District Reproductive and Child Health Coordinator(DRCHCO) Mbarali DC F WASO management CSO Management-WASO Dar es Salaam 3M, 3F Dr. Kazaura Koku Senior Branch Chief - Prevention-CDC Dar es Salaam M Neema Makyao NACP KP Coordinator Dar es Salaam F Not Disclosed Nurses Temeke F,F Not Disclosed Clinicians Temeke F,F Joseph Mapunda CSO Program Director-MUKIKUTE Temeke M Dr. Manyatta Regional Medical Officer(DMO) Mbeya M Emmanuel Petro Regional Coordinator for TACAIDS(RCT) Mbeya M Sarah K, et all CSO Program Officer-PHSRF Kinondoni F Not Disclosed Nurse Kinondoni F Not Disclosed Council HIV and AIDS Coordinator(CHAC) Kinondoni F Dr. Hance Mpumilwa District AIDS Control Coordinator(DACC) Mbarali M Dr. Einhard Mlelwa District AIDS Control Coordinator(DACC) Kyela M Amani Flexon Senior Technical Staff-NIMR Dar es Salaam M Nguhuni HIV/TB and Global Fund Coordination Dodoma F Kyela CHMT CHMT Members Kyela DC 4M, 2F Thomas Kipingili Ag. COP Dar es Salaam M Amina Shaban, et al Economic Strengthening Officer Dar es Salaam 2F Albert Komba Chief of Party-SAUTI Dar es Salaam M Shinyanga RHMT RHMT Members Shinyanga 5M Mbeya RHMT RHMT Members led by Dr. Lutaragula Masili Mbeya 1M, 3F Shinyanga Facility Level Staff Nurses Shinyanga 3F Roman Kessy, Angubatile Seme CHAC and District Community Development Officer(DCDO) Mbarali DC M,F Kilamile, Dr. Irumba Nurse and Clinician Mbarali M, F Mbeya HIV/AIDS Network Tanzania CSO Staff-MHNT Mbarali 2M, 4F Rafiki SDO CSO Staff-Rafiki SDO Shinyanga MC 4M, 3F Shinyanga Facility Level Staff Clinicians Shinyanga MC M, 2F Shinyanga CHMT CHMT Members Shinyanga MC 4M, 11F Amos Kayembele, Levi Kasitu CHAC and District Community Development Officer(DCDO) Kyela DC 2M KIWOHEDE CSO Staff-KIWOHEDE Kyela DC 5M, 2F 92 Name Title District Gender Gregory Kamugisha, et al Advocacy and Networking Manager-NACOPHA Dar es Salaam 2M, 2F Temeke CHMT CHMT Members Dar es Salaam 14M, 17F Kinondoni CHMT CHMT Members Dar es Salaam 12M, 10F Dar es Salaam RHMT Members Dar es Salaam 7M,7F Total 166 FOCUS GROUP PARTICIPANTS FGD Participants by Gender Type of Stakeholder Gender Total Participants Title District Male Female vAGYW Kyela DC 0 12 12 vAGYW Kyela DC 0 16 16 Community-Based health Service Providers/Empowerment Workers Kyela DC 0 14 14 FSW Kyela DC 0 16 16 FSW Kyela DC 0 8 8 FSW Shinyanga MC 0 15 15 FSW Shinyanga MC 0 22 22 vAGYW Mbarali DC 0 12 12 vAGYW Mbarali DC 0 11 11 FSW Mbarali DC 0 12 12 FSW Mbarali DC 0 12 12 Community-Based health Service Providers/Empowerment Workers Temeke MC 4 17 21 Community-Based health Service Providers/Empowerment Workers Kinondoni MC 0 7 7 FSW - 0 23 23 vAGYW Kinondoni MC 0 20 20 vAGYW Shinyanga MC 0 18 18 vAGYW Shinyanga MC 0 16 16 vAGYW Temeke MC 0 14 14 Community-Based health Service Providers/Empowerment Workers Temeke MC 4 7 11 FSW Temeke MC 0 20 20 MSM Temeke MC 20 0 20 vAGYW Temeke MC 0 20 20 MSM Mbarali DC 8 0 8 Community-Based health Service Providers/Empowerment Workers Mbarali DC 0 8 8 MSM Kyela DC 8 0 8 FSW Kyela DC 0 3 3 Community-Based health Service Providers/Empowerment Workers Shinyanga MC 0 3 3 FSW Shinyanga MC 0 3 3 MSM Temeke MC 5 0 5 MSM Kinondoni MC 3 0 3 93 FGD Participants by Gender Type of Stakeholder Gender Total Participants Title District Male Female vAGYW Kinondoni MC 0 5 5 Total 52 334 386 94 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST 95 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name NORAH JONATHAN KAAYA Title SOCIAL SCIENTIST DATA COLLECTOR Organization Mendez England & Associates (ME&A) Evaluation Position Team member Evaluation Award Number(contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated(Include project name(s), implementer name(s) and award number(s), if applicable) USAID SAUTI Project. I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. - I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 15/09/2018 96 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name Daud Siwalaze Title Monitoring, Evaluation and Learning Specialist Organization NORC at the University of Chicago(Data for Development) Evaluation Position? Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) SAUTI-Tz; JHPIEGO Corporation; AID-621-A-15-00003 I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. - I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 15, 2018 97 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS Name Bahati P. Tenga Title STTA – Mid -Term Performance Evaluation of USAID Funded SAUTI Project Organization Mendez England & Associates (ME&A) Evaluation Position? Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) SAUTI-Tz; JHPIEGO Corporation; AID-621-A-15-00003 I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. - I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 10 September 2018 98 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS 99 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS 100 DISCLOSURE OF CONFLICT OF INTEREST FOR USAID EVALUATION TEAM MEMBERS 101 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523