EVALUATION MID-TERM PERFORMANCE EVALUATION OF THE USAID/TANZANIA COMMUNITY HEALTH AND SOCIAL WELFARE SYSTEMS STRENGTHENING PROGRAM May 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the ME&A Data for Development Program. USAID/TANZANIA MID-TERM PERFORMANCE EVALUATION OF THE COMMUNITY HEALTH AND SOCIAL WELFARE SYSTEMS STRENGTHENING PROGRAM Submitted: May 7, 2019 Prepared by: Gary Woller, Team Lead, ME&A Jake Laden, Evaluation Advisor, NORC Nasson Konga, Monitoring and Evaluation Specialist, NORC Gerald Usika, Survey Specialist, NORC Michelle Davis, Research Director 1, NORC Letitia Onyango, Senior Research Analyst, NORC Daud Siwalaze, Monitoring and Evaluation Specialist, NORC Submitted by: David Hughes, Acting Chief of Party ME&A USAID/Tanzania Data for Development Program Plot 122 Mwai Kibaki Road, Mikocheni A, Dar es Salaam, Tanzania Email: dhughes@engl.com Cover Photo: Community Case Workers in Bukoba, Kagera Credit: Daud Siwalaze, Data for Development Program DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government ABSTRACT The Community Health and Social Welfare Systems Strengthening Program (CHSSP) is a five-year USAID and President’s Emergency Plan for AIDS Relief (PEPFAR)-funded program implemented by John Snow Inc. and World Education, Inc. The program works with the Government of Tanzania (GoT) and other health and social system actors to achieve two objectives: 1) higher performing human resources for community health and social welfare services; and 2) more functional, better coordinated community structures and systems to better serve key and vulnerable populations. The purpose of the evaluation is to: 1) assess program performance and achievements; 2) analyze the effectiveness of program implementation; and 3) document good practices, gaps/limitations, and lessons. The evaluation concludes CHSSP demonstrated a flexible management approach and took effective measures to address the multiple changes to its program description to stay on track and meet key indicator targets each program year. In addition, CHSSP made important contributions to strengthening: 1) the health and social welfare enabling environment; 2) human resource capacity at the national, regional, council, ward, and village levels; and 3) community structures and systems for providing community health and social welfare services to priority and key populations. Several threats, however, exist to the ongoing functioning and institutionalization of the contributions made by CHSSP. While the program has taken a number of steps to address these threats, any follow￾on program will need to consider how to consolidate gains achieved during CHSSP and move them toward institutionalization within the GoT and the relevant community-level structures and systems. CONTENTS EXECUTIVE SUMMARY................................................................................................................................................... i 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS...................................................................... 1 1.1 Evaluation Purpose ............................................................................................................................................. 1 1.2 Evaluation Questions......................................................................................................................................... 1 2.0 PROGRAM BACKGROUND.................................................................................................................................. 2 2.1 CHSSP Goals and Objectives........................................................................................................................... 2 2.2 CHSSP Approach................................................................................................................................................ 3 2.3 Revisions to the CHSSP Program Description............................................................................................ 3 3.0 EVALUATION METHODS AND LIMITATIONS .............................................................................................. 4 3.1 Data Collection Methods.................................................................................................................................. 4 3.2 Data Analysis Methods...................................................................................................................................... 4 3.3 Methodological Limitations.............................................................................................................................. 5 4.0 FINDINGS AND CONCLUSIONS ....................................................................................................................... 5 4.1 EQ 1: What have been the strengths and weaknesses of CHSSP implementation?........................... 5 4.1.1 Findings........................................................................................................................................................ 5 4.1.2 Conclusions ................................................................................................................................................ 7 4.2 EQ 1.1: Which factors have affected implementation and how?............................................................. 7 4.2.1 Findings........................................................................................................................................................ 7 4.2.2 Conclusions ................................................................................................................................................ 8 4.3 EQ 1.2: How did the program address factors that affected implementation and with what results?................................................................................................................................................................... 8 4.3.1 Findings........................................................................................................................................................ 8 4.3.2 Conclusions ................................................................................................................................................ 9 4.4 EQ 2: To what extent has CHSSP contributed to creating higher performing human resources for providing community health and social welfare services to priority and key populations, particularly AGYW, MVC, and PLHIV?....................................................................................................... 10 4.4.1 Findings...................................................................................................................................................... 10 4.4.2 Conclusions .............................................................................................................................................. 12 4.5 EQ 2.1: What challenges are health and social welfare actors facing in fulfilling or improving their roles in providing community health, social welfare, and HIV prevention and treatment services to key populations?........................................................................................................................................... 12 4.5.1 Findings...................................................................................................................................................... 12 4.5.2 Conclusions .............................................................................................................................................. 15 4.6 EQ 2.2: What can CHSSP and other programs, including Kizazi Kipya, do better to assist health and social welfare actors to address these challenges?........................................................................... 15 4.6.1 Findings...................................................................................................................................................... 15 4.6.2 Conclusions .............................................................................................................................................. 16 4.7 EQ 2.2: How did CHSSP support address issues related to women, girls, and youth, including, for example, factors limiting their access to community health and social welfare services and/or the quality of community health and social welfare services they receive and with what results? ............................................................................................................................................................................... 16 4.7.1 Findings...................................................................................................................................................... 16 4.7.2 Conclusions .............................................................................................................................................. 17 4.8 EQ 3: To what extent has CHSSP contributed to creating an improved enabling environment and more functional and better coordinated community structures and systems for providing community health and social welfare services to priority and key populations, particularly AGYW, MVC, and PLHIV?............................................................................................................................. 18 4.8.1 Findings...................................................................................................................................................... 18 4.8.2 Conclusions .............................................................................................................................................. 21 4.9 EQ 4: To what extent will CHSSP activities and results be sustained after the program ends? Which factors affect the prospects for sustainability and how?............................................................ 21 4.9.1 Findings...................................................................................................................................................... 21 4.9.2 Conclusions .............................................................................................................................................. 28 5.0 RECOMMENDATIONS.......................................................................................................................................... 28 ANNEXES ......................................................................................................................................................................... 31 Annex I: Evaluation Statement of Work............................................................................................................. 32 Annex II: Bibliography ............................................................................................................................................. 38 Annex III: KII and FGD Discussion Guides........................................................................................................ 39 Annex IV: KIIs, FGD, and CCW Survey Sampling Methodology.................................................................. 60 Annex V: CCW Survey Questionnaire............................................................................................................... 62 Annex VI: CCW Survey Results........................................................................................................................... 69 Annex VII: KII and FGD Participants by Stakeholder and Location............................................................. 76 Annex VIII: List of KII Participants....................................................................................................................... 78 Annex IX: Disclosure of Any Conflicts of Interest.......................................................................................... 81 LIST OF TABLES Table 1: CHSSP Performance Indicators – Targets and Actuals by Fiscal Year................................................. 6 Table 2: Policies and Guidelines Revised, Developed, or Disseminated by CHSSP........................................ 10 Table 3: Knowledge and Skills Applied by CCWs vs. Training Received........................................................... 11 Table 4: Perceived CCW Performance...................................................................................................................... 12 Table 5: CHSSP Achievements in Improving the Health and Social Welfare Enabling Environment.......... 18 Table 6: Maturity Levels Scores and Descriptions................................................................................................... 20 Table 7: CHSSP Activities Transitioned to the Government of Tanzania ......................................................... 26 LIST OF FIGURES Figure 1: Primary Challenges Faced by CCWs......................................................................................................... 13 Figure 2: Primary Reasons to Continue Working as CCWs Over the Long Term........................................ 27 Figure 3: Primary Reasons Why CCWs May Not Continue Working as CCWs Over the Long Term... 27 ACRONYMS Acronym Description ADS Automated Directives System AGYW Adolescent Girls and Young Women AIDS Acquired Immune Deficiency Syndrome ART Antiretroviral Therapy CC City Council CCW Community Case Worker CDO Community Development Officer CHAC Council HIV and AIDS Coordinator CHMT Council Health Management Team CHSSP Community Health and Social Welfare Systems Strengthening Program CHW Community Health Worker CMAC Council Multi-Sectoral AIDS Committee CMC Case Management Coordinator COCODA Community Concerns of Orphans and Development Association CSO Civil Society Organization DACC District AIDS Control Coordinator DANIDA Danish International Development Agency DC District Council DCDO District Community Development Officer DMO District Medical Officer DMT District Master Trainer DSWO District Social Welfare Officer EQ Evaluation Question ET Evaluation Team FGD Focus Group Discussion FY Fiscal Year GBV Gender-Based Violence GI Group Interview GoT Government of Tanzania HIV Human Immunodeficiency Virus IAHA American International Health Alliance Acronym Description IP Implementing Partner ISW Institute of Social Work JSI JSI Research & Training Institute KII Key Informant Interview KVP Key and Vulnerable Population LCCW Lead Community Case Worker LGA Local Government Authority LOP Life of Program M&E Monitoring and Evaluation MAC Multi-Sectoral AIDS Committee MC Municipal Council MOCSO Mwanza Outreach Care and Support Organization MOHCDGEC Ministry of Health, Community Development, Gender, Elderly, and Children MVC Most Vulnerable Children MVCC Most Vulnerable Children’s Committee NACOPHA National Council of People Living with HIV NASW National Association of Social Workers NICMS National Integrated Case Management System NIMR National Institute for Medical Research NPA National Plan of Action OCAT Organizational Capacity Assessment Tool OVC Orphans and Vulnerable Children PD Program Description PEPFAR President’s Emergency Plan for AIDS Relief PLHIV People Living with HIV PO-RALG President’s Office for Regional Administration and Local Government PSW Para-Social Worker PY Program Year RCT Regional Coordinator for TACAIDS RCDO Regional Community Development Officer RSWO Regional Social Welfare Officer Acronym Description SOW Statement of Work SWO Social Welfare Officer TACAIDS Tanzania Commission for AIDS TADEPA Tanzania Development and AIDS Prevention TC Town Council ToT Training-of-Trainers USAID United States Agency for International Development VAC Violence Against Children VAWC Violence Against Women and Children VAWC-PC Violence Against Women and Children Protection Committee WCDO Ward Community Development Officer VMAC Village Multi-Sectoral AIDS Committee WEI World Education, Inc. WMAC Ward Multi-Sectoral AIDS Committee WSWO Ward Social Welfare Officer i EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS The purpose of this final performance evaluation of the United States Agency for International Development (USAID) Community Health and Social Welfare Systems Strengthening Program (CHSSP) is to: 1) assess program performance and achievements; 2) analyze the effectiveness of various approaches used by the program implementing to achieve program results and foster sustainability; and 3) document good practices, gaps/limitations, and lessons learned to inform the Mission’s design of a potential follow￾on program, or critical approaches and interventions. The evaluation seeks to answer the following four evaluation questions found below in the section Findings and Conclusions. PROGRAM BACKGROUND CHSSP is a six-year (2014-2020)1 USAID and President’s Emergency Plan for AIDS Relief (PEPFAR)-funded program implemented by JSI Research & Training Institute (JSI) and World Education, Inc. (WEI). The program works with the Government of Tanzania (GoT) and other health and social system actors to achieve the following two objectives. Objective 1: Higher performing human resources for community health and social welfare services. Objective 1 includes two strategies: 1) work with the GoT to update and finalize policies and provide technical guidance to incorporate PEPFAR 3.0 priorities; and 2) roll out the implementation of the country’s first national integrated case management system (NICMS). Objective 2: More functional, better coordinated community structures and systems to better serve key and vulnerable populations (KVPs). This objective seeks to create an enabling policy environment and strengthen community structures, including multi-sectoral AIDS committees (MACs), violence against women and children protection committees (VAWC-PCs), people living with HIV (PLHIV) clusters, and civil society organizations (CSOs). EVALUATION METHODS The evaluation team (ET) employed a mixed-methods data collection design that included a document review, key informant interviews (KIIs), focus group discussions (FGDs), a community case worker (CCW) survey, and a review of CHSSP performance data. Overall, the ET conducted 34 KIIs with 107 individuals (65 women and 42 men) and 14 FGDs with 142 individuals (73 women and 69 men). For the CCW survey, the ET conducted a phone survey of 269 randomly-selected CCWs (148 women and 121 men). FINDINGS AND CONCLUSIONS EQ 1: What have been the strengths and weaknesses of CHSSP implementation? • CHSSP managed to cascade down to lower local government levels over time, although later than foreseen, largely achieving targets at all levels of the cascade by the end of program year (PY) 4. The delays experienced in PY 3 were due to an increase in the number of councils (and therefore wards and villages) that CHSSP was required to reach as part of changes in the program description (PD) 3, stretching program resources. • Changes to scale and scope to the CHSSP PD in PY 3 and PY 4 created significant implementation challenges. This was a necessary adaptation to pivot to support the national case management system and the inception of Kizazi Kipya. CHSSP demonstrated a flexible management approach 1 CHSSP was initially a five-year program for 2014-2019. It received a one-year extension through November of 2020. ii and took effective measures to address the challenges such that it was able to stay on track and meet most indicator targets in PY 4 (Fiscal Year [FY] 18). EQ 2: To what extent has CHSSP contributed to creating higher performing human resources for providing community health and social welfare services to priority and key populations, particularly adolescent girls, youth, and women (AGYW), most vulnerable children (MVC), and PLHIV? • CHSSP contributed to revising, developing, and disseminating policies and guidelines that together have provided an important framework for facilitating ongoing reforms in and improved functioning of the community health and social welfare structure and system. • CHSSP contributed to building the capacity of CCWs by providing them knowledge and skills to do their jobs and by educating them on the case management system operations, guidelines, and tools. • CHSSP contributed to strengthening the community health and social welfare system in terms its operations and outcomes through its support for developing the NICMS and training/supporting CCWs. CCWs reported improvements in their ability to make referrals to health and social welfare services and linkages to treatment and care at facilities. • Principal challenges included a lack of beneficiary cooperation, logistical difficulties, and human resource gaps. • Remaining primary needs include additional training and capacity development, access to additional material resources, and increased cooperation among local health and social welfare system actors. • CHSSP contributed to improving the community health and social welfare services and outcomes related to women, girls, and youth in terms of increased HIV awareness and treatment and reduced stigma; incremental improvements in gender-based violence (GBV) and violence against children (VAC) awareness, prevention, and response; and improved quality of health and social welfare services. EQ 3: To what extent has CHSSP contributed to creating an improved enabling environment and more functional and better coordinated community structures and systems for providing community health and social welfare services to priority and key populations, particularly AGYW, MVC, and PLHIV? • CHSSP contributed to improving the enabling environment and underlying structure of the community health and social welfare system at all levels of government. This included both strengthening/revitalizing existing attributes and creating new attributes of the enabling environment and community structure, including the NICMS, policies and guidelines, and tools and resources. • CHSSP contributed to improving the local systems for the coordination of health services provision and social welfare. In particular, CHSSP contributed to making ward multi-sectoral AIDS committees (WMACs), village multi-sectoral AIDS committees (VMACs), PLHIV clusters, and VAWC-PCs more active and operational. The trainings, guidance, and tools and supportive supervision provided them have had a positive impact on the quality of services and support to local structures that committee and cluster members provide. • CHSSP contributed to improving the organizational capacity and maturity of district-level CSOs. iii The extent to which these CSOs have improved their organizational capacity and maturity;2 however, awaits the follow-up administration of the organizational capacity assessment tool (OCAT). EQ 4: To what extent will CHSSP activities and results be sustained after the program ends? Which factors affect the prospects for sustainability and how? • The systems approach adopted by CHSSP is a sound strategy, relevant for the context in which CHSSP is working, and an appropriate approach to secure the sustainability of health and social welfare system reforms. • CHSSP contributed to improving the enabling environment and community-level structures. This contribution is substantive in nature and shows evidence of advancing toward institutionalization within the GoT, although still short of achieving this. • The sustainability of the reforms achieved by CHSSP depends on the ongoing functioning and institutionalization of each of the three key elements of the community health and social welfare system targeted by the program: enabling environment, community-level structure (e.g., NICMS and committee structure), and human capacity. • Several threats exist to the ongoing functioning and institutionalization of program achievements. CHSSP has taken a number of steps to address these threats, but constraints existing both within the program and within the GoT mean that they can only be addressed partially. Any follow-on program will need to consider how to consolidate gains achieved during CHSSP and move them more toward institutionalization within the GoT and the relevant community-level structures and systems. RECOMMENDATIONS • CHSSP, or a follow-on program, should continue to roll out training and capacity-building to meet targets with emphasis on the training-of-trainers (ToT) approach. Only five wards per council are being currently reached by CHSSP training, and this should be scaled up to reach local structures at the ward and village level. • CHSSP, or a follow-on program, should consider augmenting its current CCW training with other technical topics, such as violence prevention, early childhood, responding to crises, HIV prevention, treatment and care, etc. Kizazi Kipya provides some supplementary technical training and job aids outside of their initial scope, but other support is needed to bring these efforts to scale. • CHSSP, or a follow-on program, should allocate additional resources and support for district master trainers to provide follow-up trainings for new CCWs in the wards and villages. The ranks of CCWs are expected to rotate over time as new cohorts are brought on board to augment or replace existing CCWs; thus, it is necessary that they receive the same quality of training as the earlier CCW cohorts. Resources—such as venues for training, money for transportation reimbursements, and materials—are needed to perform these follow-up trainings. Bringing CCWs into the district for follow-on activities might also help social welfare officers (SWOs) provide support and fill gaps left in supportive supervision at the ward and village levels. • In light of 2018 announcements of new target councils, there were new training needs for CCWs, which were not fully met by CHSSP. There should be a clear channel of communication between 2 Organizational capacity and maturity were assessed according to the seven domains of the OCAT which include: governance, administration, human resource management, financial management, organizational management, program management, and project performance management. iv USAID, CHSSP, and Kizazi Kipya to fill gaps in training and capacity-building in new target councils. • CHSSP, or a follow-on program, should consider providing operational expenses or other forms of financial or in-kind support for selected system actors, such as CCWs, supervisors, MACs, VAWC-PCs, or PLHIV clusters. Examples of potential support include transportation, printing and maintenance of tools, space for offices and meetings, and stipends. GoT contributions to these costs could then be better articulated in a cost sharing agreement in the first years of the follow-on program together with a sustainability plan with the President’s Office for Regional Administration and Local Government (PO-RALG) so that these costs are incorporated into the planning and budgeting tool for local government budget allotments. • Working with PO-RALG, USAID could advocate for further allocation of budget for human resources, particularly SWOs at the district and ward level, to reduce SWOs’ caseloads and improve their ability to provide supportive supervision to more wards and villages. • To better implement the National Plan to End Violence Against Women and Children, CHSSP, or a follow-on program, should continue to establish VAWC-PCs as local structures for supporting this effort. Strategies for more effective linkages for local incidents of GBV or VAC are needed to ensure that cases reach the VAWC-PC and that linkages are completed with law enforcement, health, HIV testing and care, and other assistance. • Additional sensitization work with communities and working through VAWC-PCs and other community leaders to increase awareness of VAC and GBV could help change social norms to enhance community cooperation in these cases. CCWs should also be better equipped to address these issues and communicate with communities about how to handle these cases through the justice, health, and social welfare system. • Considering only five wards per selected district have been reached by the program, additional resources for capacity-building activities should be provided to enable district-level participants to better reach additional wards and villages in those districts. Furthermore, sustaining the cascade of capacity-building from the national level by master trainers to district participants will require further donor support to bring this effort to scale. • Sustaining and retaining CCWs would be better enabled by a revised incentive system for CCWs, lead CCWs, and supervisors that includes some combination of cash, in-kind, and psycho-social support. Time use, retention, and incentives should be further assessed over time to ensure the long-term commitment of CCWs. • Rather than, or in addition to, continuing to support such a large number of district CSOs, CHSSP, or a follow-on program, should consider selecting a smaller cohort of high-performing CSOs with more chance of obtaining international funding for more focused capacity-building. • Overall, CHSSP, or a follow-on program, should continue to work with the GoT to secure its commitment to funding structures that support vulnerable groups (orphans and vulnerable children [OVC], PLHIV, people with disabilities, AGYW) and sustaining the necessary level of human resources. GoT should be further be encouraged to reinforce supportive supervision and follow-up training to ensure the practices supported by CHSSP continue after the program. 1 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE The purpose of this mid-term performance evaluation of the United States Agency for International Development (USAID) Community Health and Social Welfare Systems Strengthening Program (CHSSP) is to: 1) assess program performance and achievements thus far; 2) analyze the effectiveness of various approaches used by the program implementing agency to achieve program results and foster sustainability; and 3) document good practices, gaps/limitations, and lessons learned that will inform the Mission’s design of the follow-on community-based systems strengthening program, including critical approaches, focus areas, and interventions. (See Annex I for the evaluation Statement of Work [SOW]). Primary audiences for the evaluation include the Mission’s technical officer, to inform the design of the follow-on or new community systems strengthening award, and the Government of Tanzania (GoT), to contribute to national guidelines and strategies related to community health and social welfare systems. 1.2 EVALUATION QUESTIONS The evaluation seeks to answer the following four evaluation questions (EQs) and related sub-questions: 1. What have been the strengths and weaknesses of CHSSP implementation? 1.1 Which factors have affected its implementation and how? 1.2 How did the program address these factors and with what results? 2. To what extent has CHSSP contributed to creating higher-performing human resources for providing community health and social welfare services to priority and key populations, particularly adolescents, girls, and young women (AGYW), most vulnerable children (MVC), and people living with HIV (PLHIV)? 2.1 What challenges are health and social welfare actors facing in fulfilling or improving their roles in providing community health, social welfare, and HIV prevention and treatment services to key populations? 2.2 What can CHSSP, and other programs, including Kizazi Kipya, do to better to assist health and social welfare actors to address these challenges? 2.3 How did CHSSP support address issues related to women, girls, and youth, including, for example, factors limiting their access to community health and social welfare services and/or the quality of community health and social welfare services they receive and with what results? 3. To what extent has CHSSP contributed to creating an improved enabling environment and more functional and better coordinated community structures and systems for providing community health and social welfare services to priority and key populations, particularly AGYW, MVC, and PLHIV? 4. To what extent will CHSSP activities and results be sustained after the program ends? Which factors affect the prospects for sustainability and how? 2 2.0 PROGRAM BACKGROUND 2.1 CHSSP GOALS AND OBJECTIVES CHSSP is a six-year3 (November 2014-November 2020) $36 million USAID and President’s Emergency Plan for AIDS Relief (PEPFAR)-funded program that works with the GoT and other health and social system actors to improve the health and well-being of HIV-affected and other vulnerable populations. The goal of CHSSP is to achieve the PEPFAR 90-90-90 goal4 and stop the HIV epidemic in the country. In service to the above goal, CHSSP seeks to achieve two objectives: Objective 1: Higher performing human resources for community health and social welfare services (who are able to support AGYW, MVC, and PLHIV to know their status, improve retention and adherence, and achieve viral suppression, while preventing new infections and promoting overall well-being). As part of this objective, CHSSP implements its activities under two strategies. Under strategy one, CHSSP advocates that the Ministry of Health, Community Development, Gender, Elderly, and Children (MOHCDGEC), Tanzania Commission for AIDS (TACAIDS),5 and President’s Office for Regional Administration and Local Government (PO￾RALG)6 update and finalize policies. In addition, CHSSP provides technical guidance on incorporating PEPFAR 3.0 priorities into the guidelines and policies identified for revision. Under strategy two, which aims at expanding the social welfare workforce, CHSSP rolled out implementation of the country’s first nationally integrated case management system (NICMS). The NICMS integrates the provision of HIV and health services, social welfare services, child protection services, and National Plan of Action-Violence Against Women and Children (NPA-VAWC). The NICMS guides CHSSP’s trainings for community case workers (CCWs), lead community case workers (LCCWs, formerly para-social workers, or PSWs), and CCW supervisors who support identifying MVC and their families. Training involves conducting need assessments and referring and linking MVC and their family members to necessary services.7 Objective 2: More functional, better coordinated community structures and systems to better serve key and vulnerable populations (KVPs). This objective includes activities that focus on creating an enabling policy environment and strengthening coordination, partnerships, organizational leadership, and management capacity of community structures. The community structures include civil society organizations (CSOs), PLHIV clusters, multi-sectoral AIDS committees (MACs), and VAWC protection committees (VAWC￾PCs). The objective also focuses on reducing stigma and discrimination and implementing a service model to increase access to, and quality of, core HIV health and social welfare services for priority and key populations. 3 CHSSP was initially a five-year program for 2014-2019. It received a one-year extension through November of 2020. 4 Ninety (90) percent of all people living with HIV will know their HIV status, 90 percent of people diagnosed with HIV will receive sustained antiretroviral treatment, and 90 percent of all people receiving treatment will have viral suppression. 5 TACAIDS is the GoT entity charged with overseeing the national HIV/AIDS response. TACAIDs also provides local government authorities support for coordination and adherence to national guidance on HIVAIDs. 6 PO-RALG the president’s council, which represents the national government agenda locally. 7 CCWs provide front-line support and referrals directly to the community. LCCWs and lead community case work supervisors must hold a Form IV leaving certificate and are trained and paid for their work in the community. CCWs and CCW supervisors are trained volunteers that support PSWs. CCWs do not need to hold a Form IV leaving certificate (but most hold basic literacy and numeracy). CCWs allow the more highly-trained LCCWs to reach a larger number of community members and provide significant support for the social welfare system. Both groups work with vulnerable children, families, PLHIV, women experiencing or at risk for domestic violence, and those at high risk for HIV, among other populations. They use a case management approach that was developed by CHSSP and provide referrals within the referral system strengthened by CHSSP. 3 2.2 CHSSP APPROACH CHSSP is implemented by JSI Research & Training Institute (JSI), in partnership with its implementing partner (IP) World Education, Inc. (WEI),8 and local collaborators. These collaborators include the TACAIDS within the Prime Minister’s Office and PO-RALG. Together, these actors seek to improve Tanzania’s community health and social welfare sectors and to cascade the system-strengthening approach from national- to community-level KVPs using a training-of-trainers (ToT) approach. At the national level, CHSSP works with the respective GoT entities to update and create new policies and guidelines to strengthen the HIV response. On the sub-national level CHSSP works to train local government and community structures on how to put the policies and guidelines into practice and use them to lead response efforts in their communities—including working with health and social welfare providers to ensure people are linked to care and treatment. Initially, CHSSP trained community health workers (CHWs) while using local government authorities (LGAs)/councils and CSOs to cascade training/assistance to KVPs. Beginning in November 2017, the activity switched its focus from training CHWs to training CCWs using the same cascading training/assistance model. Specific program interventions covered by the evaluation include: 1) support for the NICMS via its training of and supportive supervision (post-training assistance) to CCWs; 2) organizational strengthening of local CSOs to support the case management and community-based health and social welfare systems and related services more generally; 3) strengthening of local health and social welfare systems and actors (including regional, ward, and village MACs;9 regional and ward VAWC-PCs,10 and regional and ward PLHIV clusters);11 and 4) support to relevant GoT entities to develop policies, guidelines, and standards to strengthen community-based health and social welfare systems. 2.3 REVISIONS TO THE CHSSP PROGRAM DESCRIPTION Since its inception in 2014, CHSSP has changed the program description (PD) three times. During CHSSP’s first PD (November 2014-December 2015), it worked in ten regions and all 68 councils within those regions. During its second PD (December 2015-May 2017), CHSSP was aligned to adhere to the requirements of the PEPFAR 3.0 “pivot” by redirecting focus to 42 priority, or “saturation” and “aggressive,” councils within the ten regions with high HIV/AIDS prevalence. Saturation councils were those in which PEPFAR sought to achieve 80 percent coverage of PLHIV on antiretroviral therapy (ART) 8 CHSSP originally had four IPs, which, in addition to JSI and WEI, included the National Association of Social Workers (NASW) and Initiatives, Inc. NASW left the program in 2015, and Initiatives, Inc. left the program in 2017. The departure of Initiatives, Inc. meant that the program was no longer able to support CHWs and the development of relevant training manuals. 9 MACs responsible for facilitating coordination and community participation in HIV/AIDS-related matters at the local level. The committees consist of government officials at the council, ward, and village levels and representatives from the PLHIV, legal, CSO, religious, law enforcement, health, and social welfare systems. MACS are structured in three levels: council-level MACs (CMACS), ward-level MACs (WMACs), and village-level MACs (VMACs). MACs receive direct training coaching and mentorship from CHSSP through the council HIV/AIDS coordinator (CHAC) at the council level. The CHAC in turn is expected to cascade this training to the WMACs and VMACs at the ward and village levels. CHSSP works to form MACs at the ward and village levels and to provide training in supportive supervision to MACs at the council, ward, and village levels. 10 VAWC-PCs are comprehensive multi-sectoral committees created to address various aspects of preventing VAWC and gender-based violence (GBV) at all levels from the national to the community level. They consist of local government officials and representatives from the legal, CSO, religious, law enforcement, health, and social welfare systems. CHSSP provides assistance to form VAWC committees at the council level, in addition to training and supportive supervision, as per the NPA to end VAWC. 11 PLHIV clusters are registered district-level groups of PLHIV working on and coordinating the local response to HIV/AIDS, including working to reduce stigma related to HIV/AIDS and to ensure that people adhere to their medication (the third 90 percent of the 90-90-90 Joint United Nations Program on HIV/AIDS [UNAIDS] PEPFAR priority). PLHIV clusters are formulated and supported by the National Council of People Living with HIV (NACOPHA). PLHIV clusters receive direct training from CHSSP through the PLHIV cluster leader at the council level. The PLHIV cluster leader in turn is expected to cascade these trainings to the PLHIV groups at the ward and village levels. 4 by Fiscal Year (FY) 2017. Aggressive councils were those in which PEPFAR sought to achieve 80 percent coverage of PLHIV on ART by FY 2018 or FY 2019. During its third PD (June 2018-September 2018), the number of priority councils was increased to 84. The mutually agreed-on PD was designed to align CHSSP’s approach with that of Kizazi Kipya and resulted in the focus shifting from solely strengthening community health systems and CHWs to strengthening both community health and social welfare systems through the national case management system. In practical terms, this meant switching from a focus on working with CHWs to a focus on working with a target of 20,000 CCWs in support of the NICMS. Finally, during its fourth PD (October 2018 to present), the number of priority councils was increased from 84 to 106, and the life of program (LOP) target for number of CCWs trained was increased from 20,000 to 32,000. 3.0 EVALUATION METHODS AND LIMITATIONS This section describes the evaluation methodology, including data collection methods, sampling methods, and methodological limitations. Evaluation fieldwork took place from February 4-March 8, 2019. 3.1 DATA COLLECTION METHODS The evaluation used a mixed-methods data collection design consisting of four data collection methods: 1) document review, 2) key informant interviews (KIIs), 3) focus group discussions (FGDs), 4) a CCW survey, and 5) review of CHSSP performance data. Each of these is described below. Document Review: The evaluation team (ET) reviewed documents pertaining to CHSSP (including program documents) and other documents related to health and social welfare issues within the country. CHSSP provided some documents, and the ET found others during the literature search. (See Annex II for a comprehensive bibliography.) KIIs: The ET conducted 34 KIIs with 107 individuals (65 women and 42 men), including one KII with USAID; ten KIIs with JSI, WEI, and the implementing consortium; two KIIs with national-level GoT officials; six KIIs with regional-level GoT officials; and 17 KIIs with council-level health and social welfare actors. (See Annex III for the KII and FGD discussion guides, Annex VII for a breakdown of KII and FGD participants by stakeholder and location, and Annex VIII for a detailed list of the KIIs completed). FGDs: The ET conducted 14 FGDs with 142 individuals (73 women and 69 men) at the ward and village levels, including five with CCWs, eight with ward and village multi-sectoral AIDS committees (WMACs and VMACs), and one with a most vulnerable children’s committee (MVCC).12 CCW Survey: The ET worked through its local research partner, Utafiti, to conduct a phone survey of a representative sample of 269 (148 female and 121 male) CCWs. The survey instrument covered all elements of CCW training related to CHSSP and the casework implemented by the same CCWs under Kizazi Kipya. (See Annex IV for more details on the sampling methodology for the KIIs, FGDs, and CCW survey; Annex V for a copy of the CCW survey questionnaire; and Annex VI for a summary of the CCW survey results.) 3.2 DATA ANALYSIS METHODS To analyze the qualitative data, the ET conducted a content analysis using the Dedoose qualitative analysis software. The ET used thematic coding to identify and clarify patterns in the data among the codes 12 MVCCs used to exist at the council, ward, and village level until Program Year (PY) 4 of CHSSP. These committees focused on providing support and services to the most vulnerable children. They have seen been rolled into the VAWC-PCs. 5 allowing it to draw inferences from the data by systematically identifying themes and sub-themes and assessing their relative importance in answering the EQs. To analyze the CCW survey data, the ET used Microsoft Excel to analyze and visualize data using descriptive statistics. 3.3 METHODOLOGICAL LIMITATIONS The above evaluation methodology involved the following potential limitations: 1) reliance on, at times, unverifiable perceptions and anecdotes by key informants; 2) use of unverified CHSSP performance monitoring data; 3) recall bias by key informants commenting on past events; 4) halo bias by key informants incentivized to under-report socially undesirable answers or refrain from criticizing colleagues or people on whom they depend for support; 5) sampling bias given the purposive (non-random) selection of key informants for KIIs and FGDs and that the CCW survey sampling frame was limited to those CCWs who had access to a tele/mobile phone; and 6) the difficulty assessing the sustainability of program interventions that were ongoing at the time of the evaluation fieldwork. To mitigate the above limitations, the ET sought to verify findings from one data source and method with findings from other data sources and methods through triangulation. The ET also worked closely with JSI and Utafiti to implement sound qualitative and quantitative sampling methods and to impose strict data collection quality control methods. To address limitations related to evaluating program sustainability, the ET asked probing questions related to sustainability wherever relevant. This allowed the ET to estimate the “likelihood” of sustainability and the factors contributing to this likelihood. 4.0 FINDINGS AND CONCLUSIONS 4.1 EQ 1: WHAT HAVE BEEN THE STRENGTHS AND WEAKNESSES OF CHSSP IMPLEMENTATION? 4.1.1 Findings The ET reviewed program performance data to analyze its implementation strengths and weaknesses (see Table 1). As explained in more detail below, the program objectives and monitoring and evaluation (M&E) framework shifted several times during implementation. Thus, this review focused on the updated CHSSP objectives, which stand as of program year (PY) 3, or FY 2017, monitoring data from PY 3, and narrative reports and raw performance monitoring data for PY 4 (FY 2018). CHSSP Implementation Performance Relative to Performance Targets Both program Objectives 1 and 2 use a cascade model for training and capacity-building. Performance monitoring data for PY 3 shows that CHSSP was strong in meeting targets higher up the cascade but struggled to meet targets lower down the cascade at the ward and village levels. During PY 4, however, the program appears to have improved its roll-out of activities at the ward and village levels. Under Objective 1, the program trains LCCW supervisors, LCCWs, and CCWs using a cascade model. As seen in Table 1, the program exceeded its targets in PY 3 for the five-day training of CCW supervisors (422 percent of target) and for the ten-day and four-day LLCW supervisor training (both at 102 percent of target). It also exceeded training targets for district master trainers (DMTs) (195 percent) in PY 3. However, the program did not meet its targets further down the cascade and trained only 79 percent of the 19,603 targeted CCWs and LCCWs in FY 2017 and 71 percent of the 2,450 targeted LCCWs and LCCW supervisors. Objective 2 data shows that CHSSP met or exceeded targets for establishing or revitalizing MVCCs and MACs at the council level. In PY 3, the program revitalized or established 37 council MVCCs, or 112 percent of the target. That same year, CHSSP reached only 24 of the targeted 165 (15 percent of target) ward MVCCs and 100 of the 825 (12 percent of target) village MVCCs. MVCCs were merged with other groups to form VAWC-PCs at the council, ward, and village levels during PY 4. Similarly, while the 6 program revitalized or established 100 percent of its target 54 council-level MACs during PY 3, it established or revitalized only 59 percent of ward and village MACs. Monitoring data for PY 4 shows that the program made large numbers of MAC monitoring visits at the council and ward levels during that year. By the end of PY 4, the program had reached 280 unique WMACs and 665 VMACs, showing significant progress from PY 3. While the MVCC committees no longer existed, the new VAWC-PCs also used a cascade model and the FY 2018 Annual report (p. 21) shows a strong rollout of both council- and ward-level supervision. Table 1: CHSSP Performance Indicators – Targets and Actuals by Fiscal Year Performance Indicator 2016 2017 2018 Target Results % Target Results % Target Results % New health workers who graduated from a preservice training institution or program 200 229 114% . . . . . . Number CHWs and PSWs who successfully completed a pre￾service training program (e.g., MVCCs) 2,450 2,452 100% . . . . . . Health care workers and and PSWs who successfully completed an in￾service training program 206 432 205% . . . . . . CCWs and LCCWs trained (five￾day) . . . 19,603 15,560 79% . . . LCCWs and LCCW supervisors trained (ten-day) . . . 2,450 1,737 71% . . . Council multi-sectoral AIDS committees (CMACs) established/ revitalized . . . 54 54 100% 54 54 100% WMACs established/revitalized . . . 270 160 59% . . . VMACs established/revitalized . . . 1350 800 59% . . . Council MVCCs established/revitalized . . . 33 37 112% . . . Ward MVCCs established/revitalized . . . 165 24 15% . . . Village MVCCs established/revitalized . . . 825 100 12% . . . CSOs with strengthened capacity . . . 48 48 100% 50 50 100% Functional district PLHIV clusters . . . 33 48 145% 54 54 100% Three-day ToTs on MAC curriculum . . . . . . 270 292 108% DMT ToTs . . . 203 449 195% 137 23 17% Data demand and use training . . . . . . 270 292 108% Resource mobilization and participatory planning for HIV/AIDs and gender-based intervention workshops . . . . . . 54 54 100% Quarterly IP coordination meetings . . . . . . 108 84 78% VAWC-PCs that are established . . . . . . 52 52 100% Organizational capacity assessments . . . . . . 38 38 108% PLHIV cluster assessments . . . . . . 54 54 100% PLHIV cluster resource mobilization workshops . . . . . . 54 54 100% 7 4.1.2 Conclusions • CHSSP managed to cascade down over time (although later than foreseen) and largely achieved targets at all levels of cascade by the end of PY 4. The delays experienced in PY 3 may have been due to the increased number of target councils (and therefore wards and villages) as part of changes to the program’s PD 3, which stretched program resources. 4.2 EQ 1.1: WHICH FACTORS HAVE AFFECTED IMPLEMENTATION AND HOW? 4.2.1 Findings Changes to the CHSSP Program Description One of the main reasons that cascading down to the village and ward level may have posed challenges in PY 3 is that the CHSSP mandate, scale, and scope changed drastically since program inception. These changes, which were requested by USAID and mutually agreed-on by JSI, took place for several reasons. First, updated international PEPFAR 90-90-90 goals and framework resulted in PEPFAR in Tanzania making significant strategic changes. Second, the Tanzania Big Results Now development framework recently added a health priority area and refocused GoT programming and TACAIDS’ strategy on a systems￾strengthening approach. Lastly, USAID launched new programs such as Kizazi Kipya addressing HIV/AIDS through social welfare and the national case management system in Tanzania. Program documentation and narrative reviews, including annual reports, reflect these changes; the initial PD was updated three times, which resulted in a total of four PDs. The budget was changed during each PD update as well. The changes were made in an attempt to ensure alignment and coordination, but findings from document review, KIIs with IPs and consortium members, and regional- and district-level actors, all confirm that changes to the program’s scale and scope affected implementation. Regional- and district-level system actors raised that the changes to the program scale and scope were a challenge in program implementation.13 In addition to the larger changes in scale and scope, respondents commented—and CHSSP annual reports confirm—unplanned expansion of services also occurred. These were an attempt to coordinate implementation with other USAID interventions, including Kizazi Kipya. According to CHSSP’s FY 2017 Annual Report, “In PY 3, CHSSP planned to cover only 54 councils, but since its service delivery partner USAID-Kizazi Kipya covered more councils (68) including Zanzibar, CHSSP increased of council coverage to match districts” (p. 17). JSI explained that the scale increase was not possible given its existing resources, and that it attempted to negotiate these changes with the Mission. JSI argued (and USAID confirmed) that the changes to the program’s scale and scope created a “working [moving] target” that created significant challenges for IPs. Confusion Surrounding Program Target Communities The PD changes created implementation challenges regarding which villages and CCWs were included in the program. National, regional, and district representatives, as well as the implementing consortium and CSOs, expressed confusion or frustration about why certain district councils (DCs) were included and others were not.14 13 One of four CSOs, two of 11 district actors, one of five regional actors, five of six implementing consortium members, two of two GoT officials, and one of one USAID. 14 Two of two national GoT representatives, one of six implementing consortium members, one of four CSOs, and one of five regional stakeholders. 8 Shifting Roles and Responsibilities of Program Consortium Members Additional implementation challenges emerged around the shifting roles and responsibilities of different consortium members due to the changes in program scale and scope. The program went from working with four partners under the first PD, to only two partners under the third and fourth PDs. At the same time, CHSSP began collaborating with several additional USAID programs, such as Kizazi Kipya. Confusion and challenges related to coordination and shifting responsibilities of consortium members came up in discussions with stakeholders at all levels.15 One government stakeholder confirmed that, “We have seen some collision during the implementation. This is to say that Pact’s role is to utilize the tools and systems designed by JSI, while JSI’s role is to design and develop the tools and systems. But sometimes you find that Pact also designs its own tools and systems that interfere with the JSI ones.” A CSO also noted that this role confusion resulted in double-training, saying, “All those who were trained by JSI were re-trained by Kizazi Kipya once again.” Pact respondents confirmed this challenge. Coordination of Tools and Guides Difficulty coordinating the consortium also affected tools and guide use. A district-level stakeholder in Mbeya city council (CC) explained, “We got support from JSI to print guides, but what they printed and brought to us for the first time has finished, and we really need them. The information we have is that Pact is now responsible for printing them and not JSI. But Pact said they are waiting for the budget so as to be able to print.” In terms of ongoing coordination, program performance data confirms that, while some of the planned local consortium coordination meetings took place, the number of meetings was slightly below target. Community Level Effects of Changing Program Description Stakeholders at the regional, district, and local levels mentioned that the scale and scope changes adversely affected program implementation at the community level. According to a district social welfare officer (DSWO) in Njombe town council (TC), “Sometimes when plans change, it gives us problems. For instance, when they came in the beginning to give the program introduction, there were some things like tools that helped us social welfare officers. But later, we were told that there were no more.” Or, as a council HIV and AIDS coordinator (CHAC) in Njombe TC noted, “If I have signed agreement or contract, I won’t put it in the budget because I know I will implement this through a stakeholder. So, when you come in half way through and say, ‘I can’t do this,’ the community is the one affected.” 4.2.2 Conclusions • Repeated changes to CHSSP’s scale and scope created significant implementation challenges, both around who receives services and around coordination among consortium members, in some cases adversely affected the functioning of the health and social welfare system at the community level. 4.3 EQ 1.2: HOW DID THE PROGRAM ADDRESS FACTORS THAT AFFECTED IMPLEMENTATION AND WITH WHAT RESULTS? 4.3.1 Findings Staffing and Partner Restructuring CHSSP took significant measures to address the challenges created by its expanded scale and scope. It restructured staffing and partners and reorganized and streamlined country headquarters management and staffing to reallocate resources to the field and help accommodate increased scale. It created new positions to ensure strong technical knowledge and stakeholder coordination, including a staffing a position in Dodoma to improve coordination with the GoT. It closed field offices that became less relevant with 15 Three of five CCWs, three of four VMACs, three of four CSOs, three of three PLHIV clusters, three of five regional stakeholders, and four of six consortium members. 9 the PD changes and opened new offices in key regions. It further restructured its IP structure, including working with WEI to double the amount of PEPFAR councils reached. Coordination with Key Program Stakeholders Despite these challenges, JSI did coordinate implementation with key stakeholders (including the GoT, IPs, and CSOs). Stakeholders spoke to JSI’s attempt to meet and coordinate with them and the program’s part in helping to clarify roles and responsibilities nationally.16 According to a PLHIV leader in Njombe TC, “There has been a good relationship between us and the government and service providers. Previously there was no such coordination. After these trainings, we have started to find donors and stakeholders, and we have realized that the government is our first and main stakeholder.” Another PLHIV leader in Temeke municipal council (MC) added, “The CHSSP program has built working relations between stakeholders and has established better systems of leadership through regular meetings with [the] district council.” CHSSP worked with stakeholders to improve local coordination. For example, it participated in a meeting convened by PO-RALG and MOHCDGEC to discuss the best ways to improve coordination of IPs at the council level. The meeting also updated PO-RALG on program successes, challenges, and ways to strengthen community health and social welfare structures and systems. In KIIs, local stakeholders commended JSI for its attempts at linking and coordination. Flexible Program Management Overall, program management demonstrated flexibility by steering the program to meet key indicator targets in the face of changing on-the-ground circumstances. The program had to update its M&E Plan to align with changes in scale and scope, and it appears to have planned appropriately for most activities. As explained above, achievement of cascade targets lagged behind other achievements because the PD changes created a backlog of activities around certain areas at the ward and village levels. This meant that targets for those activities were not met or exceeded as frequently. Given the multiple changes to the PD, the evaluation focused on PY 4 results for deeper analysis. CHSSP met or exceeded indicator targets for 11 key indicators in PY 4, with only two output indicators falling short of targets—number of quarterly implementation meetings (78 percent of target) and ToTs for DMTs (17 percent of target). At the outcome level, CHSSP established 52 VAWC-PCs and increased the capacity of 50 CHSSP CSOs, exactly meeting targets in both cases. Benchmarks related to MACs were achieved and MAC ToTs slightly exceeded targets. Following the ToTs, trainers trained WMAC representatives, achieving 117 percent of target. According to PY 4 targets, CHSSP also successfully completed PLHIV workshops and cluster assessments. 4.3.2 Conclusions • CHSSP demonstrated a flexible management approach and took effective measures to address the challenges created by its expanded scale and scope. It coordinated with stakeholders from the national level down to the village level so that the program stayed on track and met key indicator targets. 16 Two of four VMACs, one of six CCWs, two of three PLHIV clusters, one of six implementing consortium members. 10 4.4 EQ 2: TO WHAT EXTENT HAS CHSSP CONTRIBUTED TO CREATING HIGHER PERFORMING HUMAN RESOURCES FOR PROVIDING COMMUNITY HEALTH AND SOCIAL WELFARE SERVICES TO PRIORITY AND KEY POPULATIONS, PARTICULARLY AGYW, MVC, AND PLHIV? The ET interprets EQ 2 to refer to the extent that CHSSP achieved program Objective 1 related to updating and finalizing policies and technical guidance and rolling out the NICMS. 4.4.1 Findings Program Contributions to Revising, Developing, and Disseminating National Policies and Guidelines At the national level, CHSSP supported TACAIDS to identify and update policies and guidelines, provided technical guidance for incorporating PEPFAR 3.0 priorities and the 90-90-90 goal into these policies and guidelines, coordinated government bodies to make these changes, and helped disseminate documents. CHSSP also helped TACAIDS to strengthen adherence and implementation of national guidelines for the NICMS while further supporting TACAIDS with the review, development, and/or dissemination of guidelines for MACs, MVCCs, VAWC-PCs, PLHIV clusters, CSOs, and CHWs from the national down to the regional, council, ward, and village levels. A list of policies and guidelines that have been revised, developed, or disseminated with program contributions is provided in Table 2. Table 2: Policies and Guidelines Revised, Developed, or Disseminated by CHSSP Revised National MVC Identification Guidelines National Guidelines for Early Identification and Intervention for Children with Disabilities National Guidelines for Early Identification and Intervention for Children with Disabilities Review Process National Multi-Sectoral HIV and AIDS Stigma and Discrimination Reduction Strategy 2013-2017 National Gender Operational Plan for HIV Response in Tanzania Mainland 2016-2018 Community Based Health Program Design CHW Supervision Guidelines MAC curriculum Developed CHW training resources, including CHW Facilitator’s Guide, CHW Practicum Guide, CHW Supervision Guidelines, Student Manual, and Assessment Plan NICMS Framework CCW training and implementation resources, including Training Manual and Practical Handbook NICMS data collection forms: 1) national MVC registration, 2) screening and enrollment, 3) child care plan, 4) HIV risk services and adherence, 5) national MVC referral, 6) national MVC monthly service tracking, 7) graduation assessment, and 8) case closure Disseminated to LGAs Gender and HIV Operational Plan HIV and AIDS Human Rights Advocacy Strategy National Gender Operation Plan for HIV Response MAC guidelines MAC curriculum MVCC guidelines PLHIV cluster guidelines CSO guidelines In KIIs and FGDs, local health and social welfare system actors uniformly confirmed that the guidelines, tools, and other resources developed with program assistance helped them fulfill their roles and helped the system function more effectively. Two quotes illustrate this point: “These activities will be sustainable because we have already formulated plans, we have guidelines meaning that we have books…even if I am not 11 available, the person that will come after me will see the continuation as embodied in these things” (WMAC member in Temeke MC) and “Right now in the guidelines, we have kept all topics about good leadership. The current guidelines explain how to identify stakeholders, how to address them, how to find resources, how to use them, how to develop them, how to do succession to new members” (TACAIDS official). According to the same TACAIDS official, CHSSP arrived at a time of need when a grant, which supported the National Multisectoral Strategic Framework and planning under the Danish International Development Agency (DANIDA), had expired, and, as a result, gains achieved under the grant had come undone. The official affirmed that CHSSP made an important contribution getting the reform process moving again and that the policies and guidelines developed with program assistance provided a framework to enable the reform process to continue. Capacity Development of CCWs (Application of New Knowledge and Skills) CCWs are the backbone of the NICMS and are the point of contact between vulnerable households and government and non-government services for orphans and vulnerable children (OVC). They use the national case management guidelines and referral system to facilitate vulnerable children and their families’ access to services. In the CCW survey, 80 percent of CCWs said that they “frequently” applied the knowledge and skills they received from the CCW training on the job, with another 4 percent saying they “sometimes” applied them and 14 percent saying they “rarely” applied them (Table 3). CCWs in Kagera and Mwanza had especially high levels of skills application at over 90 percent each. Table 3: Knowledge and Skills Applied by CCWs vs. Training Received (N=141) Topic Trained Applied Assess HIV risks, services, and adherence for children and adolescents 63.9% 39.0% Identify OVC and their families 66.2% 36.8% Other 24.5% 30.1% Linking OVC and their families with services 29.4% 27.5% Child development 29.0% 27.5% Counseling skills 20.8% 22.3% Referral provision 11.2% 21.9% Effects of neglect and abuse 32.0% 18.6% Child protection 22.3% 15.2% Identify difficulties and their families 16.0% 11.5% Basic child and family assessment 9.7% 10.8% Principles of case management 10.0% 7.8% Use of case management tools 9.3% 6.3% Processes/steps of case management 7.1% 2.2% Fifteen (15) percent CCWs who did not apply the knowledge and skills they learned on the job said that the knowledge and skills were not practical. Another 10.4 and 9.0 percent, respectively, said that they did not have enough time to apply them or they lacked funding or resources to apply them. Health and Social Welfare Outcomes as a Result of Better Performing CCWs and NICMS Around 84 percent of CCWs said that the CCW training had to “some extent” or a “large extent” improved their coordination of referrals (Table 4), while 90 percent “agreed” or “strongly agreed” that the training improved their ability to refer beneficiaries to providers, improved linkages to quality health and social welfare services, and improved the effectiveness of the referral system. Another 80 percent of CCWs “agreed” or “strongly agreed” that there is now improved access to health and social welfare services and improved HIV prevention. Finally, between 60-70 percent of CCWs “agreed” or “strongly agreed” that beneficiaries now receive better services and that those services meet their needs. More 12 pessimistically, 43.1 percent of CCWs “strongly disagreed” that the CCW training had improved VAWC prevention and response, while 49.6 percent “agreed” that it had. Table 4: Perceived CCW Performance (N=267) Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree Don’t Know Successfully refer beneficiaries to providers 0.8% 1.5% 1.1% 48.7% 47.6% 0.4% Improved linkages to quality services 0.7% 0.4% 3.4% 49.1% 46.4% 0.0% Improved referral system effectiveness 1.1% 0.7% 3.7% 51.7% 39.7% 3.0% Improved access to health & social services 0.0% 3.8% 6.8% 50.8% 37.6% 1.1% Improved HIV prevention 1.2% 9.0% 7.5% 50.2% 31.4% 0.8% Beneficiaries receive services 1.1% 6.7% 9.7% 37.5% 42.0% 1.5% Services meet beneficiaries’ needs 2.3% 11.6% 16.5% 44.2% 22.5% 2.3% Improved VAWC prevention and response 43.1% 1.5% 2.3% 49.6% 1.1% 2.3% Key drivers and support for the CCWs’ work in household case management appears to be sufficiently strong. This is evidenced by the 93 percent of survey respondents who said that they received enough supervisory support and the over 80 percent of respondents who agreed that there are enough social welfare providers to support OVC and PLHIV. In KIIs and FGDs, system actors noted that CCWs are now more successful in their roles because of CCW training and supportive supervision (two of 11 district actors; one of four CSOs; one of two national GoT officials; one VAWC-PC). In the words of a district AIDS control coordinator (DACC) in Njombe TC, “They have helped in training CCWs by preparing guidelines and tools. They have built capacity of the whole system, from the village to the national level.” Similarly, a CSO in Bukoba DC said, “CHSSP has helped much in making sure that there is national integrated case management system and in making sure these things are sustained.” 4.4.2 Conclusions • CHSSP made important contributions to revising, developing, and disseminating policies and guidelines that have provided a framework for facilitating ongoing reforms and improving the structure and systems for community health and social welfare services. • CHSSP contributed to building the capacity of CCWs and educating them on the case management system operations, guidelines, and tools. CCWs are now more successful in their roles because of CHSSP training, guidance, and supportive supervision. • CHSSP’s CCW training contributed to strengthening to the community health and social welfare system, both by building CCW capacity and by improving system functioning related to referrals, service linkages, and access to and use of health and social welfare services. CHSSP’s support has been less effective in terms of strengthening the local systems’ response to VAWC. 4.5 EQ 2.1: WHAT CHALLENGES ARE HEALTH AND SOCIAL WELFARE ACTORS FACING IN FULFILLING OR IMPROVING THEIR ROLES IN PROVIDING COMMUNITY HEALTH, SOCIAL WELFARE, AND HIV PREVENTION AND TREATMENT SERVICES TO KEY POPULATIONS? 4.5.1 Findings Key informants were asked to reflect on the challenges they have faced in providing health and social welfare services. Broadly, they cited challenges related to beneficiary engagement, logistics, human resources, material resources, and capacity-building. 13 Beneficiary Engagement Across stakeholders, beneficiary circumstances posed a significant challenge to gaining buy-in and cooperation, particularly for CCWs. Of the CCWs surveyed, 43.1 percent identified the lack of beneficiary cooperation as a principal challenge (Figure 1). In all five CCW FGDs, moreover, respondents indicated that MVC households expressed expectations of material support beyond the scope of the CCW duties, including money for basic needs (four of five FGDs), support for school fees (two of five FGDs), and support for healthcare costs (one of five FGDs). As noted by one CCW, “The challenge is when one of your beneficiaries does not have food or they are sick, and they expect to get support, while you don’t have anything to offer. It becomes a challenge.” In all FGDs, CCWs perceived that these expectations were due to previous support models that provided beneficiaries with material support. Figure 1: Primary Challenges Faced by CCWs 43.1% 39.8% 24.2% 20.8% 8.6% 4.1% 3.7% 3.3% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% 50.0% Lack of beneficiary cooperation Lack of transport Insuffient payment Wide coverage, hard to reach all Too many households to visit per CCW Poor handling of cases by service providers Lack of support from the SWO Lack of support from the lead CCW In two FGDs, CCWs noted that beneficiaries cited the CCWs’ allowance as the reason they should receive material support. Initially, this made beneficiaries reluctant to trust the CCWs, with whom they viewed their relationship as more transactional than relational. CCWs cited other barriers to establishing trust with beneficiaries, particularly around confidentiality concerns related to their HIV status. In some cases, beneficiaries were not receptive to messaging about HIV testing as they feared that community members learn their status and ostracize them (two of five CCWs and one of 11 district actors). Respondents also cited challenges around beneficiary engagement in referrals for violence cases (two of six implementing consortium members, one of 11 district actors, and one of five CCWs), including both gender-based violence (GBV) and violence against children (VAC). Consortium members reported that the primary challenge for violence prevention was tackling community attitudes and norms towards violence, which are deeply embedded and resistant to change. District actors further reported that getting beneficiaries to follow through on violence referrals was difficult, as community members were accustomed to addressing violence cases within the family and without engaging external actors (one of 11 district actors). In one CCW FGD, respondents reported cases in which children were experiencing violence in the home but would not open up about the violence in front of their parents. Logistical Challenges CCWs experience a number of logistical challenges, including report transmission and travel. CCWs reported that, although they have been provided with subscriber identification module (SIM) cards, sending reports via mobile phone was very challenging due to network connectivity problems (three of five CCWs). Such complications with report submission are time-consuming and can cause errors or inconsistencies in beneficiary data. One CCW FGD reported that such challenges create reporting problems as it makes it appear that they have not completed any service provision. In another FGD, CCWs reported that some CCWs were using paper reports, while others were using electronic reports. They attributed the difference to variations in training and added that it caused confusion for many CCWs. CCWs also reported challenges using paper case management forms. In 14 these instances, delays occur in receiving case management forms (two of five CCWs). While they are supposed to receive case management forms at the beginning of every month, they consistently received them at the middle of the month. District actors reported having difficulty delivering CCW case management forms on time due to limited resources for printing (five of 11 district actors). Respondents also reported that CCWs face logistical challenges during home visits, particularly when traveling long distances between homes.17 Each CCW is asked to cover 20 OVC households and receives a monthly stipend of TZN 35,000 (US$15) payable on submission of the requisite forms at month’s end. While CHSSP/Kizazi Kipya provided bicycles to selected LGAs, CCWs are expected to provide their own transport and often for their clients to access referred health or social services. Not surprisingly, 40 percent of surveyed CCWs identified lack of transport as one of their primary challenges. These logistical challenges were more pronounced during the rainy season when poor road conditions make travel more difficult. Human Resources Limited human resources posed challenges for health and social welfare actors. Respondents report that CCW caseloads are too high to ensure proper service provision.18 CCWs were originally assigned 20 children each, but this was increased to 20 households. This is particularly challenging households with multiple children. Not surprisingly, 30 percent of respondents cited their primary challenge to effectively doing their job is serving so many households across their assigned geographical areas. Key informants reported that the number of social welfare officers (SWOs) is too low to manage incoming cases.19 This also creates challenges for CCWs that are reflected in service provision delays. One district￾level CSO reported that only 40 percent of wards have active assistant SWOs, most of which are concentrated near Dar es Salaam. As such, other actors are taking on the SWO role to fill gaps and supervise the CCWs, in addition to their existing responsibilities. Similarly, one VMAC reported that the local government does not have the capacity to hire additional SWOs at the ward level. Personnel turnover also created human resource gaps related to CCW supportive supervision. CCW supervisor turnover is high, which means that experienced LGA supervisors are often replaced by inexperienced LGA supervisors. CCWs cited similar challenges in their coordination with the district community development officer (DCDO), where new DCDOs move in to replace better-trained DCDOs (one of five CCWS). Capacity-Building Gaps Capacity-building challenges make it difficult for local system actors to fulfill their roles. At the CCW level, capacity gaps make it difficult for CCWs to coordinate with other system actors. For example, CCWs reported that PSWs in their communities were not well-trained, which compromised referrals and linkages quality (two of five CCWs). In one community, CCWs reported that inconsistencies in LGA record-keeping made it difficult to coordinate with PSWs. While there may be several trained PSWs, not all are formally recognized by the LGAs, which interrupts the referral process. In two of five FGDs, CCWs reported that DCDOs were not adequately trained, which made it difficult to follow up or provide beneficiaries with case updates. In another FGD, CCWs reported that DCDOs were replaced with new DCDOs who were not familiar with the community or the referral processes. In addition, CCWs asked for more training so that they would be better equipped to face the challenges 17 Three of five CCWs, two of 11 district actors, and one of four CSOs. 18 Four of five CCWs, five of 11 district actors, one implementing consortium member, and one national-level CSO. 19 One national-level CSO, one of four district-level CSOs, two of 11 district actors, and one of four VMACs. 15 they encounter. Although CCWs had positive perceptions of the CCW training, they felt it was rushed and there was too much material to cover in the allotted time period (five of five CCWs). Other stakeholders noted the need for additional training for PSWs, SWOs, and DCDOs.20 MACs also reported a desire for additional training to understand their roles better, address emerging issues, and serve PLHIV more effectively (one of four VMACs and two of four WMACs). District-level CSOs argued additional capacity-building would improve actors’ ability to address bottlenecks facing children, girls, and women in accessing services and would improve understanding of policy and guidelines for proper service delivery (two of four CSOs). Regional and district actors further argued that capacity-building was necessary to address challenges around stakeholder coordination (one of five regional actors and two of 11 district actors). 4.5.2 Conclusions • Notwithstanding gains made in building human resource capacity and strengthening community health and social welfare systems, substantial challenges remain. Principal challenges involve dysfunctional modes of beneficiary engagement with CCWs and other local health and social welfare system actors resulting from improper beneficiary expectations; concerns about confidentiality, particularly surround beneficiaries’ HIV status; intransigence on attitudes and norms surrounding VAWC; logistical challenges related to the shortage of tools, equipment, and resources; heavy caseloads and geographic coverage combined with a lack of transport; and lingering capacity gaps among local system actors. 4.6 EQ 2.2: WHAT CAN CHSSP AND OTHER PROGRAMS, INCLUDING KIZAZI KIPYA, DO BETTER TO ASSIST HEALTH AND SOCIAL WELFARE ACTORS TO ADDRESS THESE CHALLENGES? 4.6.1 Findings Respondents were asked to reflect on how CHSSP and Kizazi Kipya could enable health and social welfare actors to address these challenges. Across respondents, three broad recommendations emerged: additional training, additional resources, and increased coordination. Additional Training Respondents reported that additional, or refresher, training would enable them to do better in their various roles (two of 11 district actors, two of five regional actors, two of three PLHIV clusters, two of four WMACs, and one of two KIIs national-level CSOs). As one national-level CSO reported, “After the support, there was no system of going back for a refresher course…I can personally say it was a weakness, and for me I can say it is a challenge.” Respondents also perceived that these refresher trainings would improve coordination between other actors because each actor would have a better understanding of his or her roles and responsibilities. CCWs also reported that extended trainings would enable them to better fulfill their duties. Implementing consortiums expressed an awareness of the desire for additional training and noted that more comprehensive training is needed across the board (two of six implementing consortium members). The sole interview with USAID also indicated a need for additional training across the board and noted an interest in strengthening communities and formalizing cadres. Additional Material Resources In addition to more training, respondents reported that the provision of additional material resources, including tools and funding, would improve service provision. First, respondents reported that the transport allowance allocated to CCWs was not enough to cover the distance they traveled, or the time 20 Two of three PLHIV clusters and two of 11 regional actors. 16 spent in service provision (five of five CCWs, five of 11 district actors, and one of two national-level CSOs). These respondents noted that, while CCWs were fully aware of the voluntary nature of their roles, their roles make it difficult to maintain other income-generating activities while fully serving as CCWs. One FGD with CCWs also reported that in some communities, CCWs receive their stipends late. This was demotivating for the CCWs and contributed to CCW dropout. CCWs also often use personal funds when escorting beneficiaries to health centers, which quickly exhausts their travel allocation (one CCW). Respondents cited other material resource needs necessary for improved service provision. These included: identification badges for CCWs to improve formal recognition in the community (two of five CCWs and two of 11 district actors); increased resource allocation for stocking condoms at health centers (one of five CCWs); funding to support regular stakeholder meetings to share updates and best practices (two of 11 district actors, two of three PLHIV clusters, and one of four district-level CSOs); and provision of more printed materials for HIV/AIDS committees (one of 11 district actors, one of five VMACs, and one of four WMACs). Increased Coordination Lastly, respondents reported that increased coordination would improve their ability to address challenges. District actors and PLHIV clusters perceived that increased coordination could be achieved through inclusive work plans between actors (three of 11 district actors and two of three PLHIV clusters). Regional actors and National GoT representatives (one KII each) believed that increased coordination could be achieved through better coordination with CSOs. These respondents believed that CSOs could do more to facilitate transparency and include LGAs in planning and data sharing. 4.6.2 Conclusions • Primary needs among local health and social welfare system actors reflect the challenges they face— including the need for training local system actors at all levels of government; additional tools, equipment, and resources; and improved coordination among local system actors. 4.7 EQ 2.2: HOW DID CHSSP SUPPORT ADDRESS ISSUES RELATED TO WOMEN, GIRLS, AND YOUTH, INCLUDING, FOR EXAMPLE, FACTORS LIMITING THEIR ACCESS TO COMMUNITY HEALTH AND SOCIAL WELFARE SERVICES AND/OR THE QUALITY OF COMMUNITY HEALTH AND SOCIAL WELFARE SERVICES THEY RECEIVE AND WITH WHAT RESULTS? 4.7.1 Findings Respondents were asked to reflect on the changes they have observed in access to and quality of services for vulnerable populations. Respondents cited several improvements in HIV treatment and prevention, violence response, quality services, and organizational capacity. HIV Awareness, Treatment, and Stigma All CCWs reported an increased community awareness about the importance of HIV testing contributed to reduced stigma around HIV. CCWs reported that, prior to the program, beneficiaries tended to hide their HIV status and, to a greater extent, their children’s HIV status. CCWs also reported that beneficiaries are much more open to testing their children, receiving testing, and, in cases where they tested positive, starting their medication early. In four FGDs, CCWs reported that educating beneficiaries on proper medication use enabled them to take their medicine more consistently. Next, CCWs reported that community members improved their understanding of how to minimize HIV transmission, particularly mother-to-child transmission (three of five CCWs). VMACs expressed similar sentiments, noting that community-wide awareness of and action towards HIV prevention and treatment had increased (three of four VMACs). These respondents reported that the improved accessibility of HIV commodities contributed to increased rates of treatment. 17 Gender-Based Violence and Violence Against Children Respondents also reported that CHSSP has facilitated changes in violence prevention and response services. CCWs reported changes in disciplinary practices among parents (one of five CCWs), while PLHIV clusters reported improved training around GBV. These informants reported that government actors, particularly security entities such as gender and police desks, are much better equipped to address GBV and VAC issues (two of three PLHIV clusters) than before. These findings, however, must be taken with caution in light of the other findings (see above) that community attitudes and norms of behavior around GBV and VAC are deeply embedded and resistant to change. Quality of Services Respondents reported that CHSSP helped beneficiaries access quality services. CCWs reported that beneficiaries are generally happy with the quality and speed of services and are happy with no longer having to wait in long lines to receive services (two of five CCWs). CCWs also reported that service providers within the referral system have expanded their hours of operation, allowing beneficiaries flexibility in accessing services. Regional and district actors also reported changes in service quality, indicating that service quality improved because providers are trained, and guidelines have improved (one of 11 district actors and two regional actors). Organizational Capacity Several stakeholders reported improvements in health and social welfare system capacity as a result of CHSSP. As one implementing consortium member reported, “CHSSP has strengthened the organizational and management capacity of CSOs to implement quality HIV/AIDS and social welfare services to key populations at the community level.” Respondents indicated that capacity-building activities have had positive effects, including improved record keeping and data collection (one VMAC and one regional actor). One district actor reported that the case management model has created uniformity across actors, while two regional actors report improved the advocacy and implementation capacity of WMACs and VMACs. These respondents also indicated that social welfare actors have been empowered to take ownership of their activities. PLHIV clusters also reported improved systems of leadership, improved coordination between district councils, health centers, and service providers and, importantly, improved guidelines at the grassroots level—particularly for WMACs and VMACs (three of four PLHIV clusters). VMACs perceived similar benefits, including improved cooperation of leaders at the street level, more prioritization of HIV issues among government stakeholders, and strengthened committees. These respondents perceived that these changes, particularly those around improved coordination and guidelines, allowed for consistent practices around quality service provision. 4.7.2 Conclusions • CHSSP has contributed to improvements in community-based health and social welfare services and outcomes related to women, girls, and youth in terms of increased HIV awareness and treatment and reduced stigma; incremental improvements in GBV and VAC awareness, prevention, and response; improved quality of health and social welfare services; and increased capacity within the health and social welfare system and among system actors. 18 4.8 EQ 3: TO WHAT EXTENT HAS CHSSP CONTRIBUTED TO CREATING AN IMPROVED ENABLING ENVIRONMENT AND MORE FUNCTIONAL AND BETTER COORDINATED COMMUNITY STRUCTURES AND SYSTEMS FOR PROVIDING COMMUNITY HEALTH AND SOCIAL WELFARE SERVICES TO PRIORITY AND KEY POPULATIONS, PARTICULARLY AGYW, MVC, AND PLHIV? The ET interprets EQ 3 to refer to the extent that CHSSP achieved its program Objective 2 as described in the Program Background section above. 4.8.1 Findings Strengthening the Enabling Environment CHSSP implemented number of activities to: 1) improve the policy environment and strengthen coordination; 2) build partnerships and organizational leadership; and 3) support management capacity of community structures—namely MACs, VAWC-PCs, CSOs, and PLHIV clusters at the village, ward, and/or council levels. At the national level, and as discussed above, CHSSP worked with MOHCDGEC, TACAIDS, and PO-RALG to incorporate PEPFAR 3.0 in the revision, development, and dissemination of policies and technical guidance for health and social welfare system actors. To improve system functioning at the regional and local levels, CHSSP provided a range of capacity-building and training to MACs, PLHIV clusters, VAWC-PCs, and CSOs. Key system actors participating in these activities included regional coordinators for TACAIDS (RCTs), CHACs, SWOs, and gender focal persons. In addition, CHSSP revitalized and supported regular meetings of key technical working groups, task forces, advisory groups, and coordination committees. This improved coordination and enabled them to add the national HIV response to the regular meeting agendas and to advocate for updating policies, guidelines, and implementation practices to reflect PEPFAR 3.0 and improved gender sensitivity. Table 5 summarizes CHSSP achievements in improving the health and social welfare enabling environment. Table 5: CHSSP Achievements in Improving the Health and Social Welfare Enabling Environment Achievements Worked with MOHCDGEC to introduce the NICMS, developed guidelines and tools, and trained system actors in its implementation. Integrated the HIV-sensitive case management system into the National Plan of Action to End Violence against Women and Children 2017-2022. Worked with LGAs through CMACs to cascade the revitalization of MACs at the ward and village levels. Supported the National Council of People Living with HIV (NACOPHA) to engage council PLHIV clusters to define capacity development needs. Created improvement plans and supported meetings of PLHIV clusters to improve coordination of HIV interventions. Supported and facilitated national-level coordination meetings, such as the Impact Mitigation Technical Working Group, while advocating for the MACs guideline/curriculum improvement. Worked to revitalize, form, and develop the capacity of council VAWC-PCs. Strengthened LGA capacity to coordinate system actors in the provision of HIV/AIDS services to increase uptake of care and treatment services and reduce stigma for PLHIV. Capacity-Building and Structural Strengthening of MACs As of January 2019 (in PY 5 of the no-cost extension), CHSSP disseminated the National Multi-Sectoral HIV and AIDS Stigma and Discrimination Reduction Strategy 2013-2017 in 54 councils to support reduced stigma and discrimination against PLHIV. Moreover, CHSSP helped establish/revitalize and equip 54 MACs at the council level, 270 MACs at the ward level, and 1,350 MACs at the village level, thereby strengthening capacity to coordinate HIV/AIDS interventions. In the process of orienting MACs, the program realized that the 2008 MAC curriculum was outdated and lacked guidance on achieving the new global 90-90-90 goals. It thus advocated for a process for updating 19 the curriculum for training council, ward, and village MACs. The new curriculum has two modules that incorporate PEPFAR 3.0 priorities for meeting the needs of PLHIV, MVC, and AGYW. CHSSP assistance to WMACs and VMACs has helped them become more active and operational, helped improve adherence with national guidelines, helped improve the quality of services that committee members provide, and/or helped improve their coordination and outreach (eight of eight WMACs and VMACs). The following quotes are illustrative of what the ET heard from WMAC and VMAC members during FGDs: “It’s important to mention the MAC guidelines. We are guided by the government to determine the MAC’s functionality…at the moment we speak the same language with the government, and this is embedded in the assessment tool we use.” – WMAC member, Temeke “Through capacity-building trainings provided by JSI, we have had the systems improved and strengthened. We have been trained on how the structure of these committees should be. Unlike in the past where when we did not have any guideline structures on how these committees should look like or should function. – VMAC member, Temeke “After trainings from JSI, CCWs are now able to coordinate work reports with the committee reports. Before their reports did not come directly to the committee, but after capacity-building, there are now joint plans between them and the committee. Now we visit them, they welcome us in good faith and are ready to express their problems.” – VMAC member, Nyamagana MC Capacity-Building and Structural Strengthening of VAWC-PCs As of January 2019, CHSSP had supported the establishment of VAWC-PCs in 52 councils and supported the completion of referrals for 44,914 OVC/MVC and their families to services (up from 731,327 OVC/MVC and their families during 2014-2018). Although less institutionalized, CHSSP’s efforts in capacitating VAWC-PCs have begun to take hold; guidance and training have led to their establishment at the district level in 52 of the 107 target councils. CHSSP is working with PO-RALG and MOHCDGEC to develop a Comprehensive Council Social Welfare Planning Guide to guide the planning of social welfare services, including VAWC-PC activities from the council to village level. Notwithstanding, DMTs in Bukoba noted that, while national plans are in place, further training and cascading of training for VAWC-PCs is needed to capacitate and establish them in more LGAs, “National guidance has provided directions and very good instructions, but when you go to the implementation level, there are people who need to go and revive it to make it exist and direct it what to do and how to do it.” Discussions with VAWC-PC members and other local system actors suggest that CHSSP capacity-building activities with VAWC-PCs contributed to an increased number of VAWC incidents reported and the number of PLHIV receiving treatment and psychosocial support. However, and as noted above, social norms and stigma related to VAWC and HIV/AIDS are deeply embedded within local communities and will take time and effort to redress. PLHIV Clusters In collaboration with NACOPHA, CHSSP has developed a PLHIV Cluster Management Handbook that it used to conduct assessments of 54 PLHIV clusters. CHSSP has further supported 50 PLHIV clusters to develop resource mobilization interventions, enabling them to come up with income generating activities. To strengthen PLHIV clusters, CHSSP provided the means to offer services to more people with HIV/AIDS, a point emphasized by a DSWO from Bukoba DC, “It has helped in minimizing the differences in service distribution. Before follow-up to people living with HIV did not take much consideration of the children. We had a child protection committee, but it didn’t focus much on HIV issues. Therefore, this new system has strengthened the health system.” 20 Additionally, CCWs and other local system actors received training that helped them increase access to HIV services and improve the quality of those services (one of 11 district actors, three of four CSOs, two of six implementing consortium members; and one of two national-level CSOs). Such training helped these local system actors to “become good experts in preparing a care plan and also being able to supervise a care plan. Also, it helped us to be able to improve the quality of services we provide to our beneficiaries…These supports have also helped to reduce HIV infections to the youths, women and key populations at large. For example, HIV services assessment and HIV risky assessment was not done before. But now we do this assessment. Also access to these services has improved, and our clients get services on time and at good quality.” -District-level CSO, Mwanza PLHIV cluster leaders report significant gains from the capacity-building and training they received from JSI (three of three PLHIV clusters). JSI support has helped strengthen local systems for PLHIV which, in turn, gives them voice in their access to quality health services. PLHIV leadership have seen a difference in their organizational development and the enabling environment. As a PLHIV leader in Mbeya CC explained, “JSI has built our capacity in terms of systems. It educated us on how to create groups in the grassroots.” In Njombe TC, PLHIV leaders described a change in their ability to understand and implement national guidelines and understand HIV policies, which they can now cascade to lower levels in ward- and village￾level clusters: “Firstly, JSI has created awareness among us. Secondly it enabled us to understand guidelines on how to lead our colleagues. Third it has given us education which helped to understand about HIV policies; what is it and what is in it.” This has contributed, in turn, to increased access to treatment and care and improved health outcomes for PLHIV communities. The CHSSP capacity-building on implementing the national PLHIV guidelines has helped improve understanding of issues (two of three PLHIV clusters) like access to health insurance used for treatment and care for HIV positive individuals. The guidelines also specified practices for treatment and complimentary health and nutrition and information to help PLHIV recognize their rights as clients. CSO Organizational Capacity Development To improve system functioning at the local level, CHSSP provided a range of capacity-building and training services to district-level CSOs that are Kizazi Kipya sub-grantees. Out of 50 CSOs assisted by CHSSP, the program provided training on resource mobilization to 31, supported two to obtain funds from various donors, assisted 48 with their financial management, installed QuickBooks software at 25, and trained 22 on participatory monitoring and evaluation, enabling them to develop their M&E plans. Prior to providing assistance, CHSSP assessed the CSOs’ institutional capacity using a standardized organizational capacity assessment tool (OCAT). These data were used to customize training and consultative support to each CSO’s specific needs and organizational maturity level. The OCAT assessment focused on seven main domains of CSO organizational capacity: 1) governance, 2) administration, 3) human resource management, 4) financial management, 5) organizational management, 6) program management, and 7) program performance management. Depending on how the CSO scored on the seven domains, it is assigned to one of four levels along an organizational maturity scale: basic, moderate, progressing, and robust (see Table 6). Table 6: Maturity Levels Scores and Descriptions Level (Score) Description Basic (1.0-1.4) The CSO is in the earliest stages of development. All the components measured are in the earliest stage or non-existent. Moderate (1.5-2.4) The CSO is developing capacity in structures and processes in organizational development, sustainability and resourcing, information-sharing, cooperation, and advocacy. Stakeholder and legal and regulatory relations are in place but may function inconsistently. 21 Level (Score) Description Progressing (2.5-3.4) The CSO has a record of accomplishment/achievement. Community stakeholders, government, and other CSOs in the same sector recognize its work. Robust (3.5-4.0) The CSO is fully functioning and sustainable, with a diversified resource base and partnerships with local communities and national and international networks. CSO maturity scores ranged from 2.1 to 3.7 with an average and median score of 3.0. Twenty (20) of the CSOs had maturity scores in the “progressing” range, three had scores in the “robust” range, one had a score in the “moderate” range, and zero had scores in the “basic” range. The domains for administration and financial management were at the lower end of the range (average score of 2.7) due to a few organizations scoring at the basic level in those areas. All other domains had an average score of 3.1. While a post-capacity-building picture is not available due to the lack of follow-up with OCAT, local system actors indicated that CHSSP capacity-building for CSOs has improved their organizational maturity and performance in each of the above seven domains, to varying degrees in each case (four of four CSOs, two of 11 district actors, one of two national GoT officials, and one VAWC-PC). 4.8.2 Conclusions • CHSSP contributed to improving the enabling environment and underlying structure of the community health and social welfare system at the national and regional levels and, in descending order of contribution, at the council, ward, and village levels. This contribution included both strengthening/revitalizing existing attributes and creating new attributes of the enabling environment, structure, and system, including the NICMS, policies and guidelines, and tools and resources. • CHSSP contributed to improving the local systems for coordinating health and social welfare service provision. In particular, CHSSP contributed to making WMACs, VMACs, PLHIV clusters, and VAWC￾PCs more active and operational. Program training, guidance, tools, and supportive supervision have positively impacted the quality of services and support to local structures that committee and cluster members provide. • CHSSP-supported CSOs are beginning from a point of moderate organizational capacity and are already reasonably well-progressed along the maturity scale. Nonetheless, CHSSP support to these CSOs does appear to have contributed to organizational capacity and maturity improvement in multiple capacity domains. 4.9 EQ 4: TO WHAT EXTENT WILL CHSSP ACTIVITIES AND RESULTS BE SUSTAINED AFTER THE PROGRAM ENDS? WHICH FACTORS AFFECT THE PROSPECTS FOR SUSTAINABILITY AND HOW? 4.9.1 Findings CHSSP has adopted a systems approach for improving the provision of community health and social welfare services. A systems approach in this context refers to an approach that sees health and social welfare services as a set of independent but interrelated parts that work together to produce health and social welfare outcomes at the community level. These parts are supported by an enabling environment and underlying structure that determine whether (and how) the diverse parts of the system function. Thus, the key to the long-term sustainability of CHSSP’s activities and results is to what extent these parts continue to function in a coordinated, efficient, and effective manner and the enabling environment and supportive structure remain in place once CHSSP ends. As part of its systems approach, CHSSP is pursuing a three-pronged, mutually supportive strategy. The first prong is to work with the GoT to establish an enabling environment for providing community health and social welfare services consisting of a set of standardized and coordinated policies, guidelines, and tools. The second prong is to create more functional and better coordinated community-level structures 22 and systems in the form of the NICMS, MACs, VAWC-PCs, PLHIV clusters, and CSOs. The third prong is to develop the capacity of health and social welfare actors (CCWs, LCCWs, DMTs, case management coordinators [CMCs]; CSO staff; SWOs; and MAC, VAWC-PC, and PLHIV cluster members) responsible for implementing the community-level health and social welfare system. How well CHSSP has done in implementing this systems approach can be seen in the above findings and conclusions. The question here is whether the enabling environment, structure, and system CHSSP leaves behind are strong enough to continue to function over the long term on their own and what challenges exist to achieving this outcome. These are discussed below, beginning with the challenges to program sustainability and followed by what the program has put in place to contribute to enhancing program sustainability. Challenges to Program Sustainability Lack of Funding Challenges related to funding came up consistently across all KIIs and FGDs from the national level down to the village level. The withdrawal of funding support from CHSSP and other international donor initiatives will pose a severe challenge to system sustainability. This challenge is exacerbated at the LGA level in that that LGAs are almost wholly dependent on revenue pass throughs from the national government. This gives them limited flexibility to raise or maintain spending levels on health and social welfare. The essential problem can be stated this way: “In everything we need money. For example, when they come for training, they are given money (transport allowance), when we finish the training, we hand them over to the government, and those are community volunteers so they cannot sustain themselves with whatever income that they have. You ask someone to go and help the orphans where they are, but they stop doing that because they miss money for transport. But also, on supervision from the district and ward SWOs, sometimes they lack commitment because there is no money.” – National-Level CSO Funding shortfalls extend to basic infrastructure requirements like offices, equipment, and tools, all of which affect the ability and motivation of system actors to do their jobs. As noted by a PLHIV cluster member in Njombe TC, “We do not have a cupboard to keep our files or chairs and table for our office. Also, there is no stationery and no money to print forms. We need money to run our office.” While CHSSP is working with system actors on fundraising and income generating strategies (see below), there is a finite amount of money to go around, at least for the foreseeable future, in the form of revenue pass throughs or in external financial support. In the former case, capturing health and social welfare priorities accurately in the LGA budget does not guarantee their funding, as they are competing with other LGA and higher-level government spending priorities. In the latter case, there are multiple LGAs and CSOs chasing a limited supply of external funding or in-kind donations. Some may be successful but, given the limited supply of external support available, not all will be. Thus, while some LGAs may be able to fill funding gaps, other LGAs will struggle to finance system functioning. The challenge system actors face in this regard is aptly reflected in the following two quotes. The first is from a DSWO in Mbeya CC, who said, “Currently with the funding, I see the functionality is 100 percent, but if the funding comes to an end, I think we will step back to zero.” The second quote, from a VMAC member in Njombe TC, emphasized how the absence of allowances for committee members acts as a disincentive for long-term participation in the committee: “We would love for them to continue because they are the ones that have made the work easier, but at a certain stage, continuing to work without any allowance is difficult. You can call the committee members the first time, and they come. If they go back without any allowances or bus fare, and you summon them again, only half will attend, and if you call them a third time, you might find that none will attend.” Turnover Among Key System Actors 23 Turnover among key system actors was cited consistently across KIIs and FGDs as a significant challenge to program sustainability. This is particularly true among the MACs, which include elected officials with fixed terms of office and other LGA officials who rotate from one assignment to another. Such turnover leads to implementation discontinuity, loss of institutional knowledge and memory, shifting priorities, and loss of capacity replacements often have not received the same training as their predecessors. A member of the WMAC in Temeke MC stated this challenge, “I can say that the frequent reforms of these committees will affect these practices from being sustainable. For example, the current chairperson formed this committee, but when another chairperson comes, he will deform this committee and will form a new committee and thus committee members will lack training.” According to the same WMAC member, the solution to this challenge is “continuous strengthening of these CMAC to be strong because I do see them dormant. They need to be strengthened by being provided with frequent trainings. If we receive trainings so that we can cope with these changes, then these committees will continue being strong.” Lingering System Gaps and Weaknesses Like any program of its kind, CHSSP faces its own resource constraints. This means it cannot address all the issues affecting the health and social welfare structure and system, nor can it always address its targeted issues to the degree it might wish, whether due to resource constraints or due to on-the-ground impediments like lack of full cooperation by other system actors. Thus, there will inevitably remain certain issues that are not addressed or not addressed adequately. Challenges related to providing supportive supervision is a particularly notable example of this. Changing the attitudes and behaviors of system actors requires ongoing support and reinforcement, but CHSSP is constrained in how much supportive supervision it can provide, a point noted by this RSWO-RCDO in Mbeya, “The structure and systems demand that ward committees must go down to village level, but it has not been successful due to lack of close follow-up. For example, regional level, district level, ward level, to village level, there’s been no follow-up.” LGAs are also often missing key system actors, such as community development officers (CDOs), CHACs, or assistant SWOs. In the case of the missing assistant SWOs, CMCs from the local CSOs are filling the gap.21 Meanwhile, other LGA officials are filling in the gap to supervise the CCWs, despite their other responsibilities. This is not how the system was designed to operate. Such a patchwork filling of roles by system actors is ultimately an unsustainable model. Notwithstanding CHSSP’s efforts, there will always exist capacity gaps, and the further one goes down the LGA administrative levels, the larger the gaps. As a CCW supervisor in Njombe TC noted, “I think the ward committee has much progress because it has a lot of experts, contrary to the village committee where the only educated people are the local government executive, teachers, and nurses.” Dependence on Volunteer Labor The community health and social welfare system depends on the willingness of thousands of volunteers to donate their time and effort. Salaried government officials do participate in MACs and VAWC-PCs, but they do not receive compensation for it. CCWs receive a small stipend, but that is all. While many system actors are willing to volunteer their time, others are less willing, a point emphasized by the CHAC in Yamagana MC who said, “For me the biggest challenge that I see, you know this is a voluntary work, so one can do it or decide not to do it. This makes committees to sometimes not be so strong. One has to go out to find a daily bread for his or her family.” 21 CMCs are funded by local CSOs. In a typical council with 25 wards, one CMC is assigned to cover eight wards and is expected to spend 15 days in the field visiting each ward at least once a month. CMCs meet monthly with CCWs where the CCWs report on their activity during the previous month. The CMC reviews this information and provides a scheduled training to the CCWs on different topics. Lead case workers provide CCWs with more training and support the CCWs five days a month. 24 As a cautionary tale demonstrating the above challenges, a senior GoT official told how, once the health reform/strengthening program implemented by DANIDA ended, gains were not sustained, and conditions reverted to where they were prior to the program: “That’s when committee members changed. We had no money to conduct trainings, though we encouraged people to sit in meetings even without being provided allowances. So, after this program phased out, these committees failed in performing their roles.” The silver lining to this cautionary tale was that lessons learned from this failure were incorporated into the CHSSP design, as the same official explained, “We realized that problem, and that is why when we improved these guidelines, we added some instructions to coordinate stakeholders and in writing proposals so that they can be financially stable and able to run these committees.” Incremental and Tentative Nature of Program Contributions CHSSP’s contributions to improving the community health and social welfare enabling environment, structure, and system include both substantive and incremental contributions. Substantive contributions represent significant advances from the previous status quo and show evidence of moving toward institutionalization with the GoT. These include CHSSP’s contributions to improving the enabling environment via revised or new policies and guidelines, in creating the NICMS, and in forming and revitalizing the community-level committee structures (MACs, VAWC-PCs, and PLHIV clusters). Incremental contributions represent positive improvements to the status quo but are tentative in nature because they have yet to be consolidated and show less evidence of institutionalization. These include CHSSP’s contribution to improving the capacity of local system actors and constructing incentives for them to play their roles as intended despite the numerous challenges or constraints they face. The fundamental dilemma, as implied by the systems approach, is that all individual elements of the system must function for the entire system to function. Thus, even if a strong enabling environment and community level structures exist, if the system actors do not play their intended roles, the system will break down. While evidence does indicate that the capacity of system actors has improved, there remain significant gaps. In addition to this, improvements in the community-level structures and in system actors’ capacity are unevenly distributed across the program’s operational areas and across different government levels within those operational areas. Factors Contributing to Program Sustainability Fundraising and Income Generation CHSSP is well-aware of the funding challenges faced by local health and social welfare system actors and is addressing them by working to build their capacity to generate money from external sources. Efforts include writing grant proposals, engaging in income-generating activities, advocating to ensure that health and social welfare priorities are reflected in LGA budget proposals, and holding councils responsible for distributing budgeted funds to the wards and villages. To achieve this objective, the program is promoting collaboration between the diverse system actors. CHSSP anticipated that working with LGAs and CSOs on revenue generation strategies, including mapping potential funding sources and capacities, would create competition for a limited pot of external funding. One way the program has dealt with this challenge is to work with CSOs to create partnerships or consortiums to jointly pursue external funding opportunities. JSI reports some limited success in this area, although it has fallen short of actually generating external funding. As one informant said, “We’ve been encouraging them [CSOs] to create partnerships and consortiums. When they join efforts, they can win funding together. This is something we’ve been pushing them to do. We’re seeing some partnerships and some who have become subs.” Addressing the Turnover Challenge CHSSP is addressing the turnover challenge in a couple of ways. One is its ongoing supportive supervision for the health and social welfare actors in target councils, wards, and villages. However, the number of 25 councils that CHSSP is tasked to support, coupled with the volume of turnover among key system actors within those councils are such that it is unlikely that CHSSP will be able to keep up with the pace of the turnover via its supportive supervision role. CHSSP has also set up a cascading training model using the ToT approach, by which actors at higher government levels are responsible for training actors at lower government levels. A WMAC member in Temeke MC explains how this model works in her case, “The facilitators train others that are trained already. I am one of them. Thus, we are capable to train another 50 people. They can be trained because we have all the tools for training, facilitators are there, and they are capable to train others. Even if the trained facilitators have shifted, we can borrow from the District Council or from Makete or Makambako.” Strengthened Health and Social Welfare Structure and Systems One factor contributing to enhanced prospects for program sustainability is the mere fact that the health and social welfare enabling environment, structure, and system are, on the whole, stronger than they were before CHSSP. This was a finding on which nearly all key informants agreed, as well as on the key role CHSSP played contributing to this outcome. The enabling environment is stronger, community-level structures and systems are stronger and functioning better, and system actors have stronger capacity. Notwithstanding the tentative and incremental nature of many of these improvements, they exist, and they constitute a stronger foundation for ongoing system functioning than before. This idea is captured by the WMAC member in Mbeya who observed that, “Most of practices will continue because we have budgets at district councils, though these budgets are small, but at least we have it. It may not support all activities, but I am sure some of activities will be supported, for example events like child day, women day, day for people with disabilities must be supported.” Program Activities Transitioned to the GoT Sustainability is enhanced to the extent that the program is able to transition its activities to its GoT counterparts prior to the program’s conclusion. On this criterion, CHSSP has achieved a number of successes, which are listed in Table 7. 26 Table 7: CHSSP Activities Transitioned to the Government of Tanzania Transitioned Activities MAC guidelines and curriculum to TACAIDS for implementation of committees at council, ward, and village levels. NICMS Framework, which was adopted by PO-RALG and MOHCDGEC as the standardized system to coordinate service providers working with children across the health, protection, and social welfare sectors. NICMS training, data collection, and reporting tools, including the CCW Training Manual, CCW Practical Handbook, national MVC household registration form, screening and enrollment tool, HIV risk services and adherence assessment tool for children and adolescents, child assessment job aid, child needs and care plan form, MVC referral form, monthly MVC services form, graduation tool, case closure form, and monthly tracking form. Community-based health program design guide, including guidelines on CHW technical and administrative supervision, CHW governance structure, and roles and responsibilities. While transitioning activities to the GoT does not guarantee their ongoing implementation over time, it provides a basis by which they might be continued, which would not have been the case had they remained under the program’s purview. Linking System Actors A hallmark of CHSSP’s systems approach is linking different system actors with each other to coordinate responses to community health and social welfare needs. Previously, it was common for the different system actors to operate in silos; however, CHSSP has worked to link these actors together, including merging different community structures into a single one (e.g., VAWC-PCs),22 linking governmental actors together, and linking governmental actors to non-governmental actors. Key informants consistently noted how the program has helped them forge relationships with LGA officials and other community-level system actors, whether by “linking us up with the government by getting the CHAC involved” (PLHIV Leader, Njombe TC) or “helping us build good relations with social welfare and community development officers at the ward level” (District-level CSO, Njombe TC). The regional community development officer (RDCO) in Bukoba DC described how this approach helps promote program sustainability, “Sustainability is when these systems are introduced and are connected with the government. That’s why many sectors are involved from the ward to district level, which will remain after the program has phased out. Sustainability will never be there if we do not involve community so that they take ownership of the program.” NICMS CHSSP’s contribution to the NICMS development is a potentially important pillar of program sustainability. This is not to say that the NICMS lacks its own challenges, which are described in the findings for EQ 2. Despite these challenges, 93.4 percent of CCW survey respondents said they will continue to work as a CCW over the long term (Figure 2). When asked why they expected to continue working over the long term, just over one-half of CCWs said it was because they were passionate about and committed to the work, whereas 40.5 percent it was part of their day-to-day duties and thus something to get done. As observed by this CCW in Mbeya CC, “I plan to continue because I am a health worker in a community level. This activity is in my blood. I would like it to be sustainable.” Or, as further noted by the RCDO in Bukoba DC, “They have turned their mindset to work voluntarily without pay. When a person is committed to the public 100 percent, they can’t go back; it’s unbecoming. The support which they are given is little, but it can’t be compared with the sociological benefit. They are aware that it’s part of their contribution to their community.” 22 VAWCs were established by merging the existing committees that were focusing on violence prevention and response. The merged committees included MVCCs, child protection teams, child labor, and GBV. 27 Figure 2: Primary Reasons to Continue Working as CCWs Over the Long Term (N=269) 52.8% 40.5% 10.8% 5.9% 1.9% 1.9% 1.1% 0.7% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% Passionate and committed to the work Part of my day to day duties Beneficiaries cooperate Number of households is appropriate Motivated by the payment received Receiving sufficient support from lead CCW Receiving sufficient support from SWO Referral authorities handles cases very well The survey next asked why CCWs may not continue working over the long term. Their answers, seen in Figure 3, reflect the primary challenges identified earlier, albeit in different order of importance. In this case, the lack of transport was offered as the primary reason why CCWs may choose not to continue to work as a CCW at 32.3 percent of respondents followed, at distance, by insufficient payment (17.5 percent), lack of beneficiary cooperation (11.5 percent), and challenges serving a large number of households (8.2 percent). Figure 3: Primary Reasons Why CCWs May Not Continue Working as CCWs Over the Long Term (N=269) 32.3% 17.5% 11.5% 8.2% 4.5% 3.0% 2.6% 2.2% 8.6% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% Lack of transport Insufficient payment Lack of beneficiary cooperation Wide coverage, hard to reach all Poor handling of cases by service providers Lack of support from the lead CCW Too many households to visit per CCW Lack of support from the SWO Don’t know CHSSP has taken a number of steps to increase the sustainability of the CCW model, beginning with CCW selection. CCW selection is a participatory process that involves LGA officials and other prominent community residents beginning at the council level and then moving progressively down to the ward and village levels. The process begins with an awareness meeting that explains the process and encourages the local officials/residents to take ownership. The participants identify a list of people that are drawn from the relevant communities from which it selects the CCWs using a strictly applied set of selection criteria, including: 1) must be Tanzanian citizen, 2) must be over 25 years old, 3) must be able to read and write, 4) must be from the relevant community, and 5) must be willing to volunteer. Preference is given to candidates with previous experience as a PSW or community-based volunteer, such as CHW. (A significant share of current CCWs are former CHWs). To discourage unqualified CCW candidates and encourage qualified CCW candidates, the program emphasizes transparency so that candidates clearly understand the volunteer nature of the position and what it entails. The fact that over 90 percent of CCW candidates undertake and complete the five-day CCW training suggests that the approach is successful. According to a CCW supervisor from Njombe TC, “Also, in my ward when we select the CCWs, there are criteria that we use because there are some people 28 when you tell them there is a job, they immediately ask how much they are going to be paid. When people ask such questions, we leave them. We look for people who are not driven by money.” Moreover, the fact that a large cohort of CCWs are drawn from the ranks of former PSWs and CHWs contributes positively to the sustainability of the approach. Other steps instituted by CHSSP and Kizazi Kipya to increase the sustainability of the CCW model (and NICMS) include refresher CCW trainings in addition to ongoing supportive supervision visits; turning monthly CCW meetings over to LGA officials, such as assistant SWOs, to ensure that LGAs are assuming ownership of the process; continuing the training of district trainers, including LGA officials and CSO staff, under the cascade training model so that CCW training will be continued after CHSSP’s departure; developing a YouTube channel with refresher CCW training videos; offering monthly in-service trainings at the ward level; and developing topical job aids for household visits on topics such as positive parenting, problem-solving, nutrition, violence prevention, early childhood, and responding to crises. 4.9.2 Conclusions • The systems approach adopted by CHSSP is a sound strategy, relevant for the context in which CHSSP is working; it represents perhaps the best way to secure the sustainability of health and social welfare system reforms. • CHSSP contributed to strengthening the enabling environment and community-level structures for community health and social welfare services. These contributions are substantive in nature and show evidence of advancing toward institutionalization within the GoT, particularly in the enabling environment, which is a pre-requisite for system sustainability. • A number of threats to system sustainability exist, however. They include a lack of funding to operate the community-level structures, incentivize local system actors, and pay for infrastructure, like offices and equipment. Turnover among key system actors results in implementation discontinuity, loss of institutional knowledge and memory, shifting priorities, loss of human resource capacity, and lingering human resource gaps. There is also heavy a reliance on volunteer labor and ongoing commitment and good will of system, notwithstanding disincentives. Many gains achieved are incremental and tentative, particularly related to strengthening system actor capacity, coupled with uneven program penetration into target councils and (to a much greater extent) wards and villages. • CHSSP has incorporated into its systems approach a number of strategies to address the various sustainability threats. These include working with LGAs and local CSOs on fundraising and income generation strategies; creating a system of cascade training with a cadre of trainers at the region, council, and ward levels; and transitioning key functions for running and maintaining the health and social welfare system to the GoT. CCWs are carefully screened, are committed to their work, and benefit from a variety of ongoing support strategies, all of which bode favorably for sustainability for the case management approach. • The sustainability of the reforms achieved and set in motion by CHSSP will depend on the continued functioning and eventual institutionalization of each of the three key elements of the community health and social welfare system targeted by CHSSP. 5.0 RECOMMENDATIONS CHSSP, or a follow-on program, should continue to roll out training and capacity-building to meet targets with emphasis on the ToT approach. Only five wards per council are currently being reached by CHSSP, and this should be scaled up to reach local structures at the ward and village levels. • 29 • CHSSP, or a follow-on program, should consider augmenting its current CCW training with other technical topics, such as violence prevention, early childhood, responding to crises, HIV prevention, treatment and care, etc. Kizazi Kipya provides some supplementary technical training and job aids outside of their initial scope, but other support is needed to bring these efforts to scale. CHSSP, or a follow-on program, should allocate additional resources and support for DMTs to provide follow-up trainings for new CCWs in the wards and villages. The ranks of CCWs are expected to rotate over time as new cohorts of CCWs are brought on board to augment or replace existing CCWs, thus it is necessary that these CCWs receive the same quality of training as the earlier CCW cohorts. Resources—such as venues for training, money for transportation reimbursements, and materials—are needed to perform these follow-up trainings. Bringing CCWs into the district for follow-on activities might also help SWOs provide support and fill some of the gaps left in supportive supervision at the ward and village levels. • In light of 2018 announcements of new target councils, there were new training needs for CCWs, which were not fully met by CHSSP. There should be a clear channel of communication between USAID, CHSSP, and Kizazi Kipya to fill gaps in training and capacity-building in new target councils as Kizazi Kipya proceeds with the next two years of its implementation and a follow-on program to CHSSP is established. • A follow-on program should consider providing operational expenses or other forms of financial or in-kind support for selected system actors, such as CCWs, supervisors, MACs, VAWC-PCs, or PLHIV clusters. Examples of potential support include transportation, Internet, printing, tool maintenance, office space, and stipends. GoT contributions to these costs could then be better articulated in a cost sharing agreement in the first years of the follow-on program together with a sustainability plan with PO-RALG so that these costs will be incorporated into the planning and budgeting tool for LGA budget allotment. • USAID could advocate with PO-RALG for further allocation of budget for human resources, particularly SWOs at the district and ward level, to reduce their caseloads and improve their ability to provide supportive supervision to more wards and villages. • • Considering only five wards per selected district have been reached by the program, additional resources for capacity-building activities should be provided to enable district-level participants to better reach additional wards and villages in those districts. Furthermore, sustaining the cascade of capacity-building from national-level master trainers to district participants will require further donor support to bring this effort to scale. To better implement the National Plan to End Violence Against Women and Children, CHSSP, or a follow-on program, should continue to establish VAWC-PCs as local structures for supporting this effort. Strategies for more effective linkages for local incidents of GBV or VAC are needed to ensure that cases reach the VAWC-PC and that linkages are completed with law enforcement, health, HIV testing and care, and other assistance. • CHSSP, or a follow-on program, should undertake additional sensitization work with communities working through VAWC-PCs and other community leaders to increase awareness of VC and GBV and address social norms to enhance community cooperation in these cases. CCWs should also be better equipped to address these issues and communicate how to handle these cases through the justice, health, and social welfare system. • Sustaining and retaining CCWs would be better enabled by a revised incentive system for CCWs, LCCWs, and supervisors. Increasing stipends and allowances for travel and other expenses would help ensure more use of their time on casework and their long-term commitment (this could be • 30 a shared commitment for GoT to assume over time). Non-monetary incentives, such as public acknowledgement, should also be explored. Time use, retention, and incentives should be further assessed to ensure the long-term commitment of CCWs. Rather than, or in addition to, continuing to support such a large number of district CSOs, CHSSP, or a follow-on program, should consider selecting a smaller cohort of high performing CSOs with a greater chance of obtaining international funding for more focused capacity-building, particularly in the area of fundraising and meeting the reporting and other standards that are required to work at that scale. • Overall, CHSSP, or a follow-on program, should continue to work with the GoT to secure its commitment to funding structures that support vulnerable groups (OVC, PLHIV, people with disabilities, and AGYW) and sustaining the necessary level of human resources. GoT should be further be encouraged to reinforce supportive supervision and follow-up training to ensure the practices supported by CHSSP continue after the program. • 31 ANNEXES 32 ANNEX I: EVALUATION STATEMENT OF WORK PERFORMANCE EVALUATION FOR JSI/CHSSP I. BACKGROUND INFORMATION A. Identifying Information Program/Activity Title Community Health and Social Welfare Systems Strengthening Program Award Number AID-621-A-14-00004 Award Dates November 18, 2014 – November 17, 2019 Program/Activity Funding Global Health-(HIV/AIDS, PEPFAR) – $35,996,685 Implementing Organization(s) John Snow Inc. Research and Training Institute Program/Activity AOR Linda Madeleka B. Development Context a. Problem or Opportunity Addressed by the Program/Activity Being Evaluated Health systems are a relatively well defined and understood at country level. This is not always the case for community systems; and the contribution that strong community systems can make to effective responses to health and social well-being of people. The objective of this program was to develop community systems and strengthening the existing one in order to have improved health and social outcomes. The program aims to bridge gap between facility and community-based services to ensure comprehensive service delivery. The goal of this cooperative agreement aims to improve health, well-being, and protection outcomes for HIV-affected and other vulnerable populations in Tanzania through enabling more accessible, higher quality health and social services for these targeted communities, with a focus on adolescent girls and young women (AGYW), Tanzania’s most vulnerable children (MVC), and people living with HIV (PLHIV). b. Target Areas and Groups JSI original geographic coverage was 84 scale-up councils. To date (FY 2018) CHSPP has covered 67 councils (79.8 percent). c. Intended Results of the Program/Activity Being Evaluated JSI theory of change focuses on Inputs – Outputs – Outcomes and Impact. 33 The logical framework in the diagram above describes the program’s overall strategy of strengthening community health and social welfare structures and support systems to achieve the PEPFAR 90-90-90 goal. C. Approach and Implementation a. Existing Documents – Activity description, work plan, PMP, M&E plan, JSI’s baseline data, quarterly and annual performance reports. II. EVALUATION RATIONALE A. Evaluation Purpose This is the program performance evaluation that is being conducted in order to: i. Assess the program performance and achievements thus far; ii. Analyze the effectiveness of various approaches used by JSI to achieve the four program results and foster sustainability; and iii. Document good practices, gaps/limitations, and lessons learned that will inform the Mission’s design of the follow-on community-based systems strengthening program including critical approaches, focus areas, and interventions. B. Audience and Intended Uses i. Mission’s-technical officer – to inform the design of the follow-on or new community system strengthening award. ii. Host Government – to contribute to the national guidelines and strategies development. C. Evaluation Questions i. To what extent has JSI contributed on improving environment for providing community health and social welfare services (for comprehensive, sustainable, and quality HIV and other services? 34 ii. To what extent has JSI contributed on supporting higher performing human resources for community health and social welfare services (who are able to support AGYW, MVC, PLHIV, and key populations to know their status, improve retention and adherence, and achieve viral suppression and preventing new infections and promoting overall well-being)? iii. To what extent has JSI contributed on creating and supporting more functional, better coordinated community structures and systems to better serve priority and key populations? iv. What have been the strengths and weaknesses of USAID in, and oversight of CHSSP? v. To what extent will the JSI’s funded activities be sustained after ending of the program? III. EVALUATION DESIGN AND METHODOLOGY A. Evaluation Design The principal source for the end of program evaluation study is the reviewing activity progress reports, meeting with stakeholders, and both direct and indirect beneficiaries. The focus will be to collect, analyze, and use information to answer questions as to whether the program’s objectives have been attained with effectiveness and efficiency. The evaluator will provide information on progress on implementation intended activities and outcomes. He will analyze the achievements or non-achievement, justifications, and provide recommendations for the future planned award. B. Data Collection and Analysis Methods Evaluation Questions Suggested Data Source Data Collection Method(s) Sampling or Selection Criteria Data Analysis Method(s) 1) To what extent has JSI contributed on improving environment for providing community health and social welfare services (for comprehensive, sustainable and quality HIV and other services? LGAs, IP, and Partners KII, FGD . Qualitative and quantitative 2) To what extent has JSI contributed on supporting higher performing human resources for community health and social welfare services (who are able to support AGYW, MVC, PLHIV, and key populations to know their status, improve retention and adherence, and achieve viral suppression and preventing new infections and promoting overall well￾being)? LGAs, IP, and Partners KII, FGD . Qualitative and quantitative 3) To what extent has JSI contributed on creating and supporting more functional, better coordinated community structures and systems to better serve priority and key populations? LGAs, IP, and Partners KII, FGD . Qualitative and quantitative 35 Evaluation Questions Suggested Data Source Data Collection Method(s) Sampling or Selection Criteria Data Analysis Method(s) 4) What have been the strengths and weaknesses of USAID’s in, and oversight of, CHSSP? LGAs, IP, and Partners KII, FGD . Qualitative and quantitative 5) To what extent will the JSI’s funded activities be sustained after ending of the program? IP, Partners and Private organizations KII, FGD . Qualitative and quantitative C. METHODOLOGICAL STRENGTHS AND LIMITATIONS i. Language barrier. ii. The changing of the program description over time. IV. EVALUATION PRODUCTS A. Deliverables i. Evaluation work plan/schedule indicating which step/activity should occur and when to ensure that the evaluation team fully understands the assignment approach. ii. Evaluation design and methodology clearly articulating how the evaluation questions will be answered during the pre-evaluation meeting. iii. A debrief meeting with a PowerPoint Presentation of the key findings, issues, and recommendations. iv. The draft report. The written report should clearly describe findings, conclusions, and recommendations (using the report format provided in “Reporting Requirements” below). v. A final report that incorporates the team’s responses to Mission comments and suggestions. vi. Final report format – Executive summary, Methodology, Findings, Analysis of the findings of each research question. Thirty (30) page limit. B. Reporting and Submission Guidelines Criteria to ensure quality of the Evaluation Report – Per Automated Directives System (ADS) 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.23 • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, program, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. 23 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 36 • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology should be explained in detail and sources of information properly identified and will be included as an annex in the final report. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific, with defined responsibility for the action. V. TEAM COMPOSITION, ROLES AND RESPONSIBILITIES OF TEAM MEMBERS The Mission proposes a team composition of Team Leader/Program Development and has experience working on community systems strengthening. All team members should have the following qualifications: • Master’s degree or higher level of education in a relevant technical area, experience working in community systems or health systems is desirable. • Knowledge, skills, and experience with USAID contracting and reporting requirements; policies and initiatives; and tools, such as performance monitoring plans (PMPs) and results frameworks. • Advanced written and oral communications skills in English. Team Leader/Program Development: Should have a minimum of ten years of experience in public health, with technical knowledge and experience with preferably community systems strengthening. The team leader will be responsible for 1) managing the team’s activities, 2) ensuring that all deliverables are met in a timely manner, 3) serving as a liaison between the Mission and the evaluation team, and 4) leading briefings and presentations. In addition, the team leader must have the following qualifications: • Excellent skills in planning, facilitation, and consensus building; • Demonstrated experience leading an evaluation team; • Excellent interpersonal skills; • Excellent skills in program management; and • Excellent organizational skills and ability to keep to a timeline. Monitoring and Evaluation Expert: One person, preferably locally-based, with relevant professional qualifications and skills and at least five years of experience in USAID M&E procedures, program, and organizational management. This person should also have strong knowledge, skills, and experience in qualitative and quantitative evaluation tools, as well as skills in community program designing and monitoring. The Mission M&E specialist can participate in the evaluation exercise. Gender Expert: This person should have a postgraduate degree in gender. S/he should have at least five years of experience in gender programming, i.e., analysis, planning, mainstreaming, and evaluating. Health Systems Expert: S/he should be knowledgeable in health systems and both formal and informal structures, program assessment and evaluation methodologies of community-based systems and structures, work plan development, and training in information monitoring systems. S/he should have 37 extensive experience, and demonstrate state-of-the-art knowledge, in conducting OVC programmatic designing and evaluations/assessments. VI. EVALUATION MANAGEMENT A. Logistics Outline specific kinds of support USAID and the implementing organization(s) will provide. Specify any additional logistical roles or responsibilities it expects the evaluation team to fulfill or to provide (e.g., interpreters, translators, surveyors). Clarify any specialized management arrangements, such as in the case of joint evaluations. Implementing partner will only provide basic support required including organizing appointments for planned visits. The regional based teams expect to provide logistical support and visit arrangements. B. Tasks and Scheduling Dates and Timeline – Report required (October 2018). 38 ANNEX II: BIBLIOGRAPHY • National Plan of Action to end violence against women and children in Tanzania (2017/18 - 2021/22). • Resource Mobilization module (Facilitator guide) by the JSI; New Partners Initiative Technical Assistance (NuPITA Program), 2010. • Organization Capacity Assessment Tool for Organization funded by USAID; Participant’s copy by JSI, 2014. • Capacity-building and Strengthening framework Version 02, FY 2012 by the President’s Emergency Plan for AIDS Relief (PEPFAR). • Resource Mobilization training report by JSI, written by Dr. Misanya Bingi of the University of Dar es Salaam, 2017. • JSI/CHSSP Annual Program Year 1 report (September 2015). • JSI/CHSSP Annual Program Year 1 report (September 2016). • JSI/CHSSP Annual Program Year 1 report (September 2017). • JSI/CHSSP Annual Work plan for the period of February 2015 to September 2015. • JSI/CHSSP Annual Work plan for the period of October 2015 to September 2016. • JSI/CHSSP Annual Work plan for the period of February 2016 to September 2017. • JSI/CHSSP Annual Work plan for the period of February 2017 to September 2018. • JSI Activity Description Number 1. • JSI Activity Description Number 2. • JSI Activity Description Number 3. • JSI/CHSSP Monitoring and Evaluation Plan (April 2016). • Tanzania Community Health and Social Welfare Systems: Baseline Landscape Assessment, 2015. 39 ANNEX III: KII AND FGD DISCUSSION GUIDES FGD GUIDE FOR COMMUNITY CASE WORKERS Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Number of participants: . Designations24 of the participants (Enter each participant’s designation separated by a comma): . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and purpose of the study Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague__________________. We work with the Data for Development program, a USAID-funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation for USAID to understand what has worked well, what did not work as well, and what has been sustained from the USAID-funded Kizazi Kipya Program implemented by PACT and the Community Health and Social Welfare Systems Strengthening Program, or CHSSP, implemented by John Snow that together aim to improve the well-being of people living with HIV, adolescent girls and young women, and most vulnerable children and their families. Description of study procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the two programs. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. This report will be presented to USAID for decision-making. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of participating in this study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. Right to refuse or withdraw 24 This is the role or job title of each individual participating. Do not document participant names in this table. 40 The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. Right to ask questions and report concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Gerald Usika through [gusika@engl.com] or by phone at [+255 756180413]. This evaluation has been reviewed and approved by National Institute for Medical Research ethical committee. If you have any questions concerning the study, please contact NIMR at +255222121400. Consent Do you agree to participate in this study? Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Let’s begin by talking about the Case Management training you received from CHSSP? 1. What topics did the training cover? 2. How useful was the training? In answering this question, please consider such things as the knowledge/expertise of the instructors, the quality of the instruction, the quality and usefulness of training resources, and the relevance of the topics covered to your job as a community case worker. 3. What were the most important things you learned or skills you acquired from the training as they apply to your job as a community case worker? 4. To what extent has the training you received contributed to improving your ability to provide community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children? In answering this question, please describe how you have applied what you learned from the training in your job as a community case worker? Please cite specific examples. 5. What challenges do you face in applying what you learned in the training to your job as a community case worker? How did CHSSP help you address these challenges? What more could it do? (Note to interviewer: Among other things, probe to find out about the type and frequency of follow-up support received since the Case Management training.) 6. How likely is it that you will be able to apply what you learned from the training to your job as a community case worker over the long run? Why? How can this likelihood be improved? 7. What were the strengths of weaknesses of the case management training you received from CHSSP? How could this training be improved? Now, let’s talk about your work as a community case worker as supported by the Kizazi Kipya program. 1. Please describe your work as a community case worker. What does a CCW do? 2. What support do you receive in your work as a community case worker from Kizazi Kipya? 41 3. How useful has this support been? In particular, how has it affected your capacity and performance as a community case worker? 4. What are the greatest challenges you face in your work as a community case worker? To what extent does Kizazi Kipya support address these challenges? What more could it do? 5. What are the strengths of weaknesses of the support you receive from Kizazi Kipya? How could this support be improved? 6. Can you describe the services you help people access? Please describe the services provided by Kizazi Kipya and the services provided by other organizations or government structures that you link people to. 7. Are the people you assist in your work as a community case worker accessing and using the services to which you referred them? What factors help explain why they are or are not accessing and using these services? 8. For those people who do access and use the services to which you referred them, what is the quality of care or services they are receiving? What factors explain the quality of care or services they are receiving? 9. Are there sufficient services available in the community for you to be able to link vulnerable households to meet their needs? What are the gaps? 10. Overall, how have the CHSSP and Kizazi Kipya programs contributed to improving the local system for providing health, social welfare, and HIV prevention and treatment services to key populations, both with regards to the case management system that you are involved in and in other ways you may be aware of? 11. How long do you plan to continue working as a community case worker? What factors make you want to work for a longer period? For a shorter period? What could CHSSP, Kizazi Kipya, or someone else do to make you want to keep working as a community case worker over the long term? -----------------Thank you for your time-------------------- 42 FGD GUIDE FOR MULTI-SECTORAL AIDS COMMITTEES AT WARDS AND VILLAGE LEVELS (MACS) Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Name of the committee: . Number of participants: . Designations25 of the participants (Enter each participant’s designation separated by a comma): . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the Community Health and Social Welfare Systems Strengthening, or CHSSP, program implemented by John Snow Inc. We are here to find out whether the program is bridging the gap between facility and community-based services to ensure comprehensive service delivery to beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the CHSSP program. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. This report will be presented to USAID for decision-making. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of Participating in this Study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. 25 This is the role or job title of each individual participating. Do not document participant names in this table. 43 Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. Right to Ask Questions and Report Concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Nasson Konga through [nkonga@engl.com] or by phone at [+255 767 201618]. Consent Do you agree to participate in this study? 1. Yes 2. No Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Protocol Questions 1. What role does your Multisector Aids Committee play in providing community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children? 2. What support has your committee received from the CHSSP program in helping you to fulfill or improve this role? In answering this question, please consider such things as capacity assessments, capacity development, training, awareness raising, provision of guidelines or manuals, and so forth. 3. How did this CHSSP support address issues related to women, girls, and youth, including, for example, factors limiting their access to community health and social welfare services and/or the quality of community health and social welfare services they receive? How has it affected what you do? 4. How useful has this CHSSP support been to your committee? In answering this question, please describe how CHSSP support has affected: 1) the capacity of your committee and its members; 2) how your committee addresses issues involving women, girls, and youth; 3) the types of activities your committee implements; and 4) your committee’s role or performance in providing community health, social welfare, and HIV prevention and treatment services to key populations. Please cite specific examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above four items related to the usefulness of CHSSP support.) 5. Talking again about the effects that CHSSP support has had on your committee, to what extent do you think these are likely to continue over the long term? Why? 6. What are the greatest challenges your committee faces in providing community health, social welfare, and HIV prevention and treatment services to key populations? To what extent did CHSSP support address these challenges? What more could it do? (Note to interviewer: Respondent is likely to mention challenges related to limited funding. Probe to steer the discussion away from this topic and to investigate other challenges faced.) 44 7. Overall, how has the CHSSP program contributed to improving the local system for providing health, social welfare, and HIV prevention and treatment services to key populations? In answering this question, please consider such things as: 1) government policies, regulations, and guidelines related to local health and social welfare services; 2) access to health and social welfare services; 3) capacity of health and social welfare service providers; 4) coordination among health and social welfare service providers; 5) quality of health and social welfare services provided; and 6) overall functioning of the local health and social welfare service system. Please cite examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above six items related to CHSSP support to the respondent’s committee and CHSSP support to the sector more generally, including its support for the Integrated Case Management System and CCW training and its support for other local social welfare and health system actors, such as LGAs, MVCCs, PLHIV clusters, para-social workers, CHWs, social welfare officers, CSOs, etc.) 8. Talking again about the effects that the CHSSP program has had on the local system for providing health, social welfare, and HIV prevention and treatment services to key populations, to what extent do you think these are likely to continue over the long term? Which factors affect their likely sustainability, either positively or negatively and how? What is CHSSP doing to address these factors? 9. What have been the strengths of weaknesses of the support that CHSSP has provided both to your committee and to the overall system for system for providing health, social welfare, and HIV prevention and treatment services to key populations? How could this support be improved? What additional support is needed from CHSSP or programs like it? (Note to interviewer: Respondent is likely to mention the need for greater financial support. Probe to steer the discussion away from this topic and to investigate other areas for improvement and additional support needed.) -----------------Thank you for your time-------------------- 45 FGD GUIDE FOR COMMUNITY HEALTH WORKERS (CHWS) Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Number of participants: . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the Community Health and Social Welfare Systems Strengthening Program, or CHSSP, Program implemented by John Snow. We are here to find out whether the program is bridging the gap between facility and community-based services to ensure comprehensive service delivery to beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the Case Management training you received from CHSSP. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. This report will be presented to USAID for decision￾making. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of Participating in this Study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. Right to Ask Questions and Report Concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Nasson Konga through [nkonga@engl.com] or by phone 46 at [+255 767 201618]. Consent Do you agree to participate in this study? 1. Yes 2. No Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Protocol Questions 1. Please describe the training you received from the CHSSP program. What topics did the training cover? 2. How useful was the training? In answering this question, please consider such things as the knowledge/expertise of the instructors, the quality of the instruction, the quality and usefulness of training resources, and the relevance of the topics covered to your job as a community health worker. 3. What were the most important things you learned or skills you acquired from the training as they apply to your job as a community health worker? 4. To what extent has the training you received contributed to improving your ability to provide community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children? In answering this question, please describe how you have applied what you learned from the training in your job as a community health worker? Please cite specific examples. 5. What challenges do you face in applying what you learned in the training to your job as a community health worker? How did CHSSP help you address these challenges? What more could it do? (Note to interviewer: Among other things, probe to find out about the type and frequency of follow-up support received since the Case Management training.) 6. How likely is it that you will be able to apply what you learned from the training to your job as a community health worker over the long run? Why? How can this likelihood be improved? 7. What were the strengths of weaknesses of the case management training you received from CHSSP? How could this training be improved? Now, let’s talk about your work as a community health worker as supported by the Kizazi Kipya program. 1. What support do you receive in your work as a community health worker from Kizazi Kipya? 2. How useful has this support been? In particular, how has it affected your capacity and performance as a community health worker? 3. What are the greatest challenges you face in your work as a community health worker? To what extent does Kizazi Kipya support address these challenges? What more could it do? 4. What are the strengths of weaknesses of the support you receive from Kizazi Kipya? How could this support be improved? 47 5. Are the people you assist in your work as a community health worker accessing and using the services to which you referred them? What factors help explain why they are or are accessing and using these services? 6. For those people who do access and use the services to which you referred them, what is the quality of care they are receiving? What factors explain the quality of care they are receiving? 7. Overall, how have the CHSSP and Kizazi Kipya programs contributed to improving the local system for providing health, social welfare, and HIV prevention and treatment services to key populations, both with regards to the case management system that you are involved in and in other ways you may be aware of? 8. How long do you plan to continue working as a community health worker? What factors make you want to work for a longer period? For a shorter period? What could CHSSP, Kizazi Kipya, or someone else do to make you want to keep working as a community health worker over the long term? -----------------Thank you for your time-------------------- 48 KII GUIDE FOR CSOS AND SUBGRANTEES Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Name of the CSO/Sub grantee: . Number of participants: . Designations26 of the participants (Enter each participant’s designation separated by a comma): . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the Community Health and Social Welfare Systems Strengthening, or CHSSP, program implemented by John Snow Inc. We are here to find out whether the program is bridging the gap between facility and community-based services to ensure comprehensive service delivery to beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the CHSSP program. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of Participating in this Study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. 26 This is the role or job title of each individual participating. Do not document participant names in this table. 49 Right to Ask Questions and Report Concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Nasson Konga through [nkonga@engl.com] or by phone at [+255 767 201618]. Consent Do you agree to participate in this study? 1. Yes 2. No Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Protocol Questions 1. What role does your organization play in providing community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children? 2. What support has your organization received from the CHSSP program in helping you to fulfill or improve this role? In answering this question, please consider such things as capacity assessments, capacity development, training, awareness raising, provision of guidelines or manuals, and so forth. 3. How did this CHSSP support address issues related to women, girls, and youth, including, for example, factors limiting their access to community health and social welfare services and/or the quality of community health and social welfare services they receive? How has it affected what you do? 4. How useful has this CHSSP support been to your organization? In answering this question, please describe how CHSSP support has affected: 1) your organizational capacity in areas such as management, finance, human resources, monitoring, and evaluation; 2) how your organization addresses issues involving women, girls, and youth; 3) the types of activities your organization implements; and 4) your organization’s role or performance in providing community health, social welfare, and HIV prevention and treatment services to key populations. Please cite specific examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above four items related to the usefulness of CHSSP support.) 5. Talking again about the effects that CHSSP support has had on your organization, to what extent do you think these are likely to continue over the long term? Why? 6. How well have civil society organization like yours been integrated into the local system for providing health, social welfare, and HIV prevention and treatment services to key populations? Why? How has the CHSSP program helped? What additional support is needed from CHSSP or programs like it? 7. What are the greatest challenges your organization faces in providing community health, social welfare, and HIV prevention and treatment services to key populations? To what extent did CHSSP support address these challenges? What more could it do? (Note to interviewer: Respondent is likely to mention challenges related to limited funding. Probe to steer the discussion away from this topic and to investigate other challenges faced.) 8. Overall, how has the CHSSP program contributed to improving the local system for providing health, social welfare, and HIV prevention and treatment services to key populations? In answering this 50 question, please consider such things as: 1) government policies, regulations, and guidelines related to local health and social welfare services; 2) access to health and social welfare services; 3) capacity of health and social welfare service providers; 4) coordination among health and social welfare service providers; 5) quality of health and social welfare services provided; and 6) overall functioning of the local health and social welfare service system. Please cite examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above six items related to CHSSP support to the respondent’s organization and CHSSP support to the sector more generally, including its support for the Integrated Case Management System and CCW training and its support for other local social welfare and health system actors, such as MACs, MVCCs, PLHIV clusters, para-social workers, CHWs, LGAs, social welfare officers, CSOs, etc.) 9. Talking again about the effects that the CHSSP program has had on the local system for providing health, social welfare, and HIV prevention and treatment services to key populations, to what extent do you think these are likely to continue over the long term? Which factors affect their likely sustainability, either positively or negatively, and how? What is CHSSP doing to address these factors? 10. What have been the strengths of weaknesses of the support that CHSSP has provided both to your organization and to the overall system for system for providing health, social welfare, and HIV prevention and treatment services to key populations? How could this support be improved? What additional support is needed from CHSSP or programs like it? (Note to interviewer: Respondent is likely to mention the need for greater financial support. Probe to steer the discussion away from this topic and to investigate other areas for improvement and additional support needed.) -----------------Thank you for your time-------------------- 51 KII PROTOCOL FOR IPS AND OTHER PARTNERS. Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Implementing Partner Name: . Number of participants: . Designations27 of the participants (Enter each participant’s designation separated by a comma): . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the Community Health and Social Welfare Systems Strengthening program, or CHSSP, implemented by John Snow. We are here to find out whether the program is bridging the gap between facility and community-based services to ensure comprehensive service delivery to beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the CHSSP program. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. This report will be presented to USAID for decision-making. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of Participating in this Study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. 27 This is the role or job title of each individual participating. Do not document participant names in this table. 52 Right to Ask Questions and Report Concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Nasson Konga through [nkonga@engl.com] or by phone at [+255 767 201618]. Consent Do you agree to participate in this study? 1. Yes 2. No Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Protocol Questions 1. What is your role within the CHSSP program? Specifically, what activities do you implement to support the provision of community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children as part of the CHSSP program? 2. What are your relationships with other CHSSP implementing partners and how well are these relationships managed in terms of, for example, communication, collaboration, division of responsibilities, and so forth? 3. What issues or challenges have arisen in the implementation of CHSSP, both internal and external to the program? How have they been addressed and with what result? 4. How is CHSSP addressing issues related to women, girls, and youth, including, for example, factors limiting the access of women, girls, and youth to community health and social welfare services and/or the quality of community health and social welfare services they receive? What could CHSSP do to better address these issues? 5. What are the greatest challenges in providing community health, social welfare, and HIV prevention and treatment services to key populations? To what extent did CHSSP support address these challenges? What more could it do? (Note to interviewer: Respondent is likely to mention challenges related to limited funding. Probe to steer the discussion away from this topic and to investigate other challenges faced.) 6. How has CHSSP contributed to improving the local system for providing health, social welfare, and HIV prevention and treatment services to key populations? In answering this question, please consider such things as: 1) government policies, regulations, and guidelines related to local health and social welfare services; 2) access to health and social welfare services; 3) capacity of health and social welfare service providers; 4) coordination among health and social welfare service providers; 5) quality of health and social welfare services provided; and 6) overall functioning of the local health and social welfare service system. Please cite examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above six items related to the Integrated Case Management System and CCW training and CHSSP support for other local social welfare and health system actors, such as MOHCDGEC, DSW, PO-RALG, MACs, MVCCs, PLHIV clusters, para-social workers, CHWs, LGAs, social welfare officers, CSOs, etc.) 7. Talking again about the effects that the CHSSP program has had on the local system for providing health, social welfare, and HIV prevention and treatment services to key populations, to what extent 53 do you think these are likely to continue over the long term? Which factors affect their likely sustainability, either positively or negatively, and how? What is CHSSP doing to address these factors? 8. What have been the strengths of weaknesses of the support that CHSSP has provided to improve the system for providing health, social welfare, and HIV prevention and treatment services to key populations? How could this support be improved? What additional support is needed from CHSSP or programs like it? (Note to interviewer: Respondent is likely to mention the need for greater financial support. Probe to steer the discussion away from this topic and to investigate other areas for improvement and additional support needed.) -----------------Thank you for your time-------------------- 54 KII GUIDE FOR LGA LEADERS (CHMT & CMAC) Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Name of the LGA: . Number of participants: . Designations28 of the participants (Enter each participant’s designation separated by a comma): . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the Community Health and Social Welfare Systems Strengthening, or CHSSP, program implemented by John Snow Inc. We are here to find out whether the program is bridging the gap between facility and community-based services to ensure comprehensive service delivery to beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the CHSSP program. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. This report will be presented to USAID for decision-making. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of Participating in this Study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. 28 This is the role or job title of each individual participating. Do not document participant names in this table. 55 Right to Ask Questions and Report Concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Nasson Konga through [nkonga@engl.com] or by phone at [+255 767 201618]. Consent Do you agree to participate in this study? 1. Yes 2. No Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Protocol Questions 1. What role does your local government play in providing community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children? 2. What support has your local government received from the CHSSP program in helping you to fulfill or improve this role? In answering this question, please consider such things as capacity assessments, capacity development, training, awareness raising, provision of guidelines or manuals, and so forth. 3. How did this CHSSP support address issues related to women, girls, and youth, including, for example, factors limiting their access to community health and social welfare services and/or the quality of community health and social welfare services they receive? 4. How useful has this CHSSP support been to your local government? In answering this question, please describe how CHSSP support has affected: 1) your organizational and staff capacity; 2) how your local government addresses issues involving women, girls, and youth; 3) the types of activities your local government implements; and 4) your local government’s role or performance in providing community health, social welfare, and HIV prevention and treatment services to key populations. Please cite specific examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above four items related to the usefulness of CHSSP support.) 5. Talking again about the effects that CHSSP support has had on your local government, to what extent do you think these are likely to continue over the long term? Why? 6. What are the greatest challenges your local government faces in providing community health, social welfare, and HIV prevention and treatment services to key populations? To what extent did CHSSP support address these challenges? What more could it do? (Note to interviewer: Respondent is likely to mention challenges related to limited funding. Probe to steer the discussion away from this topic and to investigate other challenges faced.) 7. Overall, how has the CHSSP program contributed to improving the local system for providing health, social welfare, and HIV prevention and treatment services to key populations? In answering this question, please consider such things as: 1) government policies, regulations, and guidelines related to local health and social welfare services; 2) access to health and social welfare services; 3) capacity of health and social welfare service providers; 4) coordination among health and social welfare service providers; 5) quality of health and social welfare services provided; and 6) overall functioning of the local health and social welfare service system. Please cite examples. (Note to interviewer: Probe for the 56 respondent’s perceptions on each of the above six items related to CHSSP support to the respondent’s organization and CHSSP support to the sector more generally, including its support for the Integrated Case Management System and CCW training and its support for other local social welfare and health system actors, such as MACs, MVCCs, PLHIV clusters, para-social workers, CHWs, social welfare officers, CSOs, etc.) 8. Talking again about the effects that the CHSSP program has had on the local system for providing health, social welfare, and HIV prevention and treatment services to key populations, to what extent do you think these are likely to continue over the long term? Which factors affect their likely sustainability, either positively or negatively, and how? What is CHSSP doing to address these factors? 9. What have been the strengths of weaknesses of the support that CHSSP has provided both to your local government and to the overall system for system for providing health, social welfare, and HIV prevention and treatment services to key populations? How could this support be improved? What additional support is needed from CHSSP or programs like it? (Note to interviewer: Respondent is likely to mention the need for greater financial support. Probe to steer the discussion away from this topic and to investigate other areas for improvement and additional support needed.) -----------------Thank you for your time-------------------- 57 KII GUIDE FOR NATIONAL & REGIONAL GOT STAFF Introduction and Consent Date (dd/mm/yyyy): . Interviewer’s name: . Name of the Institution: . Number of participants: . Designations29 of the participants (Enter each participant’s designation separated by a comma): . Start time (24 h clock, hh:mm): . End time (24 h clock, hh:mm): . To start the interview, please read the following script: Introduction and Purpose of the Study Hello. My name is_______________. This is my team [introduce note-taker(s) and translator]. We are here on behalf of USAID, which is funding the Community Health and Social Welfare Systems Strengthening, or CHSSP, program implemented by John Snow Inc. We are here to find out whether the program is bridging the gap between facility and community-based services to ensure comprehensive service delivery to beneficiaries. Description of Study Procedures If you agree to be interviewed, you will be asked to share your experiences with and views about the CHSSP program. During our conversation we would like to take notes and record the conversation. The recording and the notes are to make sure that the evaluators have heard and understood what you have shared with us today. Your ideas, together with other data, will be used to produce a report and not otherwise. This report will be presented to USAID for decision-making. Your name will not be included in the report, but the ideas you share might be. Our discussion today will take 45-60 minutes. Confidentiality The recording and the notes are strictly confidential. Only the evaluation team will listen to this recording or read the notes. The recordings and the notes will be kept in a secure location and all electronic information will be coded and secured. The recordings and notes will be destroyed after the program is completed. Your privacy will be protected; we will not include any information in any report that would make it possible to identify you. Please note that we cannot guarantee full confidentiality because of the group setting, as we cannot ensure that participants will not disclose any information shared during the group interview. Once again, we ask that what we discuss today remains here with us. Benefits of Participating in this Study There is no direct benefit to you being in this study, but we hope that the results of our study will help improve the program implementation and the important services it provides. You will receive no compensation for participating in this interview. Right to Refuse or Withdraw The decision to participate in this study is entirely up to you. You may refuse to take part in the study at any point in time, and you have the right not to answer any single question, as well as to withdraw from the discussion at any point. You will not be penalized if you choose to not answer or withdraw from this discussion. 29 This is the role or job title of each individual participating. Do not document participant names in this table. 58 Right to Ask Questions and Report Concerns You have the right to ask questions about this evaluation and to have those questions answered by me before, during, or after the interview. Do you have questions for me at this time? If you have any further questions about the study, feel free to contact Nasson Konga through [nkonga@engl.com] or by phone at [+255 767 201618]. Consent Do you agree to participate in this study? 1. Yes 2. No Do I have your permission turn on the tape recorder and begin the interview? 1. Yes 2. No Protocol Questions 1. What role does your organization play in providing community health, social welfare, and HIV prevention and treatment services to key populations, particularly people living with HIV, adolescent girls and young women, and most vulnerable children? 2. What support has your organization received from the CHSSP program in helping you to fulfill or improve this role? In answering this question, please consider such things as capacity assessments, capacity development, training, awareness raising, provision of guidelines or manuals, and so forth. 3. How did this CHSSP support address issues related to women, girls, and youth, including, for example, factors limiting their access to community health and social welfare services and/or the quality of community health and social welfare services they receive? How has it affected what you do? 4. How useful has this CHSSP support been to your organization? In answering this question, please describe how CHSSP support has affected: 1) your organizational capacity in areas such as management, finance, human resources, monitoring, and evaluation; 2) how your organization addresses issues involving women, girls, and youth; 3) the types of activities your organization implements; and 4) your organization’s role or performance in providing community health, social welfare, and HIV prevention and treatment services to key populations. Please cite specific examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above four items related to the usefulness of CHSSP support.) 5. Talking again about the effects that CHSSP support has had on your organization, to what extent do you think these are likely to continue over the long term? Why? 6. What are the greatest challenges your organization faces in providing community health, social welfare, and HIV prevention and treatment services to key populations? To what extent did CHSSP support address these challenges? What more could it do? (Note to interviewer: Respondent is likely to mention challenges related to limited funding. Probe to steer the discussion away from this topic and to investigate other challenges faced.) 7. Overall, how has the CHSSP program contributed to improving the system for providing health, social welfare, and HIV prevention and treatment services to key populations? In answering this question, please consider such things as: 1) government policies, regulations, and guidelines related to local health and social welfare services; 2) access to health and social welfare services; 3) capacity of health and social welfare service providers; 4) coordination among health and social welfare service providers; 5) quality of health and social welfare services provided; and 6) overall functioning of the 59 local health and social welfare service system. Please cite examples. (Note to interviewer: Probe for the respondent’s perceptions on each of the above six items related to CHSSP support to the respondent’s organization and CHSSP support to the sector more generally, including its support for the integrated case management system and CCW training and its support for other local social welfare and health system actors, such as MACs, MVCCs, PLHIV clusters, para-social workers, CHWs, LGAs, social welfare officers, CSOs, etc.) 8. Talking again about the effects that the CHSSP program has had on the system for providing health, social welfare, and HIV prevention and treatment services to key populations, to what extent do you think these are likely to continue over the long term? Which factors affect their likely sustainability, either positively or negatively, and how? What is CHSSP doing to address these factors? 9. What have been the strengths of weaknesses of the support that CHSSP has provided both to your organization and to the overall system for system for providing health, social welfare, and HIV prevention and treatment services to key populations? How could this support be improved? What additional support is needed from CHSSP or programs like it? (Note to interviewer: Respondent is likely to mention the need for greater financial support. Probe to steer the discussion away from this topic and to investigate other areas for improvement and additional support needed.) -----------------Thank you for your time-------------------- 60 ANNEX IV: KIIS, FGD, AND CCW SURVEY SAMPLING METHODOLOGY KII and FGD Sampling Methodology To sample national stakeholders, six locations were selected. These were Dar es Salaam, Dodoma, Temeke MC in Dar es Salaam, Mbeya CC in Mbeya, Njombe TC in Njombe, Nyamagana MC in Mwanza, and Bukoba DC in Kagera. Sites were selected purposively with the following in mind: • The ET selected similar locations to where Kizazi Kipya activities took place and considered the number of supported CSOs in the region. Discussions with the Mission and JSI were conducted to verify how active JSI interventions were in these sites. • The selected districts provided a balance between urban and rural locations and a balance between districts with CSOs providing case management and those providing both case management and HIV treatment. Both KIIs and FGDs at regional and district levels were conducted in Swahili by Swahili-speaking facilitators at locations accessible to participants. In national-level KIIs, where the KII participants spoke fluent English, the ET conducted the interview in English. All KIIs and FGDs at the regional and district levels were recorded (with the prior, informed consent by participants) and were transcribed and translated by Utafiti, Data for Development’s local research partner. The ET worked closely with JSI, USAID/Tanzania, and (as appropriate) Pact to select specific KII participants from the different stakeholder groups. Selection was made using purposive, or non￾probability, sampling methods. The main goal of the purposive sampling was to focus on particular characteristics of the target population that were of interest, which would best enable the ET to answer the EQs. The purposive sampling method was appropriate in this case because, due to the specialized knowledge they possessed, there was a limited number of people who could serve as primary data sources. The ET sought to select 10-15 FGD participants from districts and wards that provided a balance across the wards where CHSSP was active. This included both focus wards (where all activities and follow-up training were provided) and non-focus wards (where training and quarterly follow-up were provided). CCW Survey Sampling Methodology The CCW survey sampling frame consisted of the list of active CCWs trained in case management by either Kizazi Kipya or CHSSP and who were currently implementing the integrated case management model under Kizazi Kipya and for whom the program(s) had a working telephone/mobile phone number. From this list, the ET selected a random sample of 269 CCWs, plus a randomly selected set of 269 replacements for non-responses across all CHSSP/Kizazi Kipya regions and councils. After three unsuccessful attempts to reach a selected CCW, survey enumerators selected the next name on the randomly selected replacement list and continued this process until 269 surveys were complete. Sample size calculations indicated that to achieve a 90 percent confidence level with a 5 percent margin of error, a target sample size of 269 CCWs was required—given the 24,692 CCWs in the total CCW population. Enumerators captured survey responses on a tablet using a Survey Monkey form, which helped facilitate more accurate data capture and allowed survey results to be uploaded immediately to a web-based data repository. The survey was conducted telephonically by eight enumerators with prior experience in survey data collection. The enumerators were trained in data collection methods, ethics, and in how to use the specific survey instrument. Their training included a practicum supervised by Data for Development staff. Quality control was provided through a field control form used by the subcontractor’s quality control staff. The form captured the status of each call and verified the rationale for selecting replacements. Data for Development staff monitored the data on the back end and recommended cleaning and corrections 61 to the subcontractor’s team. The completed responses in the survey dataset were further cleaned and checked for quality and consistency before the ET analyzed them. The 269 CCWs who responded to the survey came from 22 districts. Of these 148 (55 percent) were females and 121 (45 percent) were males.30 In terms of the highest level of education completed, approximately one-third of the survey respondents had completed primary school and approximately one￾fifth had completed secondary school, some secondary school, or some primary school. The majority of CCWs completed primary school, while another 22 percent completed a secondary school education. Less than 7 percent of CCWs have a diploma or an advanced degree. A total of 57 percent of CCWs said that they had been trained by Pact/Kizazi Kipya (in technical skills areas) while another 45 percent said they had been trained by JSI/ CHSSP (in comprehensive case management). 30 Regions covered by the CCW survey (with the number of respondents in parentheses) include: Arusha (4), Dar es Salaam (50), Dodoma (1), Geita (15), Iringa (12), Kagera (13), Katavi (1), Kigoma (2), Kilimanjaro (11), Mara (4), Mbeya (42), Mjini Magharibi (1), Morogoro (10), Mwanza (18), Njombe (4), Pwani (13), Rukwa (2), Ruvuma (19), Shinyanga (10), Singida (5), Tabora (24), and Tanga (8). 62 ANNEX V: CCW SURVEY QUESTIONNAIRE PRE-INTERVIEW FIELD CONTROL [ENUMERATOR, COMPLETE THIS SECTION BEFORE THE INTERVIEW] ENUMERATOR ENUMERATOR: select your name from the list below. [PROG: Program list of enumerators] CCWID [PROG: MIN OF CCW1 MAX OF CCW534] Enter the CCW ID. REGION ENUMERATOR: ENTER the region that the CCW is coming from. [PROG: Program drop-down with regions] COUNCIL ENUMERATOR: ENTER the Council that the CCW is coming from. [PROG: Program drop-down with regions] WARD ENUMERATOR: ENTER the WARD that the CCW is coming from [PROG: Program drop-down with regions] VILLAGE ENUMERATOR: ENTER the VILLAGE that the CCW is coming from. [PROG: Program drop-down with regions] INTRODUCTION AND CONSENT [ENUMERATOR, TO START THE INTERVIEW, PLEASE READ THE FOLLOWING SCRIPT] Hello and thank you for agreeing to speak with us. My name is_______________ (interview name). We work with the Data for Development program, a USAID-funded program that helps improve the use of data for decision-making in Tanzania. We are evaluating the Kizazi Kipya and Community Health and Social Welfare Systems Strengthening, or CHSSP, programs that were implemented by Pact and JSI. USAID would like to understand what has worked well in these programs, and what did not work as well. In the questions I’m about to ask you, we will be mainly focusing on your work as community case worker and the case management training you received under CHSSP. Your answers will not be presented with your name and will be used for reporting. This report will be presented to USAID for decision-making. Our interview today will take not more than 45 minutes. The decision to participate in this study is entirely voluntary and you have the right not to answer any single question, as well as to skip one at any point. If you have any further questions about the survey feel free to contact Gerald Usika through [gusika@engl.com] or by phone at [+255 756180413]. CONSENT Do you agree to participate in this survey? PROG: COLLECT TIMESTAMP AFTER THIS QUESTION IS ANSWERED. 1 YES 63 0 NO PROG: IF ANSWER IS NO, THEN GO TO THE END; IF ANSWER IS YES, CONTINUE WITH QUESTIONNAIRE. DEMOGRAPHICS [PROG: COLLECT TIMESTAMP] AGE [PROG: DON’T KNOW=998, REFUSE=999] [PROG: MIN=1, MAX=120] How old are you [on your last birthday]? [Numeric Response] SEX [ENUMERATOR, PLEASE OBSERVE AND RECORD THE GENDER OF THE RESPONDENT – DO NOT ASK] 1. Male 2. Female CCW TITLE [PROG: REGULAR CCW, LEAD CCW, DON’T KNOW=998, REFUSE=999] What level of formal education have you completed? i. No education ii. Some primary iii. Completed primary iv. Some secondary v. Completed secondary vi. Vocational training vii. University How old are you [on your last birthday]? [Numeric Response] MAIN QUESTIONAIRE 1. Did you participate in any training related to Case Management? PROG: IF ANSWER IS NO, THEN GO TO THE END; IF ANSWER IS YES, CONTINUE WITH QUESTIONNAIRE. i. YES ii. NO iii. Don’t know 2.What were the qualifications for you to be selected as a community case worker? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Knows how to read and write ii. Level of education iii. Previous experience as a community worker iv. Resident of that particular village v. Vetted or recommended by the local village council 64 vi. All of the above vii. Other (please specify) viii. Don’t Know 3. Who trained you? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Social welfare officers ii. MOHCDGEC/PORALG Staff iii. PACT/KK iv. JSI/CHSSP v. Institute of Social Work (ISW) vi. Others, Specify vii. Don’t know 4. What topics were you trained on [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Identification of OVC and their families ii. Referral provision iii. Linking OVC and their families with services iv. Follow-up with OVCs and their families v. Child development vi. Effects of neglect and abuse vii. Basic child and family assessment viii. Assessment of HIV risks, services, and adherence for children and adolescents (0-18 years) ix. Child protection x. Counseling skills xi. Principles of case management xii. Coordinating structures for KK/CHSSP xiii. Processes/steps of case management xiv. Standard operation procedures guiding the implementation of KK/CHSSP xv. Roles and responsibilities of the government and key stakeholders xvi. The use of case management tools xvii. Don’t know PROG: IF ANSWER IS DON’T KNOW, THEN GO TO QUESTION 6. xviii. Other, specify 5. Are you applying the knowledge and skills acquired from the case management training? i. Yes, frequently ii. Yes, sometimes iii. Yes, rarely iv. No, not at all v. Don’t know PROG: IF ANSWER IS NO, SKIP TO QUESTION 7. IF THE ANSWER IS YES AND DON’T KNOW, CONTINUE WITH QUESTION 6 6. Of the topics you were trained on, which did you use in your work? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Identification of OVC and their families ii. Referral provision iii. Linking OVC and their families with services iv. Follow up of the OVCs and their families 65 v. Child development vi. Effects of neglect and abuse vii. Basic child and family assessment viii. Assessment of HIV risks, services, and adherence for children and adolescents (0-18 years) ix. Child protection x. Counseling skills xi. Principles of case management xii. Coordinating structures for the KK/CHSSP xiii. Processes/Steps of case management xiv. Standard Operating Procedures guiding the implementation of KK/CHSSP xv. Roles and responsibilities of the government and key stakeholders xvi. The use of case management tools xvii. Don’t know xviii. Other, specify xix. None of these were useful [ENUMERATOR, PLEASE ASK THE BELOW QUESTION] 7. If you are not applying some of the things that you were trained on, why not? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. It wasn’t practical for me ii. Lack of support from management iii. I haven’t found time iv. Lack of confidence v. Lack of funding or resources vi. Lack of transportation vii. Busy with other required activities/Too little time viii. Other reasons, please specify (limited open field) ix. Don’t know 8. What other topics do you need training in? (Open-Mention) 9. Has the case management training you received helped you coordinate referrals and services for your clients from different government departments and other sources (e.g., protection, justice, social welfare, HIV/AIDS, Health, and Education)? [ENUMERATOR, PROMPT THE RESPONSES] i. Yes, to a large extent ii. Yes, to some extent iii. Yes, a little bit iv. No, not at all v. The training has made it worse vi. Don’t know Why do you say so? (open ended question) 10. Is there any support from the Lead CCW, Social Welfare Officer (SWO), or other staff to ensure that the CCW is able to care for, support, and protect beneficiaries? i. Yes, a lot of support ii. Yes, some support iii. No, not much iv. Don’t know Please rate the following statements on a scale of 1 to 5, with 1 representing strongly agree and 5 representing strongly disagree. 66 11. There are enough social welfare service providers to protect, care and support the people living with HIV, adolescent girls and young women, most vulnerable children, and their families. [ENUMERATOR, PROMPT THE RESPONSES] i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know 12. I am able to successfully refer beneficiaries to providers so that they receive all the services they need. [ENUMERATOR, PROMPT THE RESPONSES] i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know 13. Once I refer beneficiaries to providers, they are able to receive the referred services. [ENUMERATOR, PROMPT THE RESPONSES] i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly agree vi. Don’t know Why do you say so? (open ended question) 14. The referred services provided to beneficiaries do a good job meeting their needs or addressing their problems. [ENUMERATOR, PROMPT THE RESPONSES] i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly agree vi. Don’t know Why do you say so? (open ended question) 15. The case management training I received through the CHSSP program has improved access to social and health services for the groups of beneficiaries I serve. i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know Why do you say so? (open ended question) 67 16. Because of the case management training I received through the CHSSP program, we now refer and link clients to better quality services. i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know 17. The case management training I received through the CHSSP program has improved the effectiveness of referral systems in our community. i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know 18. The case management training I received through the CHSSP program has improved HIV prevention among the beneficiaries I serve. i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know Why do you say so? (open ended question) 19. The case management training I received through the CHSSP program has improved violence against children prevention and response. i. Strongly agree ii. Agree iii. Neither agree nor disagree iv. Disagree v. Strongly disagree vi. Don’t know Why do you say so? (open ended question) 20. What challenges do you face in doing your job? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Lack of transport to do my work ii. Wide coverage, hard to reach all iii. Too many households to visit per CCW, hard to reach all iv. The payment is not enough v. Not getting enough support from the Lead CCW vi. Not getting enough support from the SWO vii. Poor handling of cases by service providers beneficiaries are referred to viii. Beneficiaries not giving required cooperation ix. Other, specify 68 x. Don’t know 21. How likely is it that you will keep working as a community case worker over the long run after CHSSP program support has ended? [ENUMERATOR, PROMPT THE RESPONSES] i. Very likely ii. Somewhat likely iii. Not very likely iv. Not at all likely v. Don’t know PROG: IF ANSWER IS VERY LIKELY OR SOMEWHAT LIKELY, THEN GO TO QUESTION 19. IF ANSWER NOT VERY LIKELY OR NOT AT ALL LIKELY, GO TO QUESTION 20. IF ANSWERS DON’T KNOW, THEN END SURVEY. 22. What motivates you to keep working as a community case worker? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Because it is my passion and I’m committed to doing my work ii. Part of my day to day duties iii. I am motivated by the payment I get from the program iv. The number of households I am serving fits my ability v. I am getting enough support from the Lead CCW vi. I am getting enough support from the SWO vii. Referral authorities handles cases very well viii. Beneficiaries provide enough cooperation ix. Other, please specify (Open field) x. Don’t know 23. What challenges do you face in continuing working as a community case worker over the long￾run after USAID program support has ended? [ENUMERATOR, PLEASE TICK ALL THAT APPLY – DO NOT ASK THE OPTIONS] i. Lack of transport to do my work ii. Wide coverage, hard to reach all iii. Too many households to visit per CCW, hard to reach all iv. The payment is not enough v. Not getting enough support from the Lead CCW vi. Not getting enough support from the SWO vii. Poor handling of cases by the referral authorities viii. Beneficiaries not giving required cooperation ix. Other, specify x. Don’t know 69 ANNEX VI: CCW SURVEY RESULTS CCW Demographics Of the 269 CCWs surveyed a majority (55 percent) were females. Of the CCWs trained the highest number were from Dar es Salaam and Mbeya regions (19 percent and 16 percent respectively). Sampled Regions Female Male Total Percentage (n=269) Arusha 4 0 4 1% Dar es Salaam 37 13 50 19% Dodoma 1 1 0% Geita 4 11 15 6% Iringa 4 8 12 4% Kagera 8 5 13 5% Katavi 1 1 0% Kigoma 1 1 2 1% Kilimanjaro 8 3 11 4% Mara 3 1 4 1% Mbeya 24 18 42 16% Mjini Magharibi 1 1 0% Morogoro 4 6 10 4% Mwanza 12 6 18 7% Njombe 2 2 4 1% Pwani 5 8 13 5% Rukwa 1 1 2 1% Ruvuma 9 10 19 7% Shinyanga 3 7 10 4% Singida 4 1 5 2% Tabora 12 12 24 9% Tanga 2 6 8 3% Grand Total 148 121 269 100% The majority of CCWs completed primary school with 22 percent who completed a secondary school education. Less than 7 percent have a diploma or advanced degree. CCW Level of Education No. of Respondents Percentage University 8 3.0% Certificate 4 1.5% Diploma 6 2.2% Completed secondary 58 21.6% Some secondary 50 18.6% Completed primary 79 29.4% Some primary 59 21.9% Vocational training 5 1.9% Grand Total 269 70 Training Received Fifty-seven (57) percent of CCWs said that they had been trained by Pact/Kizazi Kipya while another 45 percent said they had been trained by JSI/ CHSSP. 2 Multiple Reponses Percentage (n=269) Social welfare officers 3 1.1% MOHCDGEC/PO-RALG Staff 6 2.2% PACT/KK 154 57.2% JSI/CHSSP 120 44.6% Institute of Social Work (ISW) 1 0.4% All CCWs received training on identification of OVC and their families. Training on assessment of HIV risks, services, and adherence to treatment was cited among a majority of CCWs (64 percent). Roughly one-third received training on: 1) the effects of neglect and abuse, 2) linking OVC and their families to services, and 3) child development topics. Over 20 percent surveyed received training on child protection or counseling skills. Topics Multiple Reponses Percentage (n=269) Identification of OVC and their families 178 66.2% Difficulties and their families 43 16.0% Referral provision 64 75.2 Linking OVC and their families with services 79 29.4% Child development 78 29.0% Effects of neglect and abuse 86 32.0% Basic child and family assessment 26 9.7% Children’s foundations 9 3.3% Assessment of HIV risks, services, and adherence for children and adolescents (0-18 years) 172 63.9% Child protection 60 22.3% Counseling skills 56 20.8% Principles of case management 27 10.0% Coordinating structures for Kizazi Kipya/CHSSP 4 1.5% Processes/steps of case management 19 7.1% Standard operation procedures guiding the implementation of Kizazi Kipya/CHSSP 5 1.9% Roles and responsibilities of the government and key stakeholders 2 0.7% The use of case management tools 25 9.3% 71 Training Benefits and Application of Knowledge and Skills Nearly 80 percent of CCWs say they applied skills from training frequently on the job. If Applied Skills Multiple Reponses Percentage (n=268) Don’t know 1 0.4% No, not at all 4 1.5% Yes, rarely 38 14.2% Yes, sometimes 11 4.1% Yes, frequently 214 79.9% Kagera and Mwanza had especially high application of knowledge and skills (both over 90 percent). Regions Don’t Know No, Not at All Yes, Rarely Yes, Sometimes Yes, Frequently Grand Total % Yes Frequent Arusha . . . . 4 4 100.0% Dar es Salaam . 2 11 2 35 50 70.0% Dodoma . . . . 1 1 100.0% Geita . . 2 . 13 15 86.7% Iringa . . 1 2 9 12 75.0% Kagera . . 1 12 13 92.3% Katavi . . . . 1 1 100.0% Kigoma . . . . 2 2 100.0% Kilimanjar o . 1 2 . 8 11 72.7% Mara . . . . 4 4 100.0% Mbeya . . 9 . 33 42 78.6% Mjini Magharibi . . . . 1 1 100.0% Morogoro . . 1 2 7 10 70.0% Mwanza . . 1 . 17 18 94.4% Njombe . . . 1 3 4 75.0% Pwani . 1 2 . 10 13 76.9% Rukwa . . 1 . 1 2 50.0% Ruvuma . . 1 2 16 19 84.2% Shinyanga . . 1 . 8 9 88.9% Singida . . 1 . 4 5 80.0% Tabora . . 2 2 20 24 83.3% Tanga 1 . 2 . 5 8 62.5% Grand Total 1 4 38 11 214 268 79.9% Roughly 84 percent believe to some or a large extent that the training helped them to coordinate referrals. Training Helped Coordinate Referrals Multiple Reponses Percentage (n=126) Don’t know 1 0.38% No, not at all 10 3.76% Yes, a little bit 32 12.03% Yes, to some extent 97 36.47% Yes, to a large extent 126 47.37% 72 Over 96 percent agree or strongly agree that they are able to successfully refer beneficiaries to providers. Able to Successfully Refer Beneficiaries to Providers Multiple Reponses Percentage (n=267) Don’t know 1 0.37% Strongly disagree 2 0.75% Disagree 4 1.50% Neither agree nor disagree 3 1.12% Agree 130 48.69% Strongly agree 127 47.57% Over 79 percent agree that beneficiaries are able to receive services once they have made a referral. Able to Receive Services Once They Have Made a Referral Multiple Reponses Percentage (n=263) Don’t know 4 1.50% Strongly disagree 3 1.12% Disagree 18 6.74% Neither agree nor disagree 26 9.74% Agree 100 37.45% Strongly agree 112 41.95% Over two-thirds of CCWs (67 percent) agree that the referred services meet clients’ needs. Referred Services Meet Clients’ Needs Multiple Responses Percentage (n=265) Don’t know 6 2.25% Strongly disagree 6 2.25% Disagree 31 11.61% Neither agree nor disagree 44 16.48% Agree 118 44.19% Strongly agree 60 22.47% The majority of CCWs (nearly 78 percent) agreed that the CCW training had contributed to improved HIV prevention among the beneficiaries served. CCW Training Had Contributed to Improved HIV Prevention Multiple Responses Percentage (n=255) Don’t know 2 0.75% Strongly disagree 3 1.12% Disagree 23 8.61% Neither agree nor disagree 19 7.12% Agree 128 47.94% Strongly agree 80 29.96% Over half (50.8 percent) agree that case management training improved beneficiaries’ access to services and another 37.6 percent strongly agree. Case Management Training Improved Beneficiaries’ Access to Services Multiple Responses Percentage (n=266) Don’t know 3 1.1% Strongly disagree 0 0.0% Disagree 10 3.8% Neither agree nor disagree 18 6.8% Agree 135 50.8% Strongly agree 100 37.6% 73 Just under one-half (49.1 percent) agree that the case management training CCWs received has improved linkages to better quality services and another 46.4 percent strongly agree. Case Management Training Received Has Improved Linkages to Better Quality Services Multiple Responses Percentage (n=267) Don’t know 0 0.0% Strongly disagree 2 0.7% Disagree 1 0.4% Neither agree nor disagree 9 3.4% Agree 131 49.1% Strongly agree 124 46.4% The majority (51.7 percent) agree the CCW training improved effectiveness of referral systems in the community and another 39.7 percent strongly agree. CCW Training Improved Effectiveness of Referral Systems Multiple Responses Percentage (n=267) Don’t know 8 3.0% Strongly disagree 3 1.1% Disagree 2 0.7% Neither agree nor disagree 10 3.7% Agree 138 51.7% Strongly agree 106 39.7% Nearly half (48.9 percent) agree that CCW training led to improved VAC prevention and response; in contrast, another 42.5 percent strongly disagree that it has. Whether the Training Led to Improved VAC Prevention and Response Multiple Responses Percentage (n=260) Don’t know 6 2.3% Strongly disagree 113 42.5% Disagree 4 1.5% Neither agree nor disagree 10 3.8% Agree 130 48.9% Strongly agree 3 1.1% Sustainability and Success Factors Key Drivers and Support Over 93 percent believe they have enough support from their supervisors and leads. Receive Enough Support from Their Supervisors and Leads Multiple Responses Percentage (n=267) Don’t know 2 0.75% No, not much 16 5.99% Yes, some support 94 35.21% Yes, a lot of support 155 58.05% 74 Over 80 percent of CCWs agree that there are enough social welfare providers to support OVC and PLHIV. There Are Enough Social Welfare Providers to Support OVC and PLHIV Multiple Responses Percentage (n=266) Don’t know 1 0.37% Strongly disagree 8 3.00% Disagree 23 8.61% Neither agree nor disagree 18 6.74% Agree 136 50.94% Strongly agree 80 29.96% Motivation and Likelihood of Continuing Work The broad majority (86 percent) feel it is very likely they will continue to work as CCWs. Continue to Work as CCWs Multiple Responses Percentage (n=267) Don’t know 1 0.4% Not at all likely 8 3.0% Not Sure 8 3.0% Somewhat likely 21 7.9% Very likely 229 85.8% A majority are motivated by their passion for the work (53 percent) or a sense of duty (40.5 percent) and are not motivated by the financial incentive (only cited among 2 percent as a motivation). Motivation Multiple Responses Percentage (n=269) Because it is my passion and I’m committed to doing my work 142 52.8% Part of my day to day duties 109 40.5% I am motivated by the payment I get from the project 5 1.9% The number of households I am serving fits my ability 16 5.9% I am getting enough support from the Lead CCW 5 1.9% I am getting enough support from the SWO 3 1.1% Referral authorities handles cases very well 2 0.7% Beneficiaries provide enough cooperation 29 10.8% Implementation Challenges Lack of cooperation from beneficiaries (43 percent) and lack of transport (40 percent) were the most commonly cited challenges. Nearly one-quarter cited that payment was not enough. Another 21 percent cited problems with reaching/covering beneficiaries (links to transport). Implementation Challenges Multiple Responses Percentage (n=269) Lack of transport to do my work 107 39.8% Wide coverage, hard to reach all 56 20.8% Too many households to visit per CCW, hard to reach all 23 8.6% The payment is not enough 65 24.2% Not getting enough support from the Lead CCW 9 3.3% Not getting enough support from the SWO 10 3.7% Poor handling of cases by the service providers beneficiaries are referred to 11 4.1% Beneficiaries not giving required cooperation 116 43.1% Don’t know 1 0.4% 75 Sustainability Challenges Lack of resources—including lack of transport—was one of the most reported challenges among 32 percent with 17.5 percent saying that their compensation is not enough. Another common challenge is lack of cooperation from beneficiaries (cited by 11.5 percent). Challenges That Might Keep Me from Continuing to Work as a Caseworker Multiple Responses Percentage (n=269) Lack of transport to do my work 87 32.3% Wide coverage, hard to reach all 22 8.2% Too many households to visit per CCW, hard to reach all 7 2.6% The payment is not enough 47 17.5% Not getting enough support from the Lead CCW 8 3.0% Not getting enough support from the SWO 6 2.2% Poor handling of cases by the referral authorities 12 4.5% Beneficiaries not giving required cooperation 31 11.5% Don’t know 23 8.6% 76 ANNEX VII: KII AND FGD PARTICIPANTS BY STAKEHOLDER AND LOCATION31 Data Collection Activities National (Dar es Salaam and Dodoma) and Temeke Municipal Council (MC) (Dar es Salaam) Mbeya City Council (CC) (Mbeya) Njombe Town Council (TC) (Njombe) Nyamagana MC (Mwanza) Bukoba District Council (DC) (Kagera) Total National-Level KIIs/Group Interviews (GIs) Implementing Consortium: JSI, WEI, Pact-Tz, American International Health Alliance (IAHA) 6 - - - - 6 GoT: PO-RALG, & TACAIDS 2 - - - - 2 National CSOs: NACOPHA, ISW 2 - - - - 2 USAID: Technical staff 1 - - - - 1 Total National-Level KIIs 11 - - - - 11 Regional-Level KIIs/GIs RCDOs, RCT, Regional Social Welfare Officers (RSWOs), Master Trainers 1 1 1 1 2 6 Total Regional-Level KIIs 1 1 1 1 2 6 Council-Level KIIs/GIs CHACs, District AIDS Control Coordinators (DACCs), DSWOs, DMTs, DCDOs, District Medical Officers (DMOs) 1 1 1 2 1 6 CSO Subgrantees: CARITAS￾Mbeya, Tanzania Development and AIDS Prevention (TADEPA)- Kagera, Community Concerns of Orphans and Development Association (COCODA)-Njombe, Mwanza Outreach Care and Support Organization (MOCSO)-Mwanza - 1 1 1 1 4 PLHIV Clusters 1 1 1 - - 3 CCW Supervisors (Ward SWOs) - 1 1 1 1 4 Total Council-Level KIIs 2 4 4 4 3 17 Total KIIs 14 5 5 5 5 34 Village and Ward-Level FGDs WMACs 1 1 1 1 - 4 VMACs 1 1 1 1 - 4 MVCCs - - - 1 - 1 31 For confidentiality purpose, participants names are not provided. 77 Data Collection Activities National (Dar es Salaam and Dodoma) and Temeke Municipal Council (MC) (Dar es Salaam) Mbeya City Council (CC) (Mbeya) Njombe Town Council (TC) (Njombe) Nyamagana MC (Mwanza) Bukoba District Council (DC) (Kagera) Total CCWs 1 1 1 1 1 5 Total FGDs 3 3 3 4 1 14 78 ANNEX VIII: LIST OF KII PARTICIPANTS32 Date Location Region Stakeholder Type Intervie w Type (FGD, GI, KII) # Participant s # Femal e # Mal e 2/12/1 9 Bukoba DC Kagera CSO-TADEPA KII 8 3 5 2/13/1 9 Bukoba DC Kagera DMO, DACC, and DSWO KII 2 0 2 2/11/1 9 Bukoba DC Kagera RCDO_RSWO KII 2 0 2 2/12/1 9 Bukoba DC Kagera CCWs FGD 15 6 9 2/11/1 9 Bukoba DC Kagera District Master Trainer KII 1 1 0 2/13/1 9 Bukoba DC Kagera CCW Supervisors Ward Community Development Officer/Ward Social Welfare Officer (WCDO/WSWO) KII 5 3 2 2/12/1 9 Mbeya CC Mbeya VMACs FGD 10 6 4 2/12/1 9 Mbeya CC Mbeya WMACs FGD 10 7 3 2/13/1 9 Mbeya CC Mbeya PLHIV Clusters KIIs 3 0 3 2/15/1 9 Mbeya CC Mbeya CCWs FGDs 8 5 3 2/12/1 9 Mbeya CC Mbeya RCDO_RSWO, RCT KII 2 2 0 2/13/1 9 Mbeya CC Mbeya DSWO, DCDO KII 2 2 0 2/15/1 9 Mbeya CC Mbeya CSO-CARITAS KII 4 3 1 2/12/1 9 Mbeya CC Mbeya CCW Supervisors (WCDO/WSWO) FGD 5 5 0 2/26/1 9 National Dar es Salaam TACAIDS KII 2 2 0 2/20/1 9 National Dar es Salaam American International Health Alliance (AIHA) KII 2 2 0 2/21/1 9 National Dar es Salaam ISW KII 1 1 0 2/25/1 9 National Dar es Salaam CHSSP-NPA VAWC Component KII 3 1 2 2/25/1 9 National Dar es Salaam CHSSP-CSO Component KII 3 1 2 2/25/1 9 National Dar es Salaam CHSSP-MACs Component KII 3 1 2 2/25/1 9 National Dar es Salaam NACOPHA KII 1 0 1 32 For confidentiality purpose, participants names are not provided. 79 Date Location Region Stakeholder Type Intervie w Type (FGD, GI, KII) # Participant s # Femal e # Mal e 2/21/1 9 National Dar es Salaam PACT KII 1 1 0 3/4/19 National Dar es Salaam USAID KII 2 1 1 1/16/1 9 National Dodom a PO -RALG Social Welfare Department KII 1 1 0 2/20/1 9 National Dar es Salaam WEI KII 1 1 0 2/19/1 9 Njombe TC Njombe VMACs FGD 10 1 9 2/19/1 9 Njombe TC Njombe WMACs FGD 10 2 8 2/20/1 9 Njombe TC Njombe PLHIV Clusters KII 4 2 2 2/18/1 9 Njombe TC Njombe RSWO, RCDO, RCT KII 3 1 2 2/21/1 9 Njombe TC Njombe DAC C, CHAC, DSWO KII 3 2 1 2/19/1 9 Njombe TC Njombe CSO -COCODA KII 6 5 1 2/1819 Njombe TC Njombe CCWs FGD 10 7 3 2/20/1 9 Njombe TC Njombe CCW Supervisors (WCDO/WSWO) KII 3 3 0 2/6/19 Nyamagan a MC Mwanza VMACs FGD 7 7 0 2/6/19 Nyamagan a MC Mwanza WMACs FGD 12 9 3 2/7/19 Nyamagan a MC Mwanza NPA VAWC KII 7 5 2 2/8//19 Nyamagan a MC Mwanza CCWs FGD 15 12 3 2/6/19 Nyamagan a MC Mwanza DMO, DSWO KII 2 1 1 2/6/19 Nyamagan a MC Mwanza CHAC KII 1 1 0 2/7/19 Nyamagan a MC Mwanza CSO -MOCSO KII 3 1 2 2/4/19 Nyamagan a MC Mwanza RCT, RSWO KII 2 0 2 2/5/19 Nyamagan a MC Mwanza CCW Supervisors (WCDO/WSWO) KII 8 8 0 2/12/1 9 Temeke MC Dar es Salaam VMACs FGD 8 1 7 2/12/1 9 Temeke MC Dar es Salaam WMACs FGD 10 2 8 2/15/1 9 Temeke MC Dar es Salaam CCWs FGD 12 3 9 2/12/1 9 Temeke MC Dar es Salaam RCDO_RSWO KII 5 4 1 80 Date Location Region Stakeholder Type Intervie w Type (FGD, GI, KII) # Participant s # Femal e # Mal e 2/12/1 9 Temeke MC Dar es Salaam DACC, CHAC, DMT, DSWO KII 6 4 2 2/13/1 9 Temeke MC Dar es Salaam PLHIV Clusters KII 5 2 3 Totals 48 249 138 111 81 ANNEX IX: DISCLOSURE OF ANY CONFLICTS OF INTEREST DISCLOSURE OF CONFLICT OF ANY INTEREST Name Gary Woller Title Director of Monitoring and Evaluation Organization ME&A Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or another instrument) USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) Community Health and Social Welfare Systems Strengthening Program I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. - - I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 17, 2019 82 DISCLOSURE OF CONFLICT OF ANY INTEREST Name Dr. Zena M. Mabeyo Title OVC Expert Organization USAID/Data for Development Evaluation Position Team Leader Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) Community Health System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. I have no potential conflict of Interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date November 13, 2018 X X 83 DISCLOSURE OF CONFLICT OF ANY INTEREST Name Nasson Konga Title Monitoring & Evaluation Specialist Organization NORC at the University of Chicago (Data for Development) Evaluation Position Team Leader X Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) USAID Community Health System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. No conflict of interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 15, 2018 84 85 DISCLOSURE OF CONFLICT OF ANY INTEREST Name Gerald Usika Title Survey Specialist Organization Data for Development Program Evaluation Position Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) USAID Community Health and Social Welfare System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. There are no potential conflicts of interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 9/15/2018 86 DISCLOSURE OF CONFLICT OF ANY INTEREST Name MKINGAMA ADOLPH KAPINGA Title Public Health and Health System Expert Organization USAID/Data for Development Evaluation Position Team Leader Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) Community Health System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. No any conflict of Interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date December 21, 2018 X X 87 DISCLOSURE OF CONFLICT OF ANY INTEREST Name Julie Tumbo Title Local HIV and OVC Research Expert Organization Data for Development Program Evaluation Position Team member Evaluation Award Number(contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated(Include program name(s), implementer name(s) and award number(s), if applicable) USAID Community Health and Social Welfare System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 25. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 26. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 27. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 28. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 29. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 30. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. There are no potential conflicts of interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date November 15, 2018 88 DISCLOSURE OF CONFLICT OF ANY INTEREST Name Bahati P. Tenga Title Research and Evaluation Expert Organization Data for the Development Evaluation Position Team Leader Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) USAID Community Health and Social Welfare System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 31. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 32. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 33. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 34. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 35. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 36. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. No potential conflicts of interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date December 11, 2018 √ √ 89 DISCLOSURE OF CONFLICT OF ANY INTEREST Name Madihi, Mlwande Charles Dan Title Child Protection/Rights Evaluation Specialist Organization Data for Development Evaluation Position Team Leader Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) USAID Community Health and Social Welfare System Strengthening Program (CHSSP). I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 37. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 38. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 39. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 40. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 41. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 42. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. No potential conflicts of interest to disclose. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 11/15/2018 √ √ 90 DISCLOSURE OF CONFLICT OF INTEREST Name Daud Siwalaze Title M&E Specialist Organization NORC at the University of Chicago (Data for Development) Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or another instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Program(s) Evaluated (Include program name(s), implementer name(s) and award number(s), if applicable) Community Health and Social Welfare System Strengthening (CHSSP); AID-621-A-14-00004 I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the program(s) being evaluated or the implementing organization(s) whose program(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose programs are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the program(s) being evaluated, including involvement in the program design or previous iterations of the program. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose program(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose program(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular programs and organizations being evaluated that could bias the evaluation. - - I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 15, 2018 91 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523