DISCLAIMER This report was made possible by the support of the American People through the United States Agency for International Development (USAID) under the terms of the Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) contract AID-OAA-M-16-00006. CIRCLE is implemented by Social Solutions International, Inc. Views expressed are not necessarily of USAID or other affiliated institutions. TANZANIA DEVELOPMENTAL EVALUATION FINAL REPORT 2022 Coordinating Implementation Research to Communicate Learning and Evidence Project (CIRCLE) February 28, 2022 Contract Number: AID-OAA-M-16-00006 Activity Lead: Rose Kyando, USAID/Tanzania COR: Sara Sulzbach, USAID/Washington ELIZABETH MACGREGOR-SKINNER, CIRCLE PROJECT CONTENTS EXECUTIVE SUMMARY 1 INTRODUCTION 3 OBJECTIVES 3 KEY ACHIEVEMENTS 3 APPROACH 5 DESIGN AND METHODOLOGY 5 SUMMARY OF ACTIVITY IMPLEMENTATION AND TIMELINE 9 COVID-19 PANDEMIC 10 DELIVERABLES 11 CHALLENGES FACED DURING IMPLEMENTATION 13 RECOMMENDATIONS 14 FIGURES AND TABLES FIGURES FIGURE 1. STAKEHOLDER ENGAGEMENT PROCESS 6 FIGURE 2. DE IMPLEMENTATION TIMELINE 10 TABLES TABLE 1: ACTIVITIES BY PROJECT YEAR 9 TABLE 2: PROJECT DELIVERABLES 11 ABBREVIATIONS BA Boresha Afya CIRCLE Coordinating Implementation Research to Communicate Learning and Evidence CLA Collaborating, Learning, and Adapting COVID-19 Coronavirus Disease 2019 DE Developmental Evaluation DEPA-MERL Developmental Evaluation Pilot Activity – Monitoring, Evaluation, Research, and Learning GoT Government of Tanzania IP Implementing Partner MEL Monitoring, Evaluation, and Learning MoHCDGEC Ministry of Health, Community Development, Gender, Elderly and Children PY Project Year RCCE Risk Communication and Community Engagement R/CHMTs Regional and Community Health Management Teams ZDEs Zonal Developmental Evaluators 1 | TANZANIA DE FINAL REPORT USAID.GOV EXECUTIVE SUMMARY The Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) project (AID-OAA-M-16-00006) supported USAID Tanzania with designing and implementing a developmental evaluation (DE) of its flagship health project, Boresha Afya (BA), as a mechanism to support real-time evidence generation and adaptive management of health service integration in Tanzania. The DE was carried out over a four-year period that coincided with implementation of the BA activity. The objectives of the DE included understanding the fidelity of program implementation, as well as contextual challenges to program implementation. Developmental evaluation as an evaluation methodology is distinguished by embedding outside evaluators in the programs that they are evaluating. This approach, which was reluctantly accepted by some stakeholders at the beginning of the activity, was enthusiastically embraced at the end of its engagement. The DE embedded evaluators were continuously sharing learning and brainstorming solutions with implementing partners and local health management teams. In doing so, new learning with actionable information helped implementing partners and USAID to quickly adapt their programs in real￾time. Meetings with regional and national health implementing partners and USAID resulted in greater collaboration among stakeholders across regions by discussing cross-cutting issues, contextual challenges, and health systems limitations to the scale up of integrated service delivery. CIRCLE embedded evaluators in offices of three Boresha Afya implementing partners in Dodoma, Iringa, and Mara regions, who helped carry out routine facility observation and assessments with the partners and government health management teams. CIRCLE also conducted targeted qualitative studies including outcome harvesting, rapid reconnaissance, and secondary data review. CIRCLE also developed case studies on client satisfaction and provider workload perspectives in relationship to integrated services. Using the DE approach, CIRCLE helped USAID Tanzania, implementing partners, and local government health teams learn and apply lessons and effectively address service delivery gaps, leading to improvements in integrated health care delivery in three regions. Furthermore, this approach has catalyzed collaboration within and across the three regions where it was implemented. CIRCLE has shared lessons to optimize operational approaches across facilities to effectively deliver integrated health care with the policy-level decision makers (see box on CIRCLE’s approach to stakeholder engagement for this DE). Key findings from the DE included: • Validation that health service integration strengthened access, use, and quality of services, and introduced service efficiencies • Integration of family planning and HIV services led to improved quality of care and client satisfaction • Family planning and HIV integrated services were strengthened at HIV care and treatment centers and pre-natal care checkups • Collaborative learning and adaptation (CLA) approaches improved programming during the project cycle USAID.GOV TANZANIA DE FINAL REPORT | 2 Additionally, the DE helped identify systemic constraints to implementing and scaling up health service integration in Tanzania. This knowledge was synthesized and shared with USAID and the Ministry of Health for further action. Over the life of the DE, CIRCLE: • Carried out 720 site visits • Produced 12 reports from targeted data collection efforts – focusing on specific aspects of integrated service delivery • Wrote 16 briefs/blogs for stakeholders to understand new data or synthesized findings of the DE • Delivered eight stakeholder workshops • Provided 19 technical presentations • Produced two tools/dashboards for stakeholder use This report summarizes CIRCLE’s activities, achievements, challenges over the four years that the DE was conducted. 3 | TANZANIA DE FINAL REPORT USAID.GOV INTRODUCTION In 2017, USAID Tanzania commissioned a developmental evaluation (DE) of its flagship health project, Boresha Afya (BA), as a mechanism to support real-time evidence generation and adaptive management of health service integration in Tanzania. The USAID Coordinating Implementation Research to Communicate Learning and Evidence Project (CIRCLE) designed and implemented the DE from November 2017 to January 2022. The BA Project supported the Government of Tanzania (GoT) to increase access to high quality, comprehensive, and integrated health services, particularly for women and youth. The DE provided real-time evidence and learning about how the project implementers were supporting integrated health service delivery in three geographic regions, and about the quality, efficiency, and utilization of integrated health services. OBJECTIVES The DE had four interrelated objectives. 1. Evaluate on an ongoing basis, with USAID Tanzania and implementing partners, factors that promote, and challenges that arise from, integrated service delivery focusing on quality, utilization and efficiency. 2. Assess the fidelity of implementation, including feasibility, acceptability and sustainability of Boresha Afya’s integration approach. 3. Provide timely and regular feedback and recommendations to implementing partners, based on the findings; promote course shifts in implementation; and strengthen integrated and client￾centered healthcare with a view to improving health outcomes. 4. Promote optimal learning and improvement among key stakeholders from the project zones, including implementing partners, the regional representatives of the GoT, USAID Tanzania, service providers and managers. KEY ACHIEVEMENTS CIRCLE effectively carried out the DE over the time period - which led to new learning among stakeholders, as well as actionable information that helped implementing partners and USAID to quickly adapt their programs within the program cycle. The DE focused attention on the aspect of healthcare integration - which is a priority for the Government of Tanzania but was not the specific focus of any of the Boresha Afya partners. • Participating in a novel new approach for program learning and adaptation engendered collaboration among various stakeholders in the health sector on a cross-cutting issue. This included local and regional government health management teams, USAID and its implementing partners. Engaging these stakeholders elevated dialog and led to solutions to more effectively integrate health services in Tanzania. Stakeholders, as well, expressed enthusiasm and appreciation for the DE process, and interest in continuing the approach as a management approach to improve health services. • The DE demonstrated that integrated health services resulted in more health services being delivered in the regions where Boresha Afya operated. The DE tracked indicators from implementing partners as well as monitored the uptake of program changes. Programs that made USAID.GOV TANZANIA DE FINAL REPORT | 4 more changes saw uptake of a greater variety of health services including HIV, MNCH, family planning, malaria, and nutrition services. • Having a DE increased the speed of uptake of service delivery improvements for Boresha Afya Partners. CIRCLE introduced a “DE Tracker” - a dashboard which monitored the rate of uptake of prioritized actions among partners. The Tracker was shared with stakeholders each quarter. Having this accountability tool showed the increasing uptake of prioritized actions over time. Through the period that CIRCLE monitored the uptake of results, the uptake of 116 actions out of 157 prioritized actions among partners were noted. • The DE was designed to work with local and regional health service delivery platforms. It became clear, however, that for many program changes to take place, national policy and guidelines would need to be changed at the Ministry of Health (policy) national level. Recognizing this need, CIRCLE and USAID synthesized summative learnings and recommendations from the DE which were shared with the Ministry of Health of Tanzania to suggest policy changes in order to optimize scalable and efficient integrated health service delivery. • CIRCLE completed a number of reports to influence local implementers, which are noted in the report below. In addition, to share results and learning from the DE with a broader global audience, CIRCLE carried out presentations at the DE through three international meetings. CIRCLE participated and presented through USAID’s Developmental Evaluation Pilot Activity – Monitoring, Evaluation, Research, and Learning (DEPA-MERL) working group, developed a number of blogs on how to use DE for USAID’s ‘Harpnet’ website, and developed a brief comparing the Tanzania DE with Indonesia’s health DE, with the goal of sharing learnings of DE in health with a larger USAID audience. 5 | TANZANIA DE FINAL REPORT USAID.GOV APPROACH DESIGN AND METHODOLOGY A DE was identified as a useful approach given the complex nature of the activity and because it enabled Boresha Afya stakeholders to understand how to improve service integration in Tanzania as activities were being implemented. The DE employed a variety of data collection methods and engagement tactics to provide insights on project implementation and emerging results to support improving Boresha Afya within its project cycle. The DE was designed as a structured, process-driven approach utilizing a series of data collection tools and engagement tactics to gather evidence and provide insights on project implementation for quick adaptation and uptake of learning. The purpose of the DE was to evaluate the quality, utilization, efficiency, and implementation of fidelity of health service integration in the USAID Boresha Afya project in three geography zones covering 19 regions in Tanzania. The DE was carried out prospectively over the course of the implementation of Boresha Afya. Based on an initial design approved by USAID, CIRCLE worked to orient the BA partners through an enculturation process which included: meetings, and a codesign workshop that identified the BA project’s theory of change. At this workshop, stakeholders agreed on common areas and questions concerning integration that would be explored by the DE that would be of benefit to all stakeholders; CIRCLE also explained the dynamics of a developmental evaluation. Based on this workshop, CIRCLE developed the Year One evaluation plan for DE activities. A key aspect of the design was placement of embedded DE evaluators in each of the Boresha Afya regional offices. EMBEDDED EVALUATORS Developmental evaluation involved a collaborative partnership between embedded evaluators and implementing partners (IPs). Zonal Developmental Evaluators (ZDEs) were embedded with Boresha Afya IPs in each of the three regions where the activity took place. The embedded evaluators used a variety of data collection methods and engagement tactics over the four-year project lifecycle to provide insights on implementation processes and emerging results and to support the improvement of Boresha Afya. They provided continuous feedback to support timely decision-making and track milestones in the project. ZDEs also helped to build capacity among IPs on utilization of research findings and tracking recommendations, as well as developing a DE learning framework. The embedded evaluators played a critical role in providing real-time feedback through site visits, face-to-face meetings, and ongoing interaction with Boresha Afya IPs, regional and community health management teams (R/CHMTs), and health facility staff. SPECIAL QUALITATIVE STUDIES Special qualitative studies contributed to data collection while providing insights into the Boresha Afya project: • Rapid reconnaissance was a primary data collection and analysis protocol utilized for the DE, which enables relatively quick but intensive data collection and analysis. • Outcome harvesting was utilized as a means to collect data and measure partner engagement to document how Boresha Afya has contributed to changes leading to successful implementation of USAID.GOV TANZANIA DE FINAL REPORT | 6 various health services integration models that was not part of IP tracking. Outcome harvesting was conducted during its first and third years. • A client satisfaction and waiting time study was carried out to understand clients’ preference and satisfaction with integrated services, as well as perceptions about wait times. • A provider case study was conducted to understand how providers’ perceived workloads influence consistent provision of quality integrated services. • A DE process case study was developed to inform future USAID DE design and document how the DE was implemented over time. STAKEHOLDER ENGAGEMENT Throughout the life of the project CIRCLE engaged the Government of Tanzania and Boresha Afya stakeholders at the regional and national levels. Figure 1. Stakeholder engagement process CIRCLE’s DE team met with USAID’s DE management team monthly, which included an activity lead, technical advisor and the three Boresha Afya activity managers. These meetings were carried out to ensure that communication from CIRCLE was reaching the USAID managers of the Boresha Afya projects with key observations and recommendations. ZDEs were embedded in regional offices of the Boresha Afya implementing partners in Tabora (later Dodoma), Mara, and Iringa. With the IPs, the ZDE’s met regularly with the R/CHMTs through joint site visits, feedback on work plans, and real-time solution sessions. ZDEs analyzed site monitoring data and identified contextual challenges to integrated services for specific settings. These meetings took place on an ongoing basis, with ZDE’s visiting their catchment sites at least once per quarter. 7 | TANZANIA DE FINAL REPORT USAID.GOV The ZDE’s also participated with researchers from CIRCLE’s Dar es Salaam office to collect data for key qualitative studies. Findings from these periodic studies, as well as ongoing data captured through monitoring visits were shared with all stakeholders at annual joint feedback and learning meetings. These meetings included USAID, the President’s Office of Regional Administration and Local Governance, the Boresha Afya partners and CIRCLE’s DE team. At these meetings, the regional teams would identify key gaps and prioritize actions to improve health services. The regional level had more diverse and frequent modes of formal and informal engagement, owing to the embedded ZDEs. Activities at this level included evidence gathering, weekly meetings, informal consultations, joint partner planning meetings, and regional feedback and learning meetings. ZDEs also produced information gathering, brief and report writing, and action planning at a more rapid pace than was possible at the National level and collaborated frequently with regional stakeholders. DISSEMINATION AND KNOWLEDGE MANAGEMENT A key component of the DE approach to stakeholder engagement was the sharing of technical findings at the regional and national level. This evolved over time as the CIRCLE team and stakeholders identified different venues and needs for dissemination, feedback, and learning. The majority of dissemination events and engagements were conducted using participatory and consultative approaches. CIRCLE developed the DE tracker to share evidence and action planning and to document key findings, priority actions, and adaptations by the Boresha Afya programs. The tracker, which was updated monthly, was used during DE management meetings with USAID, at national-level feedback and learning meetings, and regional feedback and learning meetings. Additionally, the team also wrote and distributed “DE Snapshots'' – a newsletter that provided stakeholders with updates about recent findings, the activities of the DE evaluators, and any adaptations identified in the field. The DE tracker was a tool highly valued by USAID and the Boresha Afya team to measure monthly progress, adaptations, and improvement. ENGAGMENT WITH OUTSIDE STAKEHOLDERS AND USAID Given that the DE is a novel approach which has rarely been applied in USAID health programs, it was important for CIRCLE to share progress and learning on the DE to stakeholders not immediately involved in the project. The following actions were important to bringing greater visibility to the DE in Tanzania for Boresha Afya: • Content on USAID’s Harpnet website - including a dedicated page for the DE, and six blogs on the implementation of the DE • Learning report of the Enculturation Meeting (report on the co-design process at the end of the process of ‘enculturation’ to the DE) • Presentation at the USAID East Africa Share Fair (Entebbe, June 2018) • Collaboration with USAID’s DEPA-MERL working group (presentations, briefs) • Presentation at the Health Systems Research Symposium (Liverpool, October 2018) USAID.GOV TANZANIA DE FINAL REPORT | 8 • Presentation to global audiences through USAID’s Health Research Program Dissemination Series (October 2021) • Process Case Study of Developmental Evaluation • A report titled “Synthesis of Policy and Systems-Level lessons learned from the Boresha Afya Developmental Evaluation” - which shares summative findings from the DE with Tanzania's Ministry of Health and USAID • Subsequent dissemination meetings with USAID and the Ministry of Health to apply lessons from the DE • A brief on Lessons Learned from Health DE’s with DEPA-MERL partners 9 | TANZANIA DE FINAL REPORT USAID.GOV SUMMARY OF ACTIVITY IMPLEMENTATION AND TIMELINE Key activities for each project year (PY) are detailed in the table below: Table 1: Activities by project year Key activities / milestones PY 1 Nov 2017 - Sept 2018 ● Startup activities ○ Recruitment of project staff ○ Establishment of office in Tanzania ● Stakeholder consultative meetings ● Enculturation and inception workshop ● Year One evaluation plan/workplan ● Recruitment and initial embedding of ZDEs PY 2 Oct 2018 - Sept 2019 ● Outcome harvesting and shared with stakeholders (regional reports and cross-cutting report) ● Process tracing continued ● Design and implementation of rapid reconnaissance activities in target facility and community sites ● Learning and planning sessions with Boresha Afya partners and other stakeholders ● Design and implementation of client satisfaction study PY 3 Oct 2019 - Sept 2020 ● DE Tracker is launched ○ Monthly snapshots brief from DE Tracker is developed ● ZDE for North/Central Zone shift from Tabora to Dodoma region after changes in Boresha Afya zonal scope ● CIRCLE expands regional facility sample by 66% from 12 to approximately 20 facilities ● Outcome harvesting continues ● Technical and learning briefs developed ● CIRCLE shifts activities to primarily virtual data collection and stakeholder engagement formats due to COVID-19 pandemic restrictions PY 4 Oct 2020 - Sept 2021 ● Provider workload study ● Final dissemination events ● Systems and policy analysis, process documentation case study ● GoT and USAID feedback and learning meetings USAID.GOV TANZANIA DE FINAL REPORT | 10 Figure 2. DE implementation timeline COVID-19 PANDEMIC Prior to the COVID-19 pandemic, CIRCLE Project staff routinely traveled to Tanzania to provide technical assistance and participate in stakeholder engagement activities. In 2020, Tanzania was affected by the outbreak and put in place isolation and travel restrictions and quarantine measures to reduce transmission risks and curb the spread of the virus. The CIRCLE Project staff primarily worked from home and did not carry out routine travel for most of 2020 and 2021; this includes staff in Dar es Salaam as well as embedded evaluators in the three regions. These restrictions limited CIRCLE’s spending on travel for the DE, resulting in unused funds at the end of PY3. CIRCLE, in discussion with USAID, proposed that these unused funds be redirected to USAID Tanzania’s COVID-19 response through two new activities: • Tanzania COVID-19 risk communication and community engagement (RCCE) activities at the national, regional and community levels: CIRCLE supported the development and roll out of COVID-19 communication materials and tools including strengthening a toll-free call center; engaging social media influencers; placement of critical information on national radio stations; printing and dissemination of existing RCCE materials; and repackaging COVID-19 vaccine promotional messages. • CIRCLE supported USAID and Tanzania’s Ministry of Health, Community Development, Gender, Elderly and Children (MoHCDGEC) through the procurement and distribution of personal protective equipment and informational materials to complement government efforts in combating the COVID-19 pandemic in regions where the DE was operating. 11 | TANZANIA DE FINAL REPORT USAID.GOV DELIVERABLES Table 2. Project deliverables Deliverables by year PY1 Nov 2017 - Sept 2018 ● DE start-up plan ● Enculturation workshop slide deck ● Enculturation workshop report ● Theory of change ● Evaluation plan ● Evaluation tracker ● Semi-annual technical reports ● Annual progress report PY2 Oct 2018 - Sept 2019 ● Annual progress report ● Technical reports on evaluation findings ○ Outcome harvesting reports ● Evaluation tracker ○ Monthly snapshot reports PY3 Oct 2019 - Sept 2020 ● Technical briefs ○ Case Study: Client flow and satisfaction ○ Outcome Harvest report (round two) ● Evaluation tracker ○ Monthly snapshot reports ● Annual progress report PY4 Oct 2020 – Sept 2021 ● Case Study: DE systems and policy level learnings on integrated health services ● Case Study: Lessons learned from implementing developmental evaluation in Tanzania ● Case Study: Provider perspectives on workload and integrated services ● Development of DE manuscript for publication USAID.GOV TANZANIA DE FINAL REPORT | 12 ● Presentation of DE learning at USAID Health Research Program Dissemination Series ● Technical presentations and consultations with the Ministry of Health ● Brief: USAID Learnings from Two Developmental Evaluations for Health (based on Tanzania DE and Indonesia DE) ● DE Final Report 13 | TANZANIA DE FINAL REPORT USAID.GOV CHALLENGES FACED DURING IMPLEMENTATION While the DE offered a new and innovative approach to help understand how to improve service integration within the Boresha Afya project, it came with a unique set of challenges. The primary challenges centered on the utilization-focus and complexity and systems principles of DE. As an unfamiliar methodology to all the partners’ involved, there were recurring challenges aligning expectations of the DE to its technical scope. Complexity-aware methods were new to many of the Boresha Afya stakeholders, and uptake of the approach was slow as its value was not initially clear to all stakeholders. An earlier understanding of this might have driven systems-level learning and innovation. Key challenges are summarized below: • A significant amount of start-up time was required to enculturate/orient stakeholders to the methodology. The concept of the embedded evaluators and the difference between the DE and traditional monitoring, evaluation, and learning (MEL) approaches required orientation, and a gradual acceptance of embedded collaborators. CIRCLE shared a number of resources about developmental evaluation, focusing on the value of process learning from USAID’s DEPA-MERL site, as well as custom-developed slides explaining a DE. • Lack of familiarity with complexity-aware methods among the many stakeholders that CIRCLE had to manage made the DE challenging to implement. As the study continued, stakeholders had to be continually refreshed in terms of what could be expected from a DE, and the difference between it and traditional evaluations. There was constant pressure on the DE to demonstrate ‘findings’ that could be ‘applied’, when in reality, the learnings continually resulted in incremental changes at district and facility-level services. This resulted in recurring challenges around expectations of the DE among stakeholders and its technical scope and limitations. Despite the continued focus on learning and adapting, USAID as well as implementers naturally gravitated to wanting evaluation-type ‘results’. This cultural shift took time and continued orientation to address. • The DE Tracker was instrumental in demonstrating the value of the continual learning cycle, however, it needed to be more accessible to the stakeholders. Future DE’s should consider a more sophisticated real-time and web-based dashboard to share these reflections with stakeholders. • Carrying out a DE requires significant commitment from many stakeholders to an extended process in quality and systems improvements. It was challenging to ensure that all stakeholders placed sufficient importance on the DE to optimize its findings. The DE team used dashboards, newsletters and short reports to try to ensure that stakeholders were aware of learning and upcoming actions, and created shareable slide decks for the USAID team members who could not attend regular meetings. The DE also concentrated as much action at the most local level, where stakeholders were very engaged. • Developmental evaluation was a new methodology and approach for many working in the health space. Locally, it was challenging to find staff with familiarity with collaborative learning and adaptation approaches in public health. CIRCLE hired a US-based technical advisor to train and mentor local staff in these techniques, but the scale required by the growing number of research requests from the DE team made it difficult to maintain the focus and context of the DE at times. USAID.GOV TANZANIA DE FINAL REPORT | 14 • The global outbreak of the COVID-19 pandemic impacted implementation of the DE. Due to travel restrictions the DE suspended routine visits and observations of health facilities supported under the Boresha Afya and suspended face-to-face meetings. Where possible activities were moved to virtual channels, including meetings, workshops, and data collection (phone interviews). RECOMMENDATIONS Lessons learned from implementing the DE can be applied to the design and implementation of future DEs. • Prioritize partner and donor engagement early in the process to foster donor and IP buy-in around shared priorities and objectives. • Include IPs at evaluation sites and in the inception process to strengthen partner buy-in and share information to inform the early DE design. • Build stakeholder capacity around CLA and complexity-aware methods prior to DE inception to support gradual cultural change. • Build a culture of adaptive management among stakeholders to support priority shifts, crisis management, and innovation opportunities. 15 | TANZANIA DE FINAL REPORT USAID.GOV FOR MORE INFORMATION Submission Date: February 28, 2022 Contract Number: AID-OAA-M-16-00006 COR: Sara Sulzbach, USAID DISCLAIMER This report was made possible by the support of the American People through the United States Agency for International Development (USAID) under the terms of the Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) contract AID-OAA-M-16-00006. CIRCLE is implemented by Social Solutions International, Inc. Views expressed are not necessarily of USAID or other affiliated institutions.