[Type the y name] Tanzania Vector Control Scale-Up Project Mid-Term Performance Evaluation March 19, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by the African Strategies for Health Project. EVALUATION REPORT This document can be located in the ASH website library at http://www.africanstrategies4health.org/resources.html. Documents are also made available through the Development Experience Clearing House (www.dec.org).Additional information can be obtained from: The African Strategies for Health Project 4301 N Fairfax Drive, Arlington, VA 22203 Telephone: +1-703-524-6575 AS4H-info@as4h.org www.africanstrategies4health.org This document was submitted by African Strategies for Health Project to the United States Agency for International Development under USAID Contract No. AID-OAA-C-11-00161 Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 i ACKNOWLEDGMENTS The evaluation team would like to thank those staff members of the National and Zanzibar Malaria Control Programmes who assisted in the evaluation process, especially Dr Renata Mandike and Mr Yusufu Mwita of the NMCP, the latter of whom accompanied the team on the fieldwork in Lake Zone, and Dr Abdullah Ally, Program Manager and Mr Juma Mchau of the ZMCP. The team additionally acknowledges the information and support provided by RTI International in Tanzania, especially the staff members of the Zanzibar, Mwanza and Kagera offices, who promptly and efficiently responded to team fieldwork schedule requests. The RTI Manager in Zanzibar, Dr Issa Garimo, and the Zonal Manager for Lake Zone, Mr Joshua Mutagahywa, in particular gave welcome inputs and the three drivers in Bukoba and Mwanza provided safe conduct and were good companions on the road. At the RTI International Dar es Salaam office, the team would especially like to thank the Chief of Party, Dr. Mahdi Mohamed Ramsan, the chief epidemiologist Dr. Jeremiah Ngondi and Mr Shabbir Lalji. Local government staff members at all levels in Zanzibar and in Lake Zone were helpful and informative during their participation in key informant interviews, for which the team is grateful. Health workers at the Unguja and Lake Zone facilities visited were similarly helpful. In addition, the team is grateful to the staff members of PMI/USAID and CDC Tanzania for orientation and discussion, especially Dr Jessica Kafuko, Ms. Chonge Kitojo and Mr. Moses Busiga, the latter of whom joined the team for fieldwork in Mwanza. Ms. Pamela Lema and Mr. Thabit of the Management Sciences for Health (MSH) Tanzania office also gave considerable and timely logistical support and transportation. Rudi Thetard, Milton D'Silva and Sunday Bitrus of the MSH African Strategies for Health office in Arlington VA similarly provided invaluable support and back-up. Last, but very much not least, the evaluation team expresses its sincere appreciation to community members and IRS spray operators on Unguja and in Lake Zone, without whom this mid-term evaluation report would not have been possible. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 ii ABBREVIATIONS & ACRONYMS ACT Artemisinin Combination Therapy ANC Antenatal Care BCC Behavior Change Communication CBO Community-based Organization CCA Community Change Agent (JHU COMMIT; also acting as IRS SIMs) CCHP Council Comprehensive Health Plan CDC Centers for Disease Control CHMT Council Health Management Team COMMIT Communication and Malaria Initiative in Tanzania (JHU program) CSO Civil Society Organization DED District Executive Director DHO District (Environmental) Health Officer DHMT District Health Management Team DHT District Health Team D IEC District IEC Officer DM&E District Monitoring and Evaluation Officer DMIFP District Malaria and IMCI Focal Person DMO District Medical Officer DMSO District Malaria Surveillance Officer (Zanzibar) DMT District Management Team EIA Environmental Impact Assessment FY Financial Year HMIS Health Management Information System IDSR Integrated Disease Surveillance and Response IEC Information, Education, Communication IMCI Integrated Management of Childhood Illnesses IPC Interpersonal Communication IPTp Intermittent Preventive Treatment in pregnancy IRS Indoor Residual Spraying ITN Insecticide Treated Net IVM Integrated Vector Management JHU-CCP Johns Hopkins University Center for Communication Programs LLIN Long-lasting Insecticidal Net MCH Maternal and Child Health MCN Malaria Case Notification M&E Monitoring and Evaluation Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 iii MEEDS Malaria Early Epidemic Detection System MEEWS Malaria Early Epidemic Warning System MIM Multilateral Initiative on Malaria MOHSW Ministry of Health and Social Welfare MOP Malaria Operational Plan mRDT Malaria Rapid Diagnostic Test MSH Management Sciences for Health NEMC National Environmental Management Council NIMR National Institute for Medical Research NMCP National Malaria Control Programme PEA Preliminary Environmental Assessment PERSUAP Pesticide Evaluation Report and Safer Use Action Plan PMI President's Malaria Initiative PPE Personal Protective Equipment PPP Public-Private Partnership RAS Regional Administrative Secretary RHO Regional (Environmental) Health Officer RMIFP Regional Malaria and IMCI Focal Person RMO Regional Medical Officer RTI Research Triangle Institute SEA Supplementary Environmental Assessment SIM Site IEC Mobilizer (COMMIT CCA volunteer) SOW Scope of Work THMIS Tanzania HIV/AIDS and Malaria Indicator Survey TPRI Tropical Pesticides Research Institute TVCSP Tanzania Vector Control Scale-up Project USAID United States Agency for International Development USG United States Government ZMCP Zanzibar Malaria Control Programme Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 iv CONTENTS ACKNOWLEDGMENTS.................................................................................................................................. 1 ABBREVIATIONS & ACRONYMS.................................................................................................................. 2 EXECUTIVE SUMMARY .................................................................................................................................. 1 1. Introduction to the Evaluation Report ................................................................................................. 13 1.1 Purpose of the Mid-term Performance Evaluation ..............................................................................................13 1.2 Evaluation Audience and Recipients ........................................................................................................................13 2. Background............................................................................................................................................... 13 2.1 Malaria in Mainland Tanzania and Zanzibar ...........................................................................................................13 2.2 The Tanzania Vector Control Scale-up Project 2010 - 2015 ...........................................................................14 3. Methodology............................................................................................................................................. 16 3.1 Definitions of a performance evaluation ................................................................................................................16 3.2 Evaluation sample .........................................................................................................................................................16 3.3 Evaluation limitations ...................................................................................................................................................17 4. TVCSP Progress to Date on Targets and Core Indicators ............................................................... 18 4.1 Progress on TVCSP targets and core indicators ................................................................................................. 18 4.2 Progress on non-core TVCSP indicators ...............................................................................................................22 5. TVCSP Evaluation Objectives and Questions ..................................................................................... 23 5.1 TVCSP achievements to date ....................................................................................................................................24 5.2 Progress in capacity building .....................................................................................................................................26 5.3 Perceptions of the TVCSP .........................................................................................................................................28 5.4 Attention to gender issues ........................................................................................................................................30 5.5 Environmental compliance .........................................................................................................................................32 5.6 Cost savings/analysis ....................................................................................................................................................34 5.7 IRS impacts on entomology and epidemiology .....................................................................................................38 5.8 TVCSP action on insecticide resistance .................................................................................................................46 5.9 Lessons learned ............................................................................................................................................................47 5.10 Potential improvements and future projects ........................................................................................................49 5.11 Sustainability ..................................................................................................................................................................51 6. Additional TVCSP components ............................................................................................................ 53 6.1 Cross-border collaboration ......................................................................................................................................53 6.2 Additional TVCSP area of focus: Behavior Change Communication ..............................................................54 ANNEXES......................................................................................................................................................... 56 Annex 1: Scope of Work ..........................................................................................................................................................57 Annex 2: People Met..................................................................................................................................................................71 Annex 3: TVCSP Mid-Term Performance Evaluation: Evaluation Plan ..........................................................................79 Annex 4: Evaluation Schedule and Sample ......................................................................................................................... 108 Annex 5: The Current Malaria Epidemic Detection and Response Situation in Mainland Tanzania .................... 112 Annex 6: References ............................................................................................................................................................... 115 Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 v This page is intentionally left blank Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 1 EXECUTIVE SUMMARY 1. Introduction This Executive Summary presents key findings, recommendations and key lessons learned from the mid-term performance evaluation of the Tanzania Vector Control Scale-up Project (TVCSP), fieldwork for which was conducted between mid-September and mid-October 2013. It also addresses potential areas of focus for a successor project. The Executive Summary should be read in conjunction with all sections in the body of the report: the Introduction and Background in sections 1 and 2; detailed discussion of Methodology, Evaluation Approach and Sample in section 3; and sections 4, 5 and 6, which present detailed discussion on Findings and Conclusions, provide Recommendations for all the TVCSP mid-term performance evaluation objectives and questions and contextualize lessons learned and consider the way forward. In addition, annexes 3 and 4 (the Evaluation Plan, Schedule and Sample) and 5 (A Detailed Overview of the Current Status of Epidemic Detection and Response in Mainland Tanzania) should be reviewed. At the express request of the Zanzibar Ministry of Health, where appropriate and necessary, this report has provided separate findings, conclusions and recommendations for Zanzibar and Mainland. 2. Methodology In accordance with USAID definitions of performance evaluations, this report focuses on descriptive and normative questions, including: what a particular program has achieved; how it is being implemented; how it is perceived and valued; and whether expected results are occurring. The TVCSP performance evaluation focuses on process and implementation fidelity as it relates to achieving program goals, objectives and activities as they were envisioned, in addition to looking at some intermediate outcomes. The report considers relevant results as of early September 2013. The evaluation process has been to critically examine each of the evaluation targets, core indicators, interim results/outcomes (as befits a performance evaluation), objectives and questions and to develop findings, conclusions and recommendations that are based on robust data collection, triangulation and close-grained analysis. Key informant interviews (KII) were conducted with all respondents except community members and spray operators, who participated in focus group discussions (FGD). Mixed FGDs were undertaken with spray operators, while separate FGDs were conducted with male and female community members. KII were conducted with a wide range of national, regional and district malaria stakeholders, with health workers, with USAID/PMI and CDC staff and with representatives of RTI working on TVCSP at national, zonal and regional levels. 3. TVCSP Goal and Objectives The TVCSP goal is to contribute to the reduction of the burden of malaria by interrupting malaria transmission through IRS and the prevention of adverse effects of malaria epidemics through blanket and focal spraying activities1 . 1 Targeted versus focal spraying. In the “attack phase” of malaria control, as programs scale up coverage to all malaria risk populations, IRS is conducted in targeted areas (determined through transmission ecology, malaria endemicity, cost, and logistics), usually for a period of three to five years. In the consolidation, maintenance, pre-elimination and elimination phases, IRS is used more selectively, based on malaria surveillance, and is focused towards locations where there are residual malaria foci and towards areas where there is re-establishment or a resurgence of transmission. World Health Organisation. Indoor Residual Spraying. An Operational Manual for Indoor Residual Spraying (IRS) for Malaria Transmission and Control and Elimination, 2013. www.who.int/iris/.../9789241505123_eng. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 2 The TVCSP originally addressed five objectives at inception in March 2010: 1. Scale up IRS in Tanzania by expanding IRS coverage from one region to six or more regions on Mainland and maintain required IRS services, as required, in Zanzibar. 2. Establish an epidemic detection and a focal spraying response strategy for Zanzibar and IRS districts on Mainland. 3. Develop an environmental compliance strategy and monitoring plan for Zanzibar and Mainland. 4. Develop capacities of ZMCP, NMCP, and districts to plan and implement IRS, including logistics planning, environmental compliance, and epidemic response through focal spraying. 5. Establish a viable and sustained entomological monitoring system on Mainland and Zanzibar. A sixth objective was subsequently added: Develop an epidemic surveillance system on Mainland and Zanzibar. 4. The Scope of the TVCSP Mid-Term Performance Evaluation As set out in the Scope of Work, the purpose of the TVCSP evaluation has been:  To determine the extent to which TVCSP objectives and key result areas are being met;  To document lessons learned to inform the second phase of the program; and  To assess the various approaches used by TVCSP to determine what is working to achieve the intended results. In total, ten evaluation objectives and seven evaluation questions have been addressed, as set out in the Scope of Work (SOW). These cover achievement of results, progress in capacity building, stakeholder perceptions of the TVCSP, gender issues, environmental compliance, cost savings and cost analysis, IRS impacts on entomology and epidemiology, insecticide resistance, lessons learned to date by the project and its malaria partners, potential improvements to TVCSP and potential foci for any follow-on project, and whether and how TVCSP is addressing sustainability. As requested by USAID Tanzania, the evaluation considered three issues in particular depth: environmental concerns; how capacity has been built; and whether/how the project and its partners and stakeholders are looking forward with regard to action on sustainability. 5. TVCSP Progress to Date on Targets and Core Indicators The finding of this evaluation, based on RTI data and validation through fieldwork interviews, is that there is achievement to date of the two TVCSP targets and five core indicators. The two TVCSP targets are: 1. Over 85% of structures in Indoor Residual Spraying (IRS) target areas sprayed 2. 85% (14,056,341 persons covered in the first two years of the activity) of the population protected through IRS in the targeted geographic area Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 3 The five core TVCSP indicators are: 1. Number of houses in the IRS program areas sprayed with residual insecticide in the last twelve months with USG funds 2. Proportion of houses in the IRS program target areas sprayed with residual insecticide in the last twelve months out of the total number of occupied houses in the IRS program target area 3. Proportion of population protected through IRS in targeted geographic area (number of people residing in houses sprayed/estimated total population in targeted district) 4. Number of people trained with USG funds to deliver IRS (disaggregated by sex) 5. Number of people reached with Information, Education, Communication (IEC)/Behavior Change Communication (BCC) activities for IRS. Overview of TVCSP Mid-Term achievement of targets and core indicators Year 1 Mainland Year 1 Zanzibar Year 2 ML Year 2 Z Year 3 ML Year 3 Z Target 1 (85% + structures sprayed) 95.2 91.8 93.2 94.6 Target 2 (85% + population protected) 92 93 95 95 Core indicator 1 (# houses sprayed) 1,144,880 192,320 1,314,852 114,852 Core indicator 2 (proportion of houses sprayed) 95.2 91.8 93.2 94.6 Core indicator 3 (proportion of pop. protected) 92% 93% 95% 95% Core indicator 4 (# men & women trained - all cadres) Men 6,000 Women 2,508 Men 40 Women 41 Men 6,387 Women 3,173 Men 1,185 Women 310 Men 6,131 Women 2,870 Men 758 Women 310 Core indicator 5 (# people reached with IRS BCC) 5,827,935 5,987,539 3,233,626 6. Evaluation objectives and questions: key findings and recommendations Key findings and recommendations for the ten evaluation objectives and seven evaluation questions are discussed in this part of the Executive Summary; as a number of objectives and questions are linked, each is not considered separately. For reasons of space, conclusions are not discussed here. For full and detailed consideration of all findings, conclusions and recommendations, see sections 4, 5 and 6. Recommendations only are provided for lessons learned, looking forward to a successor project and sustainability, as these are overarching and strategic issues. 6.1 Achievement of TVCSP results The ten expected TVCSP results (by end of project in March 2015) are: 1. Reduced malaria transmission in IRS areas 2. Reduced morbidity in IRS area 3. Increased geographical IRS coverage 4. Strengthened local capacity of NMCP and ZMCP to plan and conduct IRS, including focal spraying 5. Established system for epidemic response with focal spraying Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 4 6. Established and functional entomological laboratory on Mainland and quality control for entomological monitoring for Mainland and Zanzibar 7. Established environmental plan for IRS 8. Insecticide mitigation and management plan in place 9. Established epidemic surveillance system in Zanzibar and Mainland 10. Established viable sustainability plan for IRS A key finding of the mid-term performance evaluation is that implementation of TVCSP to date has resulted in considerable work toward achievement of expected results 1-8 and for 9 specific to Zanzibar. Particular lack of progress (not solely to be attributed to TVCSP) is noted for establishment of an epidemic surveillance system for Mainland and the establishment of a viable sustainability plan for IRS - here defined as a comprehensive system that would facilitate the human resource and technical management of spray operations, including environmental safety and other key IRS components and (eventual) financial independence of Tanzanian IRS operations, or greater financial contributions from national resources. A major achievement of TVCSP (building on the earlier IRS project implemented by RTI) is that IRS is acknowledged by malaria stakeholders, notably including community members, to have contributed to the overall reduction in malaria transmission and morbidity on Mainland and in Zanzibar. While community members might not be aware that a project named TVCSP has made such a contribution, all such evaluation respondents were united in their appreciation of the reduction in malaria transmission and morbidity that they have observed. These views were echoed by National and Zanzibar Malaria Control Programme and other regional and district malaria stakeholders. While reductions in malaria transmission and morbidity in IRS areas, as most recently evidenced by the findings of the 2011/2012 Tanzania HIV/AIDS and Malaria Indicator Survey, cannot be solely attributed to the activities of TVCSP, IRS has played a significant part in the achievement of such positive trends. 6.2 Progress in capacity building Key findings common to Zanzibar and Mainland  TVCSP has built capacity through training and mentoring of NMCP and ZMCP staff; community members’ knowledge of malaria prevention and treatment has been significantly strengthened.  A common general finding for both Zanzibar and Mainland is that local government staff in the districts visited unanimously described how because RTI works closely with district staff to plan, implement and evaluate IRS rounds, technical and management capacities have been developed for IRS.  There is a strong sense of engagement and ownership at district level of the project, as well as keen appreciation of its health gains and an understanding of the fragility of such gains. Key findings specific to Zanzibar  Zanzibar has considerable capacity for preparedness and ability to manage an independent IRS operation (the allocation of adequate financial resources is a separate issue), notably more so than Mainland. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 5 Key finding specific to Mainland  TVCSP has built capacity through training and mentoring of NMCP staff; community members’ knowledge of malaria prevention and treatment has been significantly strengthened.  There have been substantial inputs to IRS infrastructural developments. Key recommendation specific to Zanzibar  Zanzibar malaria stakeholders should now develop and institutionalize a capacity development plan specific to integrated malaria management and maintenance of (pre) elimination achievements. Key recommendation specific to Mainland  Mainland malaria stakeholders and TVCSP need to take urgent steps to strengthen capacity for epidemic detection and surveillance to contain outbreaks. 6.3 Perceptions of the TVCSP Key findings common to Zanzibar and Mainland  National, regional and district administrative and health workers all without exception perceive IRS as a significant and proven contributor to a reduction in malaria cases and welcome TVCSP spray operations.  100% of community members and spray operators interviewed for the TVCSP evaluation perceive IRS as a public good.  100% of community members and spray operators interviewed credit IRS as having made a major contribution to the reduction in malaria cases.  90% plus of all female community members and 50% plus of men cite infestations (bed bugs and fleas) as an increasingly unacceptable side effect of IRS.  The majority of community members and spray operators perceive the shift from blanket to targeted (in Lake Zone) and focal (in Zanzibar) spraying as unexplained, and of deep concern, due to fears of malaria resurgence.  A minority view (< 10% and expressed chiefly by male community members on Zanzibar) is that because IRS kills mosquitoes, there is no need to continue risk averse behavior, especially the use of nets. Key recommendations common to Zanzibar and Mainland  Building on the high regard in which IRS is held, RTI/TVCSP should continue advocacy for IRS succession planning, capitalizing on the good will generated by operations, capacity building and other inputs.  BCC materials and training should be developed to address infestation, reasons for shift to targeted spraying and the need to continue vigilance against malaria through use of nets, timely diagnosis, etc. This is essential, to guard against potential future community rejection/reduced acceptance and willingness to support IRS operations. 6.4 Gender Issues TVCSP focus on gender has dealt with considerations and issues relevant to its IRS operations; it has not worked on wider gender inequalities, as this is not the mandate of the project. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 6 Key findings common to Zanzibar and Mainland  TVCSP has addressed its project gender remit specific to gender considerations from two perspectives: (1) RTI/TVCSP has sought to apply a gender-equal capacity building and employment approach; (2) pre-IRS information is provided to both female and male household members, with attention to immediate safety considerations and protective benefits for all; however, discussion with JHU-trained Site IEC Mobilizers (SIMs) did not reveal any IRS-specific BCC on under-5 and pregnancy vulnerabilities to malaria, whether provided through interpersonal communication or more general messaging.  TVCSP data on people protected show benefits to all members of society, including those most vulnerable, i.e. the under-fives and pregnant women; pregnant spray operators have been reassigned to other IRS duties.  RTI/TVCSP has no dedicated gender focus staff member; neither the NMCP nor the ZMCP has a gender focal point/person; the same is true at regional and district levels. Key recommendations common to Zanzibar and Mainland  TVCSP gender-equal employment and training approaches and gains made should be vigilantly maintained; future TVCSP/successor project BCC planning and materials development should be informed by gender and social development expertise.  The final TVCSP evaluation should include gender expertise, so as to inform planning for any successor project; gender and social development expertise should be called down on a regular basis during the lifetime of a future project, so as to avoid delays in attending to community perceptions and any emerging gender-focused issues. 6.5 Environmental compliance Key findings common to Zanzibar and Mainland  TVCSP has supported effective, safe and secure Insecticide Storage Management; close and effective attention is paid to spray operators' and community members' health and safety.  Pre-, mid- and post- spray inspections are conducted in compliance with requirements to assure continued adherence to NEMC, USAID, WHO, and FAO standards.  Operational staging sites appear to be well-managed and maintained; solid waste disposal is now effectively managed through use of dedicated incinerators.  The yearly totals of destroyed sachets match with the number of sachets that were procured, indicating no loss and pilferage. Recommendation common to Zanzibar and Mainland  The absence of reports on insecticide loss, pilferage, misuse, poisoning and accidents indicates an efficient and effective process which TVCSP has put in place to address human and environmental safety of the program. The process should be maintained. 6.6 Cost savings/analysis Key findings specific to RTI  There is a need to track budget constraints vis-à-vis malaria risk management and to factor in a health economic perspective. Budget changes introduced by USAID need to be monitored in terms of forward planning for optimal IRS and malaria risk management. Since no project or program can expect unrestricted funding, it is important that until the end of the project RTI take every opportunity to factor in potential for optimal cost effectiveness. This should be a central feature of any future project planning.  There is a lack of/insufficient TVCSP/RTI consideration of IRS cost savings lessons elsewhere. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 7  There has been a degree of TVCSP focus on cost efficiencies; these must be considered in the proposed analysis. Finding common to Zanzibar and Mainland  The significance of contributions in kind must be acknowledged and factored into cost analyses. Key findings specific to Zanzibar  Advocacy/sensitization activities are being undertaken by RTI/TVCSP, in partnership with ZMCP, including attention to budget and cost benefits of IRS.  It is recognized that surveillance systems such as those currently operating on Zanzibar are extremely costly and may indeed present significant challenges with regard to any future government funding. Key findings specific to Mainland  Insecticide costs are perceived/described as insurmountable by district government stakeholders. No evidence was found of this view based on any cost analysis or any specific cost benefit analysis.  Financial allocation decision-making processes are complex. Districts have autonomy in allocation of monies for health budgets from the district basket fund and/or the national block grant, while the NMCP and regional structures have solely an advisory/advocacy role with regard to budget allocations for malaria control at district level.  No district administrator interviewed for the evaluation provided any information on his or her district having participated in development of the new Malaria Strategic Plan, currently in draft (and presumably including a budget?). Key recommendations common to Zanzibar and Mainland  RTI, PMI, USAID and other malaria stakeholders should closely monitor and report on the best balance between available funds and malaria risk management in the context of shifting from blanket to targeted (Mainland) and targeted to focal spraying (Zanzibar).  TVCSP should continue its advocacy on Mainland regarding inclusion of more substantial IRS financial commitments in Council Comprehensive Health Plans (CCHPs) and national health budgets.  The approaches currently employed in the debate with government on Zanzibar, by the ZMCP and RTI/TVCSP, should be considered for possible lesson learning and advocacy to Mainland government structures.  Districts should be supported to undertake local IRS cost analysis/budget item costing exercise as part of CCHP/Zanzibar District Health Teams DHT plans’ development.  TVCSP and IRS partners on Mainland should consider cost benefit analysis specific to longer term impacts of insecticide changes vis-à-vis potential resistance forestalling.  Contributions in kind and intangible benefits (including reduced burden of disease) should be factored into future TVCSP/IRS cost analysis.  RTI and its IRS partners should actively seek out comparative IRS interventions funded in part/wholly by government (e.g. Zimbabwe, Swaziland) and support Tanzania stakeholders’ lesson learning and succession/sustainability planning.  RTI and its IRS partners should be informed by the cost comparison exercise (PMI vis-à-vis government) conducted in Zambezia and Nampula provinces, Mozambique. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 8 6.7 IRS impacts on entomology and epidemiology 1. Entomology Key findings specific to Zanzibar  There is an effective vector surveillance system  Changes have been observed in vector composition  Insecticide resistance has been detected Key findings specific to Mainland  TVCSP engaged National Institute for Medical Research (NIMR)-Amani to conduct entomological surveillance in the Kagera, Mwanza and Mara as part of the nationwide surveillance.  RTI/TVCSP has taken steps to strengthen entomological surveillance in Lake Zone.  RTI has supported NIMR vector research in Lake Zone, which shows changes in dominance of vector species.  Insecticide resistance has been detected. Key recommendations common to Zanzibar and Mainland  In view of the limited information on shifts in behavior of the main malaria vector from indoor feeding to outdoor, TVCSP and ZMCP should consider conducting more studies on the feeding behavior of the malaria vector.  ZMCP should maintain insecticide resistance monitoring at fixed sentinel sites and strengthen insecticide resistance management through integrated vector management (IVM).  In view of the scarcity of information of the behavior of malaria vectors in the lake zone, TVCSP should conduct vector behavior studies in the lake regions. Bed bug and flea infestation Key findings common to Zanzibar and Mainland  Community members cite a marked increase in bed bugs (and to lesser extent fleas) and view this infestation as an increasingly unacceptable side effect of IRS.  Although there is no evidence that bed bugs are vectors of human disease, the issue of bed bugs and fleas in the Lake Zone districts requires urgent attention to avoid community resistance/refusal to have houses sprayed.  The presence of abundant bed bugs and fleas in sprayed houses which also have long￾lasting insecticidal nets (LLINs) suggests that the insects are resistant to pyrethroids. It is unclear whether the introduced bendiocarb and actellic will alleviate the problem of bed bugs and fleas. It is also common knowledge that bed bugs and fleas thrive in areas with poor buildings and hygiene standards. IRS is about controlling malaria, and its success or failure is dependent on community cooperation. Therefore, TVCSP and malaria partners need to address this issue. Key recommendations common to Zanzibar and Mainland  Although there is no evidence that bed bugs transmit diseases, TVCSP should address the issue to enhance community cooperation and acceptability of IRS.  TVCSP should intensify BCC/health education efforts on the objectives of IRS.  TVCSP should educate the communities on the importance of clean rooms, bedclothes etc.  TVCSP should be more proactive with regard to community members' concerns over infestations. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 9 2. Epidemiology Key findings specific to Zanzibar  Zanzibar is in a robust pre-elimination phase, to which IRS through TVCSP has contributed.  There is documented reduction in morbidity and mortality (THMIS 2012, MEEDS). TVCSP data (weekly MEEDS reports from Unguja and Pemba) show current higher levels of malaria cases in the >5 population, compared to under-fives.  An epidemic detection and response strategy has been successfully implemented in Zanzibar health facilities, covering all public primary health care facilities, hospitals and 20% of private health facilities. Surveillance activities use MEEDS.  TVCSP has supported ZMCP in producing maps showing malaria hot spots in Unguja and Pemba. These maps have been produced on a quarterly basis from July 2012 to date.  TVCSP has trained 10 District Malaria Surveillance Officers (DMSO), one for each of the 10 districts in Zanzibar. The DMSO structure is effective but workload per DMSO risks overload; RTI and ZMCP are addressing this issue.  TVCSP has been supporting response activities by providing technical and financial support through USAID/PMI and coordination of all response activities in high transmission sites.  The Malaria Early Epidemic Warning System (MEEWS) is work in progress. TVCSP has started receiving rainfall data from Tanzania Meteorological Agency.  There have been changes to the threshold criterion for determination of focal spraying. ZMCP respondents described how”due to reductions in the IRS budget” the threshold for deciding whether a community should receive IRS has gone up, from 2/1,000 population in 2012 to 7/1,000 in 2013. There are concerns that this change might be less than optimal, despite the undoubtedly impressive advances made in epidemic detection and surveillance. Key recommendations specific to Zanzibar  TVCSP should prioritize continued working with the ZMCP to develop and implement a sustained plan for malaria elimination.  Epidemiological profile research should be considered with a view to targeted interventions and BCC.  Attention should be given to the end of project to tracking threshold impacts, with consideration of review in any future project, should epidemiological indicators suggest this is advisable. Key findings specific to Mainland  IRS has contributed to the documented reduction in malaria prevalence. However, such gains are fragile. In 2011 and 2013 RTI/TVCSP supported NMCP investigations to verify reports of abnormal increases in malaria cases in Muleba District, Kagera Region. TVCSP noted a dramatic increase in the malaria positivity rate in May and June 2013.  There is no fully functional malaria epidemic detection system on the Mainland. The Electronic Integrated Disease Surveillance and Response (eIDSR) system is not yet effectively in place. Manual IDSR is partially functioning. IDSR paper tools have been developed but not yet fully distributed to all health facilities and training is not yet complete. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 10  Due to delays in implementing eIDSR, the NMCP, TVCSP and other partners plan to introduce MEEDS on Mainland. MEEDS is now being piloted in Muleba with the intention of it being scaled up progressively to other Kagera districts and other Lake Zone regions.  TVCSP activities have supported the development of malaria risk maps to ward level. These maps will be used to target epidemiological and entomological monitoring and focal spraying. The intention is to use the maps to define the districts with the highest risk of malaria and to apply focal spraying.  Availability of funds determines choice and number of communities to receive IRS, linked to epidemiological stratification and inclusion/exclusion criteria. TVCSP and its partners have had to balance finite funds with ensuring highest risk populations are protected. Malaria risk mapping is one aspect of this balancing. The intention is that lower risk yet still vulnerable populations will be supported through other aspects of integrated malaria control - specifically improved case management and surveillance. In this context the reported frequent stock outs of malaria Rapid Diagnostic Tests (mRDTs) in particular and the ongoing weaknesses in surveillance are of concern, as are reports of increased malaria positivity rates in parts of Lake Zone.  Draft Guidelines for Malaria Surveillance and Response have been developed by NMCP with the support of TVCSP. Key recommendations specific to Mainland  As a matter of urgency, all malaria stakeholders should continue to address how to ensure the Mainland epidemic detection system is functional and operational (including ensuring constant mRDTs supply).  All stakeholders should address the delayed implementation of eIDSR.  Attention should be given to the optimal Epidemic Detection System approach, e.g. MEEDS or more TVCSP support of IDSR.  Mainland should be informed by the Zanzibar approach to integrated malaria control, as transition from blanket to target IRS gathers pace.  Close attention should be given until the end of project, to the issue of budget constraints and overall epidemic detection and surveillance, to track whether the best balance is being achieved in terms of malaria risk management (such action is dependent on a range of stakeholders). See section 5.7 for discussion of focal spraying on Zanzibar and its epidemiological implications. 6.8 TVCSP action on insecticide resistance Key findings specific to Zanzibar  Insecticide resistance management systems are in place and appear effective.  There is increasing vector resistance, which ZMCP and RTI are addressing, through an insecticide resistance management strategy that involves rotation and assaying. Key findings specific to Mainland  Insecticide resistance management systems are in place and appear effective.  Vector resistance has been found. An insecticide mitigation plan has been developed, with three resistance management strategies: rotation of insecticides; monitoring of insecticide resistance at fixed sites; and avoiding IRS with pyrethroids. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 11 Key recommendations common to Zanzibar and Mainland  RTI and partners should finalize the insecticide resistance mitigation plan and its implementation plan.  TVCSP should ascertain vector composition and behavior in the Lake Zone IRS districts. 6.9 Key lessons learned to date  There is an imperative need to establish and sustain epidemic surveillance systems to support IRS.  It is better to take short-term action on setting up and sustaining surveillance systems, rather than awaiting optimal conditions.  Attention has been given by TVCSP to leveraging Public-Private Partnership opportunities; these should be expanded.  The importance of continuous attention to BCC cannot be overstated, so as to support community members' continued acceptance of IRS.  It is never too early to begin the development of an exit strategy and succession planning. 6.10 Looking forward to a successor IRS project  It cannot be reiterated often enough that one absolutely key activity for any future project is that it supports the existence of an effective and sustainable epidemic detection and response system.  There is a need for constant attention to how best to balance budget realities and optimal IRS.  Continuous monitoring of, and responsiveness to, community perceptions is essential with regard to malaria control, not least in terms of sustainability of gains in reduction of malaria cases.  Ensure that feedback of M&E and of other information to all levels of IRS partnership (down to health dispensary level) is effective and timely. 6.11 Sustainability Sustainability throughout the evaluation report refers to three broad topics: financial, epidemiological/technical sustainability, and community acceptance and support. Financial sustainability  RTI/TVCSP has been diligent in prompting discussion with national, regional and district authorities regarding sustainability; however, there is much still to be done, especially on Mainland.  Insecticide costs are considered on Mainland to present an insuperable barrier to government-funded IRS, it seems, without any cost analysis having been done at district level.  In order to promote understanding of how IRS costs can be sustained by government, or at least where government can take on more of the financial responsibility, RTI/TVCSP should be more proactive in providing more information on cost-savings and government￾funded IRS operations in other African countries. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 12 Epidemiological and technical sustainability  Yet again, it is clear that one absolutely key activity for any hope of sustainability is to support the existence of an effective and sustainable epidemic detection and response system.  Technical capacity has been developed to a standard able to manage and (it is hoped) sustain IRS, which augers well for sustainability; however, there needs to be more attention to the institutionalization of such technical capacity, through a health systems approach that factors in regular training for all cadres, a supportive supervision structure and quality assurance mechanisms, so as avoid loss of such expertise should key staff members be transferred. Clearly such technical expertise needs to be matched by an effective epidemic detection and response system.  Action to support retention of spray operators and other IRS workers is a strong component of working toward sustainability. Community acceptance and support of sustainability  Fully informing community members is essential for ensuring sustainability. This point refers to community concerns over bed bug and flea infestations, the shift from blanket to target or target to focal spraying and the potential implications for malaria control, and also to emerging, minority views on IRS obviating the need for continued use of LLINs and even possibly conferring some sort of immunity to infection. It is essential that community members' concerns be addressed throughout the lifetime of an intervention; perceptions will change over time and there may be different opinions. As has been seen in South Africa, communities can begin to reject IRS due to infestation. Sustainability both of IRS per se and wider gains in malaria control might be jeopardized should communities not be seen as full partners and stakeholders in IRS. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 13 1. Introduction to the Evaluation Report 1.1 Purpose of the Mid-term Performance Evaluation The TVCSP mid-term evaluation is a performance evaluation. Its overall purpose is: 1. To determine the extent to which the RTI Tanzania TVCSP objectives and key result areas are being met; 2. To document lessons learned to inform the second phase of the project; and 3. To assess the various approaches used by the TVCSP to determine what is working [in order] to achieve the intended results. The evaluation process has been asked to critically to examine each of the evaluation targets, core indicators, interim results/outcomes (as befits a performance evaluation), objectives and questions (see sections 4 and 5 below), developing findings, conclusions and recommendations that are based on robust data collection, triangulation and close-grained analysis. 1.2 Evaluation Audience and Recipients The intended audience for the evaluation report includes Mainland and Zanzibar Malaria Control Programme (MNCP and ZMCP) staff members, regional and district administrators and health authorities working on malaria and integrated disease management, health workers at facilities in IRS communities and districts, RTI staff members working on TVCSP and PMI, USAID and CDC Tanzania and Washington DC staff members. Additional recipients might include National Medical Research Institute (NMRI) entomologists, Tanzania partners working on malaria control, and development partners active in the malaria and maternal and child health (MCH) fields. 2. Background 2.1 Malaria in Mainland Tanzania and Zanzibar National aggregate malaria prevalence in Tanzania was 18% in 2007; it had reduced to 9% by 2011 (overall a 47% reduction). In Zanzibar malaria prevalence was 25% in 2006 and less than 1% by 2011. The 2011/2012 Tanzania HIV/AIDS Malaria indicator survey (THMIS) puts malaria prevalence at 9.5 % (9.7% for Mainland, 0.2% for Zanzibar). The malaria prevalence in Zanzibar reduced from 0.8 in the 2007/2008 THMIS to 0.2%, indicating 75% reduction in malaria prevalence. Prevalence rates in children 6-59 months reduced significantly in a number of Lake Zone regions between 2007/8 and 2011/12, both in those benefiting from IRS and others not: from 41.1% to 9% in Kagera Region; in Mwanza from 31.4% to 19% (both IRS regions); and in Simiyu from 29.5% to 3%. In Mara (an IRS region) prevalence in the same age group reduced from 30.3% to 26%. Despite such gains, in Mainland Tanzania malaria continues to be a significant cause of illness and death and an impediment to welfare and to socioeconomic growth. Zanzibar has previously twice controlled malaria, in the 1960s and 1990s, each time with large￾scale indoor residual spraying (IRS) operations, only for malaria to rebound. Currently in Zanzibar, malaria is controlled and not a major public health issue. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 14 The goal of the NMCP, as expressed in the Tanzania National Malaria Medium-term Strategic Plan 2008-2013, is to reduce the burden of malaria by 80%. The successor plan is in draft. The long-term goal of the ZMCP is to eliminate malaria throughout Zanzibar. THMIS 2011/12 data show that IRS is cited by 7% of Tanzanian women and 15% of men as the most common way to avoid malaria (the figures for women on Mainland are 6.3%, for men 14.4%, while those for Zanzibar are 20.1% and 25.1%). The higher percentages of men are noteworthy, in view of the evaluation finding that a small minority of men sampled view IRS as having removed mosquitoes and thereby the threat of malaria. The 2012 President's Malaria Initiative (PMI) Tanzania Malaria Operational Plan (MOP) notes that the Medium-term Strategic Plan 2008-2013 targets IRS in 60 of Mainland's 123 districts (48.5%) over a five-year period. Such scale-up has not been possible, primarily for financial reasons. PMI is currently the sole development partner contributing to IRS in Tanzania. The background to the Tanzania Vector Control Scale-up Project (the TVCSP) is that RTI provided assistance, funded by PMI, under the previous Malaria Control in Mainland Tanzania and Zanzibar project. That project ended in March 2010, with a number of activities under the 2010 implementation plan continuing until December 31, 2010. RTI has provided support to the ZMCP with its IRS program from 2006, contributing to a substantial drop in new malaria infections; part of its support was to facilitate the ZMCP in its establishment of a malaria early epidemic detection system (MEEDS). RTI began supporting IRS in Mainland Tanzania in 2007, assisting the Ministry of Health and Social Welfare (MoHSW) to control malaria outbreaks in Karagwe and Muleba districts in Kagera Region. In 2009, operations were scaled up to spray in the remaining stable and high transmission areas. 2.2 The Tanzania Vector Control Scale-up Project 2010 - 2015 Background The TVCSP is managed by RTI International and funded by USAID, to the total value of USD129,828,000; actual expenditures to end-September 2013 have been USD53,200,722, as against the co-agreement estimate of USD75.26 M. Year 1 of TVCSP ran for 18 months, between March 2010 and September 2011, so as to align the project financial year (FY) with the US government (USG) fiscal year. As mentioned in 2.1, a number of activities under the previous project continued until end-2010, so as to assure operational integrity. This TVCSP performance evaluation is being undertaken at the end of project year 3. RTI works in partnership with the NMCP and the ZMCP, with regional and district governments and health structures and with communities to implement IRS, entomological and environmental monitoring and epidemic malaria surveillance activities in Mainland and Zanzibar. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 15 On Mainland, TVCSP operates IRS in accordance with the objectives of the current Medium￾term Malaria Strategic Plan 2008-2013. Zanzibar’s pre-elimination strategy relies on continued universal and/or focal spraying, (operated by RTI/TVCSP in partnership with the ZMCP and other stakeholders) universal coverage of insecticide-treated nets (ITNs), and strong surveillance to maintain control and prevent a rebound of malaria. On Mainland, the project works in 18 districts in Lake Zone: 5 in (the old) Mwanza region, 6 in Mara and 7 in Kagera, while on Zanzibar 8 districts receive IRS, 6 on Unguja and 2 on Pemba. The TVCSP expanded IRS operations from Kagera to Mwanza and Mara regions in Lake Zone during 2010, undertaking blanket spraying. During the 2011/12 spray season the IRS strategy shifted from blanket to targeted spraying, beginning in Muleba and then in 2012/13 expanding to the remaining 17 IRS districts in Lake Zone. To date, Muleba and Karagwe districts have had six and five rounds of IRS, respectively. On Zanzibar 6 blanket spray rounds were conducted between 2006 and 2011, with focal spraying initiated under TVCSP in 2012. For discussion of planned and actual scale up, see 5.1 below. TVCSP goal and objectives The TVCSP goal is to contribute to the reduction of the burden of malaria by interrupting malaria transmission through IRS and the prevention of adverse effects of malaria epidemics through focal spraying. The TVCSP originally addressed five objectives at inception in March 2010: 1. Scale up IRS in Tanzania by expanding IRS coverage from one region to six or more regions on Mainland and maintain required IRS services as required in Zanzibar. 2. Establish an epidemic detection and a focal spraying response strategy for Zanzibar and IRS districts on Mainland. 3. Develop an environmental compliance strategy and monitoring plan for Zanzibar and Mainland. 4. Develop capacities of ZMCP, NMCP, and districts to plan and implement IRS, including logistics planning, environmental compliance, and epidemic response through focal spraying. 5. Establish a viable and sustained entomological monitoring system on Mainland and Zanzibar. A 6th objective was subsequently added: Develop an epidemic surveillance system on Mainland and Zanzibar. The TVCSP has seven areas of focus: IRS operations, environmental compliance, support to entomological monitoring systems, support to epidemiological surveillance and response, capacity building and sustainability, crosscutting activities (gender and cross-border collaboration) and monitoring and evaluation. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 16 TVCSP Behavior Change Communication inputs In TVCSP years 1 and 2 the JHU Communication and Malaria Initiative in Tanzania (COMMIT) project was the Behavior Change Communication (BCC) and community mobilization partner (see also 5.3 for discussion of BCC inputs) on Mainland. Civil society organizations mobilized communities before IRS, initially in Kagera and subsequently in Mwanza and Mara. This partnership ceased in 2012, due to several reported reasons: the sheer proliferation of CSOs and CBOs made it difficult to supervise and quality assure BCC activities; several of the CSOs were either national or international, had their own approaches and sometimes paid insufficient attention to local sociocultural mores; and cost considerations came increasingly into play. For further discussion, see 6.2 below. 3. Methodology For more extensive discussion of the methodology applied in this mid-term performance evaluation, please see Annex 3, the Evaluation Plan. A detailed definition of a performance evaluation as per the 2011 USAID Evaluation Policy is provided there. 3.1 Definitions of a performance evaluation USAID defines performance evaluations as focusing on ‘descriptive and normative questions: what a particular program or program has achieved; how it is being implemented; how it is perceived and valued; whether expected results are occurring and other questions that are pertinent to program design, management and operational decision-making’. Thus a performance evaluation focuses on process and implementation fidelity as it relates to achieving program goals, objectives and activities as they were envisioned, in addition to looking at some intermediate outcomes. 3.2 Evaluation sample Key informant interviews (KII) were conducted with all respondents except community members and spray operators, who participated in focus group discussions (FGD). Mixed FGDs (a total of 6) were undertaken with spray operators, while separate FGDs were conducted with male and female community members (a total of 12 FGDs, i.e. 6 with men and 6 with women). The evaluation team was able to conduct KII with a wide range of national, regional and district malaria stakeholders, with health workers, with USAID/PMI and CDC staff and with representatives of RTI working on TVCSP at national, zonal and regional levels. See Table 1 for further details. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 17 Table 1: Evaluation Sample TVCSP Evaluation: Sample Dar es Salaam Zanzibar (West & North B) Lake Zone (Kagera - Bukoba & Muleba & Mwanza - Magu & Misungwi) N/ZMCP officials ✓ ✓ Zanzibar MoH (Permanent Secretary & Director, Preventive Services) ✓ JHU- COMMIT ✓ RTI staff members ✓ ✓ ✓(Mwanza Zonal & Kagera Regional) USG staff members ✓ Regional authorities: RAS/RMO/RMIFP/RHO ✓ (Kagera & Mwanza) District authorities: Acting DED (Bukoba, Misungwi)/(Acting) DMO/ DMIFP/DVCO/ DHO/DIEC/DM&E ✓ ✓ DMSO ✓ Health Facility staff ✓x 2 HF ✓x 4 HF Spray Operators (58 in total: 28M/30F) ✓ x 2 (mixed groups) ✓x 4 (mixed groups) IRS Site Managers ✓ (1) ✓ (4 site managers) Community members (68 in total: 39M/29F) ✓ x 4 (2 M & 2 F) ✓ x 8 (4 M & 4 F) SIMs ✓ (2 SIMs) NIMR scientists & insectary staff ✓ (5 members of NIMR staff) 3.3 Evaluation limitations It was not possible on Mainland to meet a District Executive Director (DED); these local government officials have until recently led the process of developing Council Comprehensive Health Plans (CCHP), a role that has now been given to the District Medical Officer (DMO). While all efforts were made to reach local government authorities on Mainland, particularly Regional Administrative Secretaries, District Executive Directors (DED) and District Medical Officers (DMO), the evaluation team was on several occasions only able to speak to individuals who were acting in this capacity, e.g. acting DEDs in Bukoba Rural and Misungwi and the acting DMO in Misungwi. These discussions sought clarification on issues such as district planning and budget deliberations, engagement with the TVCSP, district contributions in cash or kind, and other evaluation foci. Inevitably it was not possible to cover topics in great depth with acting administrators, e.g. to discuss CCHP. A number of key RTI staff members have recently resigned from the project, such as the Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 18 former Lake Zone epidemiologist; therefore, while new staff members were helpful and informative, depth of institutional memory and completeness of TVCSP overview were not always present. Moreover, due to the timing of the evaluation, it was not possible to observe either pre-spray activities or a spray operation; assessment of the completeness and quality of previous TVCSP spray operations on Zanzibar and Lake Zone was achieved through detailed interviews, direct observation and document review. It is noted here that TVCSP, the NMCP and ZMCP, PMI and other malaria stakeholders have spray operation quality assurance procedures in place; the evaluation team could not make any assessment of those activities. Another impact of the evaluation timing was that a number of key stakeholders, e.g. the NMCP Programme Manager, were absent at the 6th Multilateral Initiative on Malaria (MIM) conference in South Africa. It has not been possible to view the new Mainland Malaria Strategic Plan, currently in draft and the replacement for the Medium-term Malaria Strategic Plan 2008-2013. The year 3 TVCSP Annual Progress Report was available only at the end of October, 2013; as a result, this performance evaluation has not been able to review its findings and overview. 4. TVCSP Progress to Date on Targets and Core Indicators In order to provide completeness of the mid-term performance evaluation, this section of the report addresses progress since the project start date in March 2010 to the completion of the in-country phase of the evaluation (early October 2013). 4.1 Progress on TVCSP targets and core indicators TVCSP targets are: 1. Over 85% of structures in IRS target areas sprayed (this is also a PMI Tanzania indicator) 2. 85% of the population protected through IRS in the targeted geographic area (the second PMI Tanzania indicator addressing IRS is: At least 85% of pregnant women and children under five will have slept under an ITN the previous night or in a house that has been sprayed with IRS in the last 12 months). There are five core indicators, as set out in the SOW and project documents: 1. Number of houses in the IRS program areas sprayed with residual insecticide in the last twelve months with USG funds 2. Proportion of houses in the IRS program target areas sprayed with residual insecticide in the last twelve months out of the total number of occupied houses in the IRS program target area 3. Proportion of population protected through IRS in targeted geographic area (number of people residing in houses sprayed/estimated total population in targeted district) 4. Number of people trained with USG funds to deliver IRS (disaggregated by sex) 5. Number of people reached with Information, Education, Communication (IEC)/Behavior Change Communication (BCC) activities for IRS For expected results/objectives, see section 5.1 below. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 19 Evidence from RTI data indicates achievement to date of all TVCSP targets and core indicators, as against percentages and proportions agreed with USAID and PMI Tanzania, unspecified numbers trained and reached, and targeted geographical areas and districts. Criteria and definitions agreed by RTI and USAID/PMI Tanzania are those applied in this mid-term performance evaluation. It should be noted that at the time of the evaluation complete TVCSP data were not available for all targets and core indicators specific to project year 3 (October 2012-end September 2013). Table 2: Overview of TVCSP mid-term achievement of targets and core indicators Year 1 Mainland Year 1 Zanzibar Year 2 ML Year 2 Z Year 3 ML Year 3 Z Target 1 (85% + structures sprayed) 95.2 91.8 93.2 94.6 Target 2 (85% + population protected) 92 93 95 95 Core indicator 1 (# houses sprayed) 1,144,880 192,320 1,314,852 114,852 Core indicator 2 (proportion of houses sprayed) 95.2 91.8 93.2 94.6 Core indicator 3 (proportion of pop. protected) 92% 93% 95% 95% Core indicator 4 (# men & women trained - all cadres) Men 6,000 Women 2,508 Men 40 Women 41 Men 6,387 Women 3,173 Men 6,387 Women 3,173 Men 6,131 Women 2,870 Men 758 Women 310 Core indicator 5 (# people reached with IRS BCC) *, ** 5,827,935 5,987,539 3,233,626 * Data provided by RTI are for Number of people reached with IEC/BCC for ITNs, IRS, IPTp, ACTs, RDTs (Mainland and Zanzibar), i.e. not solely for IRS Behavior Change Communication ** Aggregate data for Mainland and Zanzibar Table 3: TVCSP Y1 and Y2 progress: targets 1 and 2 and core indicators 1, 2, 3 Structures sprayed % structures sprayed Population protected (PP) % PP <5 protected Pregnant women protected > 5 protected Mainland Y1 1,144,880 95.2 6,305,683 92 1,397,621 187,318 4,720,744 Y2 1,314,852 93.2 6,520,353 93 1,412,579 184,780 4,922,994 Mainland Total Y1+2 2,459,732 12,826,036 2,810,200 372,098 9,643,738 Zanzibar Y1 192,320 91.8 643,648 95 120,005 15,295 534,328 Y2 114,852 94.6 586,657 95 103,983 14,080 468,594 Zanzibar Total Y1+2 307,172 1,230,305 223,988 29,375 1,002,922 National total Y1 & Y2 2,766,904 14,056,341 3,034,188 401,473 10,646,660 Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 20 The data in Table 3 are taken from the TVCSP Annual Performance Reports for years 1 (March 9, 2010-March 31, 2011) and 2 (dated October 1, 2011-September 30, 2012). Please note the following: as set out in Table 3, information is provided for each spray round on the total percentages of structures sprayed and population protected. These figures do not achieve 100%. Therefore, as an example, for TVCSP Y1 on Mainland, 92% of the population in the communities that received IRS were protected (the remaining 8% might have refused, been absent or other reasons might apply.These data have not been made available). TVCSP provides disaggregated data for three categories of people within the total number of population protected: under-fives, over fives and pregnant women. Thus the number of pregnant women protected in TVCSP Y1 in Mainland communities receiving IRS is given. As mentioned, because percentages do not achieve 100%, this number does not represent the total number of all pregnant women, but those who have been protected by IRS. Disaggregated data have not been made available as to the percentage of pregnant women, U5 and those aged 5 and above who have been protected in the IRS communities vis-à-vis those who have not. TVCSP data are available per district on aggregate numbers of people protected in the three categories. Given that percentages of people protected have not fallen below 92%, it may be assumed that a similarly high proportion of pregnant women have been protected. Table 4: TVCSP Y1, Y2 and Y3 progress: core indicator 4 Cadre Trained/Type of Training Year 1 Year 2 Year 3 Male Female Male Female Male Female Mainland Spray Operators Lake Zone 5116 2171 5, 363 2,744 5070 2436 Other training for IRS implementation (master trainers, trainers of trainers (ToTs), master store keepers, clinicians) 884 337 790 304 946 391 DEMOs' training on environmental compliance for IRS 0 0 16 2 0 0 Drivers from DAMCO trained on awareness and knowledge of handling and transporting IRS insecticides 0 0 6 0 9 0 RCH health workers, malaria focal persons at district and regional level trained on monitoring malaria parasitemia 0 0 32 53 0 0 Trainers of Trainers' IDSR training in Lake Zone 0 0 24 9 35 18 National IRS Core Facilitators 0 0 19 6 0 0 Journalists' orientation on IRS 0 0 17 10 0 0 SIMS 0 0 120 45 71 25 Total trained 6,000 2,508 6,387 3,173 6,131 2,870 Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 21 Cadre Trained/Type of Training Year 1 Year 2 Year 3 Male Female Male Female Male Female Zanzibar Health workers' and district representatives' training on part of the MEEDS scale-up (90-143 health facilities) 40 41 112 81 10 6 Meteorological officers' malaria elimination, surveillance and use of mobile phones 0 0 10 0 0 0 IRS ToTs 0 0 160 38 103 23 IRS Spray Operators 0 0 890 187 645 129 IRS Clinicians 0 0 4 3 0 0 DMSO training on application of RCD system in Zanzibar 0 0 9 1 0 0 Total trained 40 41 1,185 310 758 310 Please see sections 5.2 for discussion of the quality of capacity building and 5.4 for discussion of the relative number of men and women trained. Table 5: TVCSP Y1 and Y2 progress: core indicator 5 Number of people reached with IEC/BCC for ITNs, IRS, IPTp, ACTs, RDTs (Mainland and Zanzibar) Baseline Target Y1 Actual Y1 Target Y2 Actual Y2 Target Y3 Actual Y3 1,400,000 5,000,000 5,827,935 3,584,112 5,987,539 2,090,400 3,233,626 Aggregate for ML and Z Female: 3,124,337 Male: 2,863,202 TVCSP project data are not sex-disaggregated specific to people reached with messages solely about IRS; the project Performance Monitoring Plan (revised in December 2012 and approved by USAID Tanzania) provides sex-disaggregated data on the total number of men and women reached in FY 2011 by BCC/IEC on 'ITNS, IRS, IPTp, ACTs, RDTs'. Disaggregated data are available from TVCSP documentation for FY2011, as seen above in Table 5. Aggregate figures for FY 2010 and 2012 are also given. The evaluation team was told by USG Tanzania that the project proxy indicator used for TVCSP BCC reach is the percentage of people who refuse IRS in their houses. In this context it must be noted that the evaluation found widespread evidence of community members on both Mainland and Zanzibar who had no opportunity to refuse IRS, due to the power of community leadership structures and potential sanctions. This is not to suggest Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 22 that more people would necessarily have refused IRS, just that most may have little room for independent choice. Information from RTI is that TVCSP also measures reach by examining how many people aged 5 and over have received BCC on IRC (the project assumption being that people in that age range would understand such messages; it would be worthwhile investigating such assumptions). 4.2 Progress on non-core TVCSP indicators In addition to tracking core indicators during the IRS implementation, TVCSP monitored input, process and output non-core indicators in all thematic areas that included environmental compliance, entomology and epidemiology. These non-core indicators have been included to provide an overview of additional activities conducted by RTI and its malaria partners in work towards implementation of IRS operations that are compliant with environmental standards and supported by effective entomological interventions. Please also see detailed discussion in sections 5.5 and 5.7 below, on environmental compliance and entomological capacity. Table 6: Non-core indicators - targets and achievements Indicators Baseline FY 2010 Actual FY 2011 Target FY 2011 Actual FY 2012 Target FY 2012 Actual FY 2013 Target Number of Environmental Impact Assessment/Supplementary Environmental Assessment (EIA)/SEA carried out 1 1 0 0 0 0 0 Number of Environmental Compliance Monitoring & Mitigation Plans 0 1 0 0 0 0 0 Number of Environmental Compliance inspections conducted 0 6 8 3 3 6 6 Number of insecticide sachets/bottles used N/A 473,157 601,400 574,538 445,700 343,593 221,120 Number of empty sachets/bottles properly disposed of N/A 473,157 601,400 574,538 445,700 343,593 221,120 Number of new effluent waste disposal structures constructed 21 50 22 0 0 0 0 Number of new entomological surveillance sites established 7 12 6 16 17 0 0 Number of health staff trained in entomological monitoring at district and community level N/A 36 18 9 9 0 0 Number of entomological laboratories established 1 1 1 1 1 1 1 Number of insecticide resistance Mitigation Plans developed 0 1 1 1 1 1 1 Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 23 Environmental compliance: TVCSP monitored the number of Environmental Impact Assessments (EIA) conducted in new IRS districts, the number of environmental inspections carried out, the number of sachets used and empty sachets destroyed, the number of staging sites constructed for effluent disposal and the number of incinerators installed. In some cases TVCSP achievements exceeded targets. Of special note is the number of sachets used equaled exactly the number of empty sachets that were destroyed. This shows excellent stock management and accountability of the TVCSP staff. Mitigation Monitoring Plans were developed and implemented. Pre, mid and post environmental compliance inspections were conducted for every round of IRS. The number of staging sites TVCSP constructed exceeded the target. Entomology: TVCSP established two sentinel sites in each IRS district in the Lake Zone, supported Zanzibar to establish 21 sentinel sites, and established 21 sentinel sites in Mainland regions where IRS is not implemented. NIMR-Amani and NIMR-Mwanza use the sites for monitoring entomological indicators including insecticide resistance. TVCSP trained one vector control focal person in each Lake Zone IRS district and two mosquito collectors per sentinel site. Epidemiology: TVCSP set out to establish MEEDS in a number of districts, with staff trained in malaria surveillance and with trained surveillance and response teams. In Zanzibar, targets were achieved, while at the time of this mid-term performance evaluation there is no such achievement in the Lake Zone districts. See also 5.7 and Annex 5. One indicator currently not measured for TVCSP purposes, but which would be worthwhile to track in terms of capacity built, cost saving implications and potential for sustainability, is retention of spray operators. Information from RTI is that such data are not collected, because it is not a USAID/PMI reporting requirement. Evaluation findings from both Zanzibar and Mainland are that significant numbers of spray operators have now participated in several rounds in their local communities. 5. TVCSP Evaluation Objectives and Questions Introduction During discussion with USAID and PMI on Friday 20th September 2013, agreement was reached on two points: 1. The definition of 'each objective area' in the SOW was interpreted to mean that the evaluation team in its report addresses the evaluation questions as set out in the SOW, linking to the 6 TVCSP objectives and the evaluation objectives. All objectives and questions would be appropriately considered and discussed. 2. The evaluation team would consider three questions in depth: environmental concerns; how capacity has been built; and whether/how the project and its partners and stakeholders are looking forward with regard to action on sustainability. As expressly requested by the Principal Secretary and the Director, Directorate of Preventive Services of the Zanzibar Ministry of Health, this evaluation provides separate Zanzibar and Mainland findings, conclusions and recommendations wherever relevant in response evaluation questions and objectives. This approach is also in accordance with the PMI Malaria Operational Plan (MOP) 2013 (and earlier years' MOPS too), which notes that Mainland and Zanzibar have separate malaria control Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 24 programs; furthermore, each program has its independent arrangements with RTI and the TVCSP. A number of the evaluation questions, e.g. perceptions of IRS/TVCSP and gender issues, do not require separate Zanzibar and Mainland discussion. 5.1 TVCSP achievements to date This section discusses the evaluation objective that states: ‘Determine the extent to which the achievements have been made or not in relation to the expected results of RTI TVCSP since the beginning of the program’ and the evaluation questions: ‘What have been RTI TVCSP key achievements to date?’ and ‘What approaches has RTI used and what is working towards the achievement of the intended results and improved acceptability of IRS?’. This is a mid-term performance evaluation of the TVCSP; as such, its function is not to measure impacts or outcomes of the entire TVCSP, not least because the project does not close until March 2015. However, this evaluation will consider relevant and interim results to date, to early September 2013. Further discussion of separate expected results can be found in sections 5.2-5.11, in response to specific evaluation objectives and questions. Those expected results discussed here in 5.1 are the ones not covered directly by specific objectives and questions. Expected results todate The ten expected TVCSP results are: 1. Reduced malaria transmission in IRS areas 2. Reduced morbidity in IRS area 3. Increased geographical IRS coverage 4. Strengthened local capacity of NMCP and ZMCP to plan and conduct IRS, including focal spraying (see 5.2) 5. Established system for epidemic response with focal spraying (see 5.7) 6. Established and functional entomological laboratory on Mainland and quality control for entomological monitoring for Mainland and Zanzibar (see 5.7) 7. Established environmental plan for IRS (see 5.5) 8. Insecticide mitigation and management plan in place (see 5.8) 9. Established epidemic surveillance system in Zanzibar and Mainland (see 5.7 and Annex 5 specific to Mainland) 10. Established viable sustainability plan for IRS (see 5.11) Conclusions and recommendations are not provided here, as these are discussed in more depth for each of the evaluation questions and objectives that address the specific issues linked to expected results. A key finding is that implementation of TVCSP to date has resulted in considerable work to date toward achievement of expected results 1-8 and for 9 specific to Zanzibar. Particular lack of progress (not solely to be attributed to TVCSP) is noted for establishment of an epidemic surveillance system for Mainland (see especially 5.7 and Annex 5) and the establishment of a viable sustainability plan for IRS (here defined as a comprehensive system that would facilitate the human resource and technical management of spray operations, including environmental safety and other key IRS components and (eventual) financial Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 25 independence of Tanzanian IRS operations, or greater financial contributions from national resources. For further discussion of this issue, see section 5.11 in particular and also 5.2, 5.4 and 5.6. Interim results common to Mainland and Zanzibar Reduced malaria transmission in IRS areas and reduced morbidity in IRS area (hospital attendances, admissions, MPR). A major achievement is that IRS is acknowledged by malaria stakeholders, notably including community members, to have contributed to the overall reduction in malaria transmission morbidity on Mainland and in Zanzibar. While community members might not be aware that a project named TVCSP has made such a contribution, all such evaluation respondents were united in their appreciation of the reduction in malaria transmission and morbidity that they have observed. These views were echoed by National and Zanzibar Malaria Control Programme and other regional and district malaria stakeholders. While reductions in IRS areas in malaria transmission and morbidity, as most recently evidenced by the findings of the 2011/2012 Tanzania HIV/AIDS and Malaria Indicator Survey, cannot be solely attributed to the activities of TVCSP, IRS has played a significant part in such positive trends. See section 2.1 for detailed discussion of THMIS data showing reductions in malaria prevalence on Mainland and Zanzibar. Interim result specific to Zanzibar Geographical IRS coverage Evidence from this evaluation indicates that in both Zanzibar and Mainland, due to reduced funding for TVCSP, targeted or focal IRS became the only realistic option and increased geographical coverage became unfeasible. The challenge has been how most effectively to tailor such operations based on the best epidemiological data. Information from RTI is that data were collected from health facilities on Zanzibar and subsequently on Mainland. Malaria morbidity was ranked and a cut-off applied according to the number of structures that could be sprayed with the funds available. Therefore, availability of funds calibrated through epidemiological data dictated, at least in part, the number of communities to be sprayed and the threshold (risk stratum) to be applied. Interim result specific to Mainland Increased geographical IRS coverage See also above for common findings. The original intention was that the TVCSP would scale up after year 2 to a total of six mainland regions (adding Mtwara, Lindi and Ruvuma). This has not occurred with the stated reason being financial constraints. The strategic shift to targeted spraying in Kagera, Mwanza and Mara regions was influenced primarily by cost imperatives, partially by reductions in malaria cases, as evidenced by health facility epidemiological data as well as the THMIS 2011/12, and also by reported universal mosquito net coverage of ca 80% (use estimated at ca 60%). Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 26 5.2 Progress in capacity building Discussion here is centered on the evaluation objective: Determine progress made in the capacity building and institutional strengthening of the Ministry of Health and Social Welfare, NMCP and Zanzibar MOH ZMCP. This evaluation objective responds specifically to the fourth TVCSP component: Develop capacities of ZMCP, NMCP and districts to plan and implement IRS, including logistics planning, environmental compliance and epidemic response through focal spraying. See also 5.4 below, the section on gender issues. Capacity building represents one of the three areas for additional attention as requested by USAID Tanzania. Background The RTI TVCSP Technical Application (2009) details its proposed support to capacity building and institutional strengthening under objective number 4. Recognizing the existence of capacity weaknesses both in Zanzibar and Mainland and the commitment needed to ensure that gains made by TVCSP be maintained once the project closes, RTI proposed to pursue a two-pronged approach to building capacity: 1) focusing on capacity of districts to plan and implement IRS campaigns; and 2) focusing on capacity and commitment at the national level. At the district level, a sustainability gap analysis was to inform a tailored approach to building district-level technical, managerial, and logistic skills for IRS implementation. As part of its proposed exit strategy, TVCSP planned to build NCMP and ZMCP capacity to take over these functions in new districts, according to project roll-out strategies for IRS. The project was also to provide technical assistance to districts where vector control strategies were to evolve from blanket to focal spraying. At the national level, TVCSP planned to establish an IRS cell under the Vector Control Unit in NMCP, and to strengthen strategic and implementation planning capacities of the NMCP and ZMCP. NMCP and ZMCP staff members would be mentored in IRS technical and logistics skills and TVCSP would fully support participation of NMCP/ZMCP staff in the planning and implementation of BCC activities in partnership with the JHU-CCP COMMIT Project. Findings common to Zanzibar and Mainland A common general finding for both Zanzibar and Mainland is that local government staff in the districts visited during the evaluation unanimously described how because RTI works closely with district staff to plan, implement and evaluate IRS rounds, technical and management capacities have been developed for IRS. There is a strong sense of engagement and ownership at district level of the project, as well as keen appreciation of its health gains and an understanding of the fragility of such gains. An evaluation finding is that for a number of reasons, Zanzibar has considerable capacity for preparedness and ability to manage an independent IRS operation (the allocation of adequate financial resources is a separate issue). Findings specific to Zanzibar Substantial progress in capacity building: Evaluation interviews and TVCSP documentation show that considerable progress has been made to date in training and mentoring individuals and groups from the national to the district levels (see Table 4 in section 4.1 for a summary of numbers trained in Zanzibar by cadre). ZMCP, district level staff and health facility staff all agree that their technical capacity for epidemic detection and surveillance, IRS planning and Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 27 implementation has been strengthened by TVCSP. The ZMCP Program Manager expressed some measure of confidence on Zanzibar’s ability to sustain the project beyond the life of the TVCSP. TVCSP has developed comprehensive training manuals informed by needs assessments and IRS manuals for planning and implementation as part of its capacity building approach. TVCSP has built capacity through training and mentoring of ZMCP staff such as the M&E and BCC teams, and training of core IRS facilitators, district staff including district surveillance officers and over a thousand spray operators. TVSCP has significantly enhanced infrastructural capacity for IRS and related activities through development of IRS operation sites. Community knowledge: Community members’ knowledge of malaria prevention and treatment has been significantly strengthened - this applies equally on Mainland. Conclusions specific to Zanzibar  TVCSP has provided substantial capacity building support at ZMCP, district and community levels.  ZMCP has demonstrable capacity to manage its vertical malaria program. Recommendations specific to Zanzibar  Zanzibar malaria stakeholders should now develop and institutionalize a capacity development plan specific to integrated malaria management and maintenance of (pre) elimination achievements. Findings specific to Mainland Substantial progress in capacity building: Key informants interviewed and TVCSP documentation indicate that substantial progress has been made in training and mentoring individuals from national to district levels, as is the case also for Zanzibar. National, regional and district level authorities as well as health workers all without exception agreed that TVCSP has provided them with technical and managerial capacity to implement IRS. While CCHP documentation was examined in Misungwi district showing 2013 financial inputs (extremely limited) to IRS operations, none of the national, regional and district level personnel interviewed feel they are ready to take on the financial responsibility for IRS operations on a large scale. Strengthened local capacity of NMCP and district administrators to plan and conduct IRS, including focal spraying Local government staff in the 4 Lake Zone districts visited unanimously described how because RTI works closely with district staff to plan, implement and evaluate IRS rounds, technical and management capacities have been developed for IRS. There is a strong sense of engagement and ownership at district level of the project, as well as keen appreciation of its health gains and an understanding of the fragility of such gains. Technical capacity development has to date not been tested in isolation from TVCSP support. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 28 TVCSP signed Memoranda of Understanding with all districts and updated them in year 2, detailing the roles of the districts in IRS operations. To date TVCSP has implemented a comprehensive training program. 10,000 + individuals trained in IRS implementation including Master Trainers, Trainers of Trainers and a total of 8380 male and female spray operators have been trained in Lake Zone regions. TVCSP has developed detailed manuals and SOPs to guide IRS planning and implementation informed by initial needs assessments. In addition, TVCSP has supported NIMR in strengthening its entomological monitoring capacity. Substantial inputs to infrastructural development specific to IRS: TVCSP has provided substantial infrastructure inputs to Lake Zone regions and districts for IRS implementation including the building of several IRS operation sites in each district of implementation and installing an incinerator in Mwanza. In the expectation of a marked decrease in malaria cases resulting from the range of interventions implemented in the Lake regions, TVCSP planned to strengthen malaria case reporting through the integrated disease surveillance response (IDSR) system, in accordance with NMCP and MoHSW plans. District personnel interviewed in the lake regions note that cases are increasing in villages not covered during targeted spraying. Implementation of the IDSR has proved to be a challenge and leaves districts implementing IRS vulnerable to outbreaks. For detailed discussion, see section 5.7. Conclusions specific to Mainland  Regions and districts have benefited from TVCSP infrastructural capacity support for IRS implementation.  Districts are able to mobilize communities to participate in IRS using SIMs and community leaders.  Districts’ technical capacity for IRS and malaria management has been enhanced and is effective.  Community members’ knowledge of malaria prevention and treatment has been significantly strengthened.  Weak epidemic detection and surveillance systems represent a major area of weakness that hampers efficient and timely management of malaria. Recommendation specific to Mainland  Mainland malaria stakeholders and TVCSP need to take urgent steps to strengthen capacity for epidemic detection and surveillance to contain further outbreaks such as happened in Muleba. 5.3 Perceptions of the TVCSP This section considers the evaluation objective that states: ‘Establish how the project is perceived by the NMCP, ZMCP, local governments and the community’ and the evaluation question: ‘How is the project perceived and valued by the United Republic of Tanzania (MOHSW on Mainland, MOH in Zanzibar, NMCP, ZMCP, local governments in IRS areas of operation) and key malaria stakeholders in Tanzania?’. All evaluation key informants and focus group discussion participants were asked about their perceptions of TVCSP, so as to enable a comprehensive overview. Findings were predominantly the same for both Zanzibar and Mainland; where different perceptions were found, these are discussed. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 29 Findings common to Zanzibar and Mainland Administrators' and health workers' overall perceptions: National, regional and district administrative and health workers all without exception perceive IRS as a significant and proven contributor to a reduction in malaria cases and welcome TVCSP spray operations. Similarly, national, regional and district administrative and health workers welcome and acknowledge TVCSP's contributions to capacity building and technical expertise. The project is perceived as having enhanced human resource capacity. Community members' and spray operators' perceptions of IRS as a public good: 100% of community members interviewed perceive IRS as a public good and welcome its contributions (none mentioned TVCSP). Similarly, 100% of community members and spray operators credit IRS as having made a major contribution to the discernible reduction in malaria deaths. Bed bug and flea infestation: 90% plus of all female community members and 50% plus of men cite infestations of bed bugs (and to a lesser extent fleas) as an increasingly unacceptable side effect of IRS. None of the community members was able to describe any BCC being provided from any source (e.g. JHU COMMIT volunteers, called Site IEC Mobilizers (SIMs) by TVCSP, community leaders or spray operators) that provided information on how to deal with infestations; the SIMs interviewed made no such reference. Information from RTI is that such BCC has been provided, that high IRS acceptance rates indicate infestation is not a major issue and the 'myth' has been dispelled. However, fieldwork findings are that infestation is a major concern (see the example of South Africa, where IRS was almost halted due to infestation challenges). In this context the evaluation finding that most community members have little opportunity to make an independent choice as to whether or not to accept IRS needs to be considered; while evaluation and other evidence indicates that acceptance rates would, in all probability, continue to be high, no direct correlation should be claimed between those rates and the relative seriousness of community members' views on the need to address infestation. Furthermore, Tanzania-specific studies on bedbug infestation and insecticide resistance have been available to RTI for several years. 2 See section 5.8 for further discussion. ”If bed bugs caused malaria, all my children would be dead” “I yearn for a good night's sleep” Female community members, Misungwi district, Mwanza Concerns over the shift to targeted/focal spraying: The majority of community members, including leaders, and spray operators perceive the shift from blanket to targeted (in Lake Zone) and focal (in Zanzibar) spraying as unexplained and of deep concern, due to fears of malaria resurgence. 2 See e.g. Kweka et al 2009 for a discussion of bedbug infestation in Tanzania. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 30 Views on malaria risk: A minority view (< 10% and expressed chiefly by male community members on Zanzibar) is that because IRS kills mosquitoes, there is no need to continue risk averse behavior, especially the use of nets. Concern was expressed by ZMCP staff that this perception of immunity might increase. More than 80% of all female community members on both Zanzibar and Mainland indicate that post-IRS is precisely the time to continue the use of nets, to reinforce its positive impacts. 3 Findings specific to Zanzibar  Partnership coordination is a major focus of TVCSP on Zanzibar. The TVCSP office is in the ZMCP building and weekly joint meetings are held; arrangements are considered effective by both parties. This close relationship is considered by ZMCP to have contributed substantially to its evolving ownership of IRS and its capacity to advocate for increased funding from government. Conclusions common to Zanzibar and Mainland  IRS is acknowledged by regional and district administrators and health workers as a significant contributor to effective malaria control.  Community members hold IRS in deep regard as having reduced malaria infection and deaths.  TVCSP is perceived as having fostered and supported general sense of ownership at all levels.  TVCSP is perceived as having significantly contributed to technical malaria expertise at all government levels.  TVCSP is perceived as having significantly supported infrastructural development for IRS.  IRS stakeholders are perceived as having paid insufficient attention to increasing community concerns over infestations, types of spraying and false perceptions of lower risk due to IRS. Recommendations common to Zanzibar and Mainland  Building on the high regard in which IRS is held, RTI/TVCSP should continue advocacy for IRS succession planning, capitalizing on the good will generated by operations, capacity building and other inputs.  BCC materials and training (again to be supported by JHU?) should be developed to address infestation, reasons for shift to targeted spraying and the need to continue vigilance against malaria through use of nets, timely diagnosis, etc. This is essential so as to guard against potential future community rejection/reduced acceptance and willingness to support IRS operations. 5.4 Attention to gender issues The focus here is on the evaluation objective: ‘Determine the ways in which RTI TVCSP is addressing gender gaps’ and the evaluation question: ‘In what ways has [the] RTI project integrated gender considerations into its activities and how have gender issues been addressed in the implementation of the project?’. These points also link into the capacity building question, in 5.2 above. All discussion in this section of the report acknowledges socio-cultural aspects of Zanzibar and Mainland gender roles and relations are beyond the scope and capacity 3 For further discussion of Zanzibar community members' perceptions of risk, see Bauch et al 2013. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 31 of TVCSP, or any one project or program, to address in their entirety. Such as it is, TVCSP focus on gender has addressed considerations and issues relevant to its IRS operations; it has not addressed wider gender inequalities, as this is not the mandate of the project. Findings common to Zanzibar and Mainland TVCSP has addressed its project gender remit specific to gender considerations from two perspectives. Firstly, RTI/TVCSP has sought to apply a gender-equal capacity building and employment approach, e.g. enabling women to be trained and employed (and retained) as spray operators, working with male and female SIMs, making training open to both men and women and engaging with male and female district and community administrators in equal measure (again, within the wider realities of a less than gender-equal social structure). See Table 4 in 4.1 for sex-disaggregation of the cadres and numbers trained by the project. The information from RTI stakeholders is that the numbers of women and men trained are representative of the gender realities of Mainland and Zanzibar society, in that more men than women currently can and have fulfilled the various educational and other criteria necessary to participate in training and have capacity built by TVCSP. This position was echoed by NMCP and ZMCP and by regional and district stakeholders. Thus the perception is that TVCSP has worked to achieve gender parity, again within the confines of its specific remit. Secondly, pre-IRS information is provided to both female and male household members, with attention to immediate safety considerations and protective benefits for all. TVCSP data on people protected show benefits to all members of society, including those most vulnerable, i.e. the under-fives and pregnant women; as such it can be deemed gender-balanced within its remit. Please see discussion above in section 4 with regard to the (disaggregated) number of people protected, including pregnant women and the (aggregate) data on people reached with IRS and other malaria-related BCC. TVCSP has applied a gender-balanced approach to employment and capacity building: It has employed female spray operators and IRS ancillary staff, and training at all levels has been equally open to men and women, from the side of TVCSP. Pregnant spray operators have been reassigned to other IRS duties. Specific gender expertise: RTI/TVCSP has no dedicated gender focus staff member. Neither the NMCP nor the ZMCP has a gender focal point/person; the same is true at regional and district levels (whether specifically for malaria, for IDSR or for wider gender and health issues). The absence of such expertise may have contributed to relative TVCSP and partner inattention to potential emerging BCC issues, e.g. infestations and possible changes in the epidemiological profile (such as vectors shifting outdoors; men becoming potentially more vulnerable during evenings and women during early morning hours). Discussion with JHU-trained SIMs did not reveal any IRS-specific BCC on under-5 and pregnancy vulnerabilities to malaria, whether provided through interpersonal communication (IPC) or more general messaging. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 32 Conclusions common to Zanzibar and Mainland  TVCSP has been diligent in supporting gender-balanced employment and training opportunities.  Government structures on both Zanzibar and Mainland appear open to equal opportunities for employment and training.  There is scope for RTI and its malaria partners (including USAID/PMI) to focus more closely on emerging gender-specific issues linked to malaria and IDSR.  There is scope for more gender-specific TVCSP BCC messaging/IPC training, e.g. on changing vector behavior and potential gendered vulnerabilities (such as men socializing late into the evening, women rising early).4 Recommendations common to Zanzibar and Mainland  TVCSP gender-equal employment and training approaches and gains made should be vigilantly maintained.  Future TVCSP/successor project BCC planning and materials development should be informed by gender and social development expertise.  The final TVCSP evaluation should include gender expertise, so as to inform planning for any successor project.  Gender and social development expertise should be called down on a regular basis during the lifetime of a future project, so as to avoid delays in attending to community perceptions and any emerging gender-focused issues.  Attention should be given to the benefits of TVCSP/future projects' greater linking into support for increased IPTp and infant treatment and to gendered patterns of behavior and potential links to (relative) vulnerability; both are relevant as IRS becomes targeted/focal (NB: also issues of cost effectiveness). 5.5 Environmental compliance This section addresses the evaluation objective that states: ‘Assess the processes put in place to address human and environmental safety of the program [TVCSP], in relation to US government and local environmental laws’. It also directly considers TVCSP component three: ‘Develop an environmental compliance strategy and monitoring plan for Zanzibar and Mainland’. As mentioned in the introduction to section 5, environmental compliance is one of the three areas that the evaluation team was requested to review in particular detail; discussion below reflects this. Background The safety of the population from insecticide poisoning and the prevention of unnecessary contamination of the environment are crucial for any program that uses pesticides. A spraying operation generates liquid and solid waste that requires appropriate disposal to prevent environmental contamination. This section addresses the findings on the processes TVCSP put in place to address human and environmental safety of the program. 4 While out of the specific scope of this evaluation, there does appear also to be scope also for more focused gender attention at N/ZMCP, e.g. in the BCC departments. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 33 Findings common to Zanzibar and Mainland Insecticide Storage Management: To ensure safety and security of IRS commodities including insecticides, TVCSP has established a network of storage facilities at regional, district and operational sites. There are 4 regional warehouses at Mwanza, Bukoba, Musoma and Zanzibar, 28 district stores (18 on Mainland and 10 in Zanzibar) and 190 temporary sub-stores at operational sites (180 on Mainland and 10 in Zanzibar). The stores at Bukoba, Musoma and Zanzibar and the 28 district stores are government stores. The TVCSP pays rent only for the central warehouse in Mwanza. According to the six store keepers and logisticians interviewed for the evaluation, TVCSP has put in place best practices for stock management that uses stores records such as bin cards, store ledgers, vouchers complimented by weekly trackers, physical counts and regular audits. The system prevents pesticide pilferage and potential misuse and promotes safer use of pesticide to protect the environment and human health. Spray operators' and communities' health safety: In fulfillment of the Tanzania Occupational Safety and Health Act 2003 and WHO Health and Safety Regulations, TVCSP provides full personal protective equipment (PPE) and enforces the use and care of the equipment through training and close supervision. In six FGDs (of 6-13 mixed female and male spray operators each), in Zanzibar, Kagera and Mwanza, all respondents described the proper use and care of PPE. All 29 female spray operators interviewed were tested for pregnancy prior to operations. TVCSP uses spray operators and JHU-trained COMMIT community mobilizers (the SIMs) to give resident safety information before and after spraying. TVCSP trains District Medical Officers who in turn train nurses in emergency management of insecticide poisoning (however, health facility staff interviewed in West District on Unguja and in Misungwi, Mwanza, were unable to describe the correct procedures for dealing with such cases). Pre, mid and post spray inspections: In compliance with requirements in the EIA and SEA, the TVCSP engages environmental inspectors to conduct pre-, mid and post-spray environmental compliance inspection to assess continued adherence to NEMC, USAID, WHO, and FAO standards throughout IRS implementation. The inspectors monitor storage and warehouse management practices, field operations and effluent waste disposal structures and sanitary facilities and produce environmental compliance inspection reports which are given to TVCSP for action. Post-spray environmental compliance inspections look at decontamination and decommissioning of all IRS sites to make them safe. Operational staging sites: IRS operations generate liquid waste resulting from washing pumps, PPE and bathing that requires appropriate disposal to prevent environmental contamination. In 2010 and 2011, TVCSP constructed 72 new effluent disposal facilities in the Lake Zone IRS districts. Four operational sites were visited: all contained storage facilities, soak pits, washing area slabs, and bathing shelters for spray operators. There were no activities at the sites because there were no spray operations at the time of the evaluation. The four site managers at the four sites understood their roles and responsibilities in the management of effluents. All spray operators interviewed for the evaluation described understood the proper Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 34 use and care of pumps including triple rinsing, inspection of pumps for leakage, disposal of washouts, care of solid waste and proper use of PPE. Solid Waste Disposal: TVCSP experienced challenges of identifying suitable incineration facilities for solid waste during the first one and half years of project. By mid-2011, two steel mill plants, Sayona and Nyakato, both located in Mwanza, were identified and recommended by NEMC for disposal of contaminated pesticide waste. The use of steel mills ended in 2013 when RTI procured new incinerators that were installed at Kivunge Hospital in North A District in Zanzibar and at the Nyanguge Health Centre, Magu district in Mwanza region. A total of 473,157; 574,538; 343,593 and 221,120 empty sachets were incinerated in 2010, 2011, 2012, and 2013, respectively. The yearly totals of destroyed sachets match with the number of sachets that were procured, indicating no loss and pilferage. Findings specific to Mainland TVCSP compliance structures: In fulfillment of the national Regulation 26 of the Environmental Impact Assessment and Audit Regulations, 2005 and USAID 22 CFR 216.6, TVCSP conducted an EIA and SEA in Mwanza and Mara, and NEMC approved the documents. The approved documents guided the TVCSP to develop an Environment Monitoring Mitigation Plan, which TVCSP uses to put all mitigation measures in place before, during and after spraying. It is also a national requirement by TPRI ACT 1979, that all insecticides to be imported and used in IRS should be registered by TPRI. TVCSP received TPRI certificates to import and use lambdacyhalothrin, bendiocab and recently actellic. Limitations: There were no spraying operations at the time of the field visits. All visited operational sites were decommissioned and only empty stores, washing slabs, soaking pits, washrooms and toilets were on site. Conclusion common to Zanzibar and Mainland  TVCSP is implementing an effective environmentally compliant IRS that satisfies international and national regulations and ensures human and environmental health and safety. Recommendation common to Zanzibar and Mainland  The absence of reports on insecticide loss, pilferage, misuse, poisoning and accidents indicates an efficient and effective process which TVCSP has put in place to address human and environmental safety of the program. The process should be maintained. 5.6 Cost savings/analysis The evaluation objective: ‘Describe any cost savings and any interventions made to reduce the cost of the structures sprayed and people protected’ is considered here. This objective is discussed both from the Mainland and Zanzibar government perspectives as well as from the position of RTI as managing TVCSP. This section should be read in conjunction with 5.9 (lessons learned), 5.10 (future projects) and 5.11 (sustainability). Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 35 Background TVCSP has conducted two cost analyses of its IRS interventions; the first (prepared in 2011) was of 2008 and 2009 costs associated with IRS conducted under the earlier (2007-2010) RTI￾managed Malaria Control in Mainland Tanzania and Zanzibar project, while the second (published in 2012) was of the 2010 TVCSP spray round. As such, those data are historic and cannot be considered to represent the current situation with regard to TVCSP. The intention is that there will be a 3rd cost analysis exercise before TVCSP closes in March 2015. While the 2010 analysis found reductions in cost per structure sprayed and persons protected for both Mainland and Zanzibar vis-à-vis 2008, indications are that necessary changes to insecticides due to resistance, linked insecticide rotation and also overall inflation will have significantly increased operational costs. The view of USAID Tanzania is that there has been no reduction of unit costs throughout Tanzania as the TVCSP has scaled up, and no apparent economies of scale; this has not yet been examined through analysis. This situation is deemed to have considerable connotations in terms of IRS sustainability. However, the point should be made that the success of IRS may well have enabled wider cost benefits in terms of reduced disease burden and reduced malaria drug commodities' expenditure. This may explain the perception in Zanzibar of reduced malaria expenditures due to improved malaria control measures. In this context it should also be noted that TVCSP has seen a reduction in budget from USAID as against amounts originally agreed in 2010. This reduction has occurred as from USG FY 2012 (TVCSP Y2 October 2011-September 2012). Table 7: Co-agreement vs. actual TVCSP funding USG FY 2012/ TVCSP Y2 Co-agreement budget USG FY 2012/ TVCSP Y2 Actual budget USG FY 2013/ TVCSP Y3 Co-agreement budget USG FY 2013/ TVCSP Y3 Actual budget USD25.7M USD16.74M USD26.6M USD14.58M Mainland IRS 12.9M 10.1M Zanzibar IRS 793,000 550,000 Ento Monitoring 35,000 500,000 Mainland outbreak mapping/reporting 75,000 Source: PMI Tanzania MOPs 2012 and 2013 Please see discussion below in 5.7 specific to epidemiology and thresholds/criteria used to determine whether a community/district should receive IRS. As recommended there, it would be helpful to track how TVCSP balances its budget vis-à-vis optimal malaria risk management. It must be assumed that the decision not to scale-up IRS to three additional regions was at least in part determined by PMI budget considerations. Another aspect of budgeting is that there is a need to factor in cost increases due to insecticide rotation to carbamate, an increase Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 36 per structure protected of ca 40%. These matters are beyond the control of RTI in its role as implementer of TVCSP. Findings specific to RTI The need to track budget constraints vis-à-vis malaria risk management and to factor in health economic perspective: budget changes need to be monitored in terms of forward planning for optimal IRS and malaria risk management. Since no project or program can expect unrestricted funding, it is important that until the end of the project, RTI take every opportunity to factor in potential for optimal cost effectiveness. This should be a central feature of any future project planning. Lack of/insufficient TVCSP/RTI consideration of IRS cost savings' lessons elsewhere: TVCSP evaluation interviews and documentation do not discuss IRS cost saving analyses conducted elsewhere in Africa, e.g. the comparative review undertaken in Nampula and Zambezia provinces in Mozambique, which showed that government-funded IRS was less costly and maintained sufficient quality vis-à-vis PMI-funded operations, or evidence from countries funding their own IRS, e.g. Zimbabwe and Swaziland. TVCSP focus on cost efficiencies: TVCSP has continued the recruitment of local spray operators initiated in the earlier IRS project, in order to reduce camp and transport costs and minimize time spent on transfers of workers. Cost efficiencies to be considered in the proposed analysis: The previous TVCSP cost analyses did not include attention to cost savings linked to retention of spray operators and IRS ancillary staff, e.g. savings on training of new cadres of IRS workers (the project does not report on retention, as this is not a core indicator). In addition, the 3rd TVCSP cost analysis will be able to include savings due to the incinerators. Findings common to Zanzibar and Mainland The significance of contributions in kind: All Z/MNCP, regional and district officials interviewed class contributions in kind as supporting cost effectiveness (as well as ownership) within the overall IRS funding envelope: human resources (such as the District Malaria and IMCI Focal Person (DMIFP), the DHO and on Zanzibar the DMSO), employed by the Ministry of Local Government, are closely involved with IRS planning, implementation and evaluation and/or with epidemic surveillance; districts provide land for sites; village, ward and hamlet leaders participate in IRS sensitization activities and ensure community members' compliance with IRS implementation; and communities on occasion provide water. Findings specific to Zanzibar Advocacy/sensitization processes: Respondents from both ZMCP and RTI/TVCSP state that they are in the process of “sensitizing policy makers”. A draft budget for maintenance of IRS (with continued external support) has been submitted. A key aspect of the debate has been the cost-benefit of achieving pre-elimination; ZMCP (and perhaps also Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 37 RTI/TVCSP is using cost effectiveness analysis to make its case to the Zanzibar Revenue Board and other stakeholders. 'Now we only spend $35,000 on malaria commodities, whereas before it was $1M per year, so the cost-benefit of maintaining effective malaria control is obvious' ZMCP Programme Manager Costs of maintaining surveillance systems: It is recognized that surveillance systems such as those currently operating on Zanzibar are extremely costly and may indeed present significant challenges with regard to any future government funding. Findings specific to Mainland Insecticide costs are perceived/described as insurmountable by district government stakeholders. However, the evaluation found no evidence of this view being based on any cost analysis or any specific cost-benefit analysis, or of any district having conducted a detailed IRS cost analysis, specifically as part of the development of its annual Comprehensive Council Health Plan (CCHP). Financial allocation decision-making processes: Districts have autonomy in allocation of monies for health budgets from the district basket fund and/or the national block grant. The NMCP and regional structures have solely an advisory/advocacy role with regard to budget allocations for malaria control at district level. No district administrator interviewed for the evaluation provided any information on his or her district having participated in development of the new Malaria Strategic Plan, currently in draft (and presumably including a budget). Conclusions common to Zanzibar and Mainland  Districts and communities are prepared to continue in-kind support to IRS.  RTI TVCSP activities in FY2 and FY3 have been shaped by PMI budget changes.  RTI appears not to have effectively included cost savings lessons learned/best practices from other national IRS interventions in its advocacy to government stakeholders.  TVCSP has not managed to engage Mainland government stakeholders in effective discussion of financial contributions, despite considerable advocacy efforts.  The evaluation found no indication of lesson learning in terms of the Zanzibar cost￾benefit and effectiveness analysis approaches being used to advocate to Mainland districts.  TVCSP may have achieved modest economies of scale. This will have to be evaluated in the planned analysis.  TVCSP has not included all savings, e.g. on training, in its previous analyses. Recommendations common to Zanzibar and Mainland  RTI, PMI, USAID and other malaria stakeholders should closely monitor and report on the best balance between available funds and malaria risk management in the context of shifting from blanket to targeted (Mainland) and targeted to focal spraying (Zanzibar).  TVCSP should continue its advocacy on Mainland regarding inclusion of more substantial IRS financial commitments in CCHPs and national health budgets. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 38  The approaches currently employed on Zanzibar by the ZMCP and RTI/TVCSP in the debate with government should be considered for possible lesson learning and advocacy to Mainland government structures.  Districts should be supported to undertake local IRS cost analysis/budget item costing exercise as part of CCHP/Zanzibar DHT plans development.  TVCSP and IRS partners on Mainland should consider cost benefit analysis specific to longer term impacts of insecticide changes vis-à-vis potential resistance forestalling.  Contributions in-kind and intangible benefits (including reduced burden of disease) should be factored into future TVCSP/IRS cost analysis.  RTI and its IRS partners should actively seek out comparative IRS interventions funded in part/wholly by government (e.g. Zimbabwe, Swaziland) and support Tanzania stakeholders’ lesson learning and succession/sustainability planning.  RTI and its IRS partners should be informed by the cost comparison exercise (PMI vis￾à-vis government) conducted in Zambezia and Nampula provinces, Mozambique. 5.7 IRS impacts on entomology and epidemiology The evaluation objective: 'Review the documentation to assess the impact of IRS on the entomology and epidemiology in the IRS districts' is discussed here. In addition, the following two TVCSP components are considered: ‘Establish an epidemic detection and a focal spraying response for Zanzibar and IRS districts on Mainland’ and ‘Establish a viable and sustained entomological monitoring system on Mainland and Zanzibar’. The sixth TVCSP component is also addressed: ‘Develop an epidemic surveillance system on Mainland and Zanzibar’. 1. Entomology Background Vector surveillance should be an integral component of any malaria vector control intervention. It provides information on the impact of control measures on vector population density, rates of infection, susceptibility to insecticides, and residual effects of insecticides on treated surfaces. Such information guides decision-making on the selection of appropriate intervention, suitable insecticides and target areas. This section presents findings on the impact of IRS on entomology in TVCSP districts in Zanzibar and on Mainland. RTI was implementing entomological monitoring in Zanzibar until 2009 when the ZMCP took charge of all vector surveillance activities. Under TVCSP there is an established and functional laboratory at NIMR Mwanza, supported by TVCSP; TVCSP has also supported training of Mainland district vector control focal persons and community-based mosquito collectors in basic field entomology. The ZMCP has a modern insectary, a trained entomologist and a network of 21 functional sentinel sites. 5 Findings specific to Zanzibar Effective vector surveillance system: The ZMCP has a modern insectary, a trained entomologist and a network of 21 functional sentinel sites. 5 This part of section 5.7 discusses IRS impacts on entomology, including insecticide resistance. Section 5.8 discusses TVCSP action to address insecticide resistance, how the project and its partners are managing insecticide resistance in malaria vectors. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 39 Changes in vector behavior: As noted in ZMCP entomological assessment reports, Anopheles (An.) arabiensis represented less than 4% of the vector population before the scaling up of vector control interventions, but from 2010 to date, it accounts for more than 90%. An gambiae s.s. and An funestus are now virtually undetected by traditional sampling methods leaving An arabiensis as the primary malaria vector in Zanzibar. The proportion of human blood index for An arabiensis has increased from 13% in 2007 to 36% in 2010 in Pemba. Entomological studies in Pemba also show that An arabiensis has shifted its biting and resting behavior from indoor to outdoor biting behavior with 50% of the outdoor biting occurring before 23:00 hrs. Insecticide resistance: Susceptibility studies in 2010 detected resistance in An. arabiensis to lambda-cyhalothrin in Pemba, but full susceptibility in Unguja. These findings have implications on the control of the vector by IRS and LLINs. See also 5.8 below. Findings specific to Mainland TVCSP engaged NIMR-Amani to conduct entomological surveillance in the Kagera, Mwanza and Mara as part of the nation-wide surveillance. In order to strengthen entomology surveillance in Lake Zone, TVCSP established an insectary and entomology laboratory at NIMR-Mwanza and supported training of district vector control focal persons and community-based mosquito collectors in basic field entomology. Anopheles gambiae s.s., An. arabiensis and An. funestus are the malaria vectors in the lake regions with An. gambiae s.s. and An. funestus predominating along the coast, while An. arabiensis is predominant inland and in arid areas. NIMR report of 2010 shows a significant reduction in the population density of An. funestus and An. gambiae s.s. in Karagwe and Muleba following the IRS. An arabiensis is now the dominant vector species in the two districts. According to the 2012 NIMR study report, all sampled malaria vectors in Muleba and Karagwe districts. In Kagera region tested negative for sporozoite and in the other 16 districts the sporozoite rate reduced from 6% to 1%. Resistance in the March 2013 report on detection and monitoring of insecticide resistance in malaria vectors in Tanzania Mainland, NIMR-Amani described resistance to Lambdacyhalothrin in the An gambiae s.l mosquito population in the following locations: Moshi (Moshi region); Ngara (Kagera region); Geita (formerly in Mwanza, now in Geita region); Bariadi (Simiyu region) and Arumeru (Arusha Region), with mortality rates of 56%, 53.8%, 78%, 91% and 68%, respectively. The report results also showed that mosquitos were resistant to Permethrin in Arumeru, Moshi, Geita and Ngara, with mortality rates of 74%, 59.6%, 80% and 69%, respectively. Reduced susceptibility to DDT was recorded in Bariadi and Geita. Mosquito populations were highly susceptible to Fenitrothion and Bendiocarb, except in Moshi and Arumeru, which registered a reduced level of susceptibility to bendiocarb, with a mortality rate of 97% - 97.5%. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 40 Conclusions common to Zanzibar and Mainland  Evidence suggests that IRS has contributed to the changes in malaria vectors' composition, abundance/density, infectivity and behavior in Zanzibar.  Evidence suggests that IRS contributed to insecticide resistance in An arabiensis in Pemba.  The continued presence of bed bugs and fleas in pyrethroid-sprayed houses where residents possess LLINs may suggest that the insects are resistant to the pyrethroid insecticides used in IRS and in LLINs. Recommendations common to Zanzibar and Mainland  In view of the limited information on shifts in behavior of the main malaria vector from indoor feeding to outdoor, TVCSP and ZMCP should consider conducting more studies on the feeding behavior of the vector.  NMCP and ZMCP should maintain insecticide resistance monitoring at the fixed sentinel sites and share annual insecticide resistance reports with PMI and other malaria stakeholders, to facilitate evidence-based decision-making.  In view of the scarcity of information of the behavior of malaria vectors in the Lake Zone, TVCSP should conduct further vector behavioral studies in the lake regions. Bed bug and flea infestation This section of the evaluation report addresses infestation from a technical perspective. It should be read in conjunction with section 5.3, which considers community and other perspectives on this issue and also 5.8 immediately below. Findings common to Zanzibar and Mainland Notwithstanding TVCSP achievements in reducing biting mosquitoes and contributing to the reduction in malaria cases and deaths, over 90% + of female (on Unguja and especially in Mainland Lake Zone IRS districts) and 50% of male (particularly in Magu and Misungwi districts in Mwanza region) community members cite a marked increase in bed bugs (and to lesser extent fleas) and view infestation as an increasingly unacceptable side effect of IRS. Although there is no evidence that bed bugs are vectors of human disease, the issue of bed bugs and fleas in the Lake Zone districts requires urgent attention to avoid community resistance/refusal to have their houses sprayed. It seems that the spraying aggravates resistant bed bug and flea populations resulting in bites that cause considerable pain and distress. Fleas are known vectors of bubonic plague and their presence in large numbers should draw urgent health official attention. The presence of abundant bed bugs and fleas in sprayed houses which also have LLINs suggests that the insects are resistant to pyrethroids. However, it is unclear whether the introduced bendiocarb and actellic will alleviate the problem of bed bugs and fleas. It is also common knowledge that bed bugs and fleas thrive in areas with poor buildings and hygiene standards. IRS is about controlling malaria, and its success or failure is dependent on community cooperation. Therefore, TVCSP and malaria partners need to address this issue. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 41 Conclusions  The problem of increased irritation from bed bugs and fleas is real.  The continued presence of bed bugs and fleas in pyrethroid-sprayed houses with LLINs may suggest that the insects are resistant to the pyrethroid insecticides used in IRS and LLINs.  Communities in the lake zone and Unguja districts currently continue to accept IRS despite the nuisance insects. Recommendations  Although there is no evidence that bed bugs transmit diseases, TVCSP should address the issue to enhance community cooperation and acceptability of IRS.  TVCSP should substantially intensify BCC/health education efforts on the objectives of IRS.  TVCSP should educate the communities on the importance of clean rooms, bedclothes etc.  TVCSP should be more proactive with regard to community members' concerns over infestations. 2. Epidemiology Background Please also see Annex 5, entitled ‘The current malaria epidemic detection and response situation in Mainland Tanzania’. This considers in detail issues discussed here specific to the current Mainland situation with regard to IDSR. TVCSP component 2: Establish an epidemic detection and a focal spraying response strategy for Zanzibar and IRS districts on Mainland covers 7 areas of focus: 1. Provide technical assistance for the scale up of the Malaria Early Epidemic Detection System (MEEDS) in Zanzibar to all remaining government health facilities and at least 25% (approximately 20) of private health facilities, and introduce in the mainland as part of the Integrated Disease Surveillance and Response system (IDSR). 2. Conduct epidemiological surveillance in a select number of health facilities to measure the impact of IRS and other interventions in IRS and non-IRS areas. This activity includes collecting information from the patient and laboratory registers to measure malaria testing and positivity rates for under-five and above-five year populations. 3. Monitor Parasitemia Prevalence among Pregnant Women and Infants. This operational research aims to explore the feasibility of screening two groups attending reproductive and child health clinics (RCH) for malaria using RDTs. The two RCH groups will include 1) all pregnant women at the time of their first antenatal (ANC) visit and 2) all infants at the time of measles immunization (approximately 9 months of age). This approach is being implemented at 60 RCH sites (20 each) in Kagera, Mwanza, and Mara Regions in Lake Zone. 4. Explore the use of MEEDS, entomological, and facility-based sentinel site data to identify and develop a stratified map of malaria transmission hot spots, that will be targeted for meteorological and entomological monitoring and support outbreak investigation and response activities. 5. Support establishment of the Malaria Early Epidemic Warning System (MEEWS), which will provide incremental early warnings based on known meteorological (e.g. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 42 rainfall, temperature), environmental, social or occupational (e.g. migration, agricultural developments) risk factors and other potential indicators, in order to enhance malaria epidemic preparedness and prevention. 6. Participate in the review of the existing epidemic response guidelines and training manual for both NMCP and ZMCP and recommend course of action. 7. Procure the required logistics for focal spraying and when necessary, and in partnership with NMCP, ZMCP and districts, conduct focal spraying for malaria outbreaks and in high malaria transmission hot spots. Findings specific to Zanzibar Zanzibar is in a robust pre-elimination phase, which IRS through TVCSP has contributed. Malaria profile: There is documented reduction in morbidity and mortality (THMIS 2012, MEEDS). TVCSP data (weekly MEEDS reports from Unguja and Pemba) show current higher levels of malaria cases in the >5 population, compared to under-fives. An epidemic detection and response strategy has been successfully implemented in Zanzibar health facilities, covering all public primary health care facilities, hospitals and 20% of private health facilities. Surveillance activities using MEEDS include passive and active surveillance of individual malaria cases from health facilities and through the Malaria Case Notification System (MCN). Of these cases, 870/1,457 (60%) were followed up at their households where other family and community members were tested for malaria. TVCSP has also supported ZMCP in producing maps showing malaria hot spots in Unguja and Pemba. These maps have been produced on a quarterly basis from July 2012 to date. The role of the DMSO: TVCSP has trained 10 District Malaria Surveillance Officers (DMSO), one for each of the 10 districts in Zanzibar. The DMSO structure is effective, but workload per DMSO risks overload. In order to address this issue, ZMCP has recruited 5 Assistant DMSOs for the five Zanzibar districts with the highest malaria burden. TVCSP plans to purchase 10 tablets to support the expansion of MCN and to maintain the current timely follow-up of cases (within 48 hours). TVCSP has been supporting response activities by providing technical and financial support through USAID/PMI and coordination of all response activities in high transmission sites in 13 shehias (4 from Pemba and 9 from Unguja). The response activities included mass testing and treatment (mass reactive case detection), distributing LLINs, implementing BCC activities, commencing MDA in four shehias and conducting IRS. MEEWS is work in progress. TVCSP has started receiving rainfall data from Tanzania Meteorological Agency. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 43 There have been changes to the threshold criterion for determination of focal spraying: ZMCP respondents described how 'due to reductions in the IRS budget' the threshold for deciding whether a community should receive IRS has gone up, from 2/1,000 population in 2012 to 7/1,000 in 2013. There are concerns that this change might be less than optimal, despite the undoubtedly impressive advances made in epidemic detection and surveillance. Conclusions specific to Zanzibar  ZMCP, TVCSP and other stakeholders’ partnership is effective and proactive in Zanzibar.  The elimination strategy appears to be achievable, so long as all intervention components remain in place.  Changes to the threshold criterion may be a concern. Recommendations specific to Zanzibar  TVCSP should prioritize continued work with the ZMCP to develop and implement a sustained plan for malaria elimination.  Epidemiological profile research should be considered with a view to targeted interventions and BCC.  Attention should be given in the remaining time to end of project to tracking threshold impacts, with consideration of review in any future project, should epidemiological indicators suggest this is advisable. Findings specific to Mainland IRS has contributed to the documented reduction in malaria prevalence. However, such gains are fragile. In 2011 and 2013 RTI/TVCSP supported NMCP investigations to verify reports of abnormal increases in malaria cases in Muleba District, Kagera Region. TVCSP noted a dramatic increase in the malaria positivity rate in May and June 2013. There is no fully functional malaria epidemic detection system on the Mainland. The Electronic Integrated Disease Surveillance and Response (eIDSR) system is not yet effectively in place. Manual IDSR is partially functioning: IDSR paper tools have been developed but not yet fully distributed to all health facilities and training is not yet complete. Due to delays in implementing eIDSR, the NMCP, TVCSP and other partners plan to introduce MEEDS on Mainland. MEEDS is now being piloted in Muleba with the intention of it being scaled up progressively to other Kagera districts and other Lake Zone regions. TVCSP activities have supported the development of malaria risk maps to ward level. These maps will be used to target epidemiological and entomological monitoring and focal spraying. The intention is to use the maps to define the districts with the highest risk of malaria and to apply focal spraying. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 44 Availability of funds determines choice and number of communities to receive IRS, linked to epidemiological stratification and inclusion/exclusion criteria: TVCSP and its partners have had to balance finite funds (reducing over the lifetime of the project to date - see 5.6) with ensuring highest risk populations are protected. Malaria risk mapping is one aspect of this balancing. The intention is that lower risk yet still vulnerable populations will be supported through other aspects of integrated malaria control - specifically improved case management and surveillance. In this context the reported frequent stock outs of mRDTs in particular and the ongoing weaknesses in surveillance are of concern, as are reports of increased malaria positivity rates in parts of Lake Zone. An additional option is to expand ITN coverage as one of the key responses to support populations that cannot be served by IRS due to reduced funding. Draft Guidelines for Malaria Surveillance and Response have been developed by NMCP with the support of TVCSP. From June 2012, TVCSP has conducted operational research on malaria parasitemia in 60 health facilities and trained a total of 85 (32 men and 53 women) health workers (RCH workers, malaria focal persons and laboratory technologists) on how to monitor malaria parasitemia at RCH clinics and how to provide appropriate and timely monthly report and requesting of malaria rapid diagnostic tests. The main parameters recorded in this research are: ANC attendance; total malaria tests; and total positive results. Data analysis to date show a low testing rate at ANC of <20% and a positivity rate of 10-17%. Conclusions specific to Mainland  There is insufficient capacity at district level to detect and manage outbreaks in timely fashion (e.g. Muleba).  The epidemic detection system is not yet fully operational in the Lake Zone  IDSR is not yet fully established or functional: it is largely dependent on individual health facility staff members’ inputs; no supervision systems or quality control mechanisms have been developed or instituted.  Lake Zone malaria risk management in the context of finite funding requires effective and integrated control; capacity for this is lacking at district level. Recommendations specific to Mainland  As a matter of urgency, all malaria stakeholders should continue to address how to ensure the Mainland epidemic detection system is functional and operational (including ensuring constant mRDT supply).  All stakeholders should address the delayed implementation of eIDSR.  Attention should be given to the optimal Epidemic Detection System approach, e.g. MEEDS or more TVCSP support of IDSR.  Mainland should be informed by the Zanzibar approach to integrated malaria control, as transition from blanket to target IRS gathers pace.  Close attention should be given to end of project to the issue of budget constraints and overall epidemic detection and surveillance, to track whether the best balance is being achieved in terms of malaria risk management (such action is dependent on a range of stakeholders). Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 45 Focal spraying in Zanzibar and epidemiological considerations This discussion addresses not only the current situation in Zanzibar, but provides information relevant as Mainland begins its move toward focal spraying, especially in the light of recent malaria outbreaks in Lake Zone. The identification of areas of higher than average malaria transmission (hotspots) becomes crucial when transitioning from malaria control to malaria elimination. Following the identification of a hotspot, appropriate cost-effective targeted interventions must be employed. Zanzibar generates malaria case data from all MEEDS health facilities (passive surveillance) and from the malaria case notification system (active surveillance). All health facilities including private ones participate in the MEEDS. TVCSP supports ZMCP to produce weekly reports from MEEDS. The malaria notification system involves testing each family member of each individual case notified by the health facility. Those who test positive are treated. ZMCP and partners including TVCSP use the data from the two systems to identify malaria transmission hotspots. Based on data, a decision on whether the area is eligible for focal intervention(s) is made. Until mid￾2013, areas that reported 2 cases per 1,000 population were eligible for focal intervention(s). However, information from ZMCP is that due to 'financial difficulties' the figure was revised to 7 cases per 1,000 population. ZMCP targeted interventions include IRS, larviciding, active case detection, reactive screening and treatment of individuals living in close proximity to a confirmed malaria case, inventory of breeding sites and treating those positive for larvae, distribution of LLINs, IRS and mass drug administration in shehias. TVCSP supports ZMCP to plan, implement and monitor the implementation of the focal PMI-endorsed interventions. Conclusions  Focal IRS in Zanzibar proceeds in an evidence-based manner.  Evidence suggests substantial benefits in targeting control activities at foci of transmission. Concentration of activities on known areas of high risk may offset high costs associated with blanket spraying.  Blanket IRS is not sustainable, but a more focal approach would deliver more effective IRS. Recommendations  ZMCP and partners should consider establishing two thresholds for determining hotspots, one to serve during malaria transmission season and the other to serve during the non-malaria transmission season.  There is a need for far more speedy and effective transfer of lessons learned in Zanzibar to Lake Zone regions on Mainland, especially in the context of finite resources. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 46 5.8 TVCSP action on insecticide resistance The following evaluation objective is considered here: ‘Determine the extent to which RTI TVCSP has addressed insecticide resistance of the IRS program’. Background Insecticide resistance poses a serious threat to continued use of IRS and LLINs for malaria. The Global Plan for Insecticide Resistance Management (GPIRM 2012) recommends, among other strategies: rotation of insecticides, monitoring of insecticide resistance at fixed sites and avoiding use of pyrethroid IRS in areas with LLINs universal coverage. This section presents how TVCSP is addressing insecticide resistance of the IRS program. 6 Findings specific to Zanzibar Insecticide resistance management systems: There is a Vector Control Technical Committee that advises the ZMCP on malaria vector control including insecticide resistance management. TVCSP supported ZMCP to establish a network of 22 sentinel sites for monitoring insecticide resistance in Zanzibar. Zanzibar has a modern insectary to rear mosquitoes, a qualified entomologist and entomological technical assistants. Vector resistance: Until 2010 An arabiensis was fully susceptible to deltamethrin, permethrin, lambda-cyhalothrin, DDT and bendiocarb, but in 2010 permethrin and lambda-cyhalothrin resistance was found in the vector in Pemba. To manage the resistance, ZMCP replaced lambda-cyhalothrin with bendiocarb in 2012 and established an insecticide resistance management strategy that involves rotation of bendiocarb with actellic, insecticide resistance monitoring at 21 sentinel sites and avoiding IRS with pyrethroids. The ZMCP also conducts post-spray monthly cone bioassays for six months to monitor insecticide decay rate on insecticide treated surfaces. Findings specific to Mainland Insecticide resistance management systems: TVCSP established an IRS Working Group to advise NMCP on pesticide resistance management. TVCSP supported NIMR-Mwanza to establish a network of 7 functional sentinel sites in the lake regions (3 in Kagera, 2 in Mwanza and 2 in Mara) to monitor insecticide resistance. TVCSP assisted NIMR-Amani to establish additional district sentinel sites countrywide to cover all 21 regions. There are now 26 district sentinel sites for monitoring insecticide resistance in Mainland. NIMR collects monthly entomological data from the lake zone sites with support from district vector control focal persons (1 per district) and community-trained mosquito collectors (2 per site) and shares quarterly and annual reports with NMCP, districts and RTI. NIMR-Amani monitors insecticide resistance annually at 14 district sentinel sites across the country. Vector resistance: In 2011-2012, NIMR reported insecticide resistance to lambda-cyhalothrin in Muleba district. In response to this finding, TVCSP replaced lambda-cyhalothrin with bendiocarb in Muleba and Karagwe. In 2012, TVCSP engaged NMCP, ZMCP, RTI and WHO 6 This section 5.8 discusses TVCSP action to address insecticide resistance, how the project and its partners are managing insecticide resistance in malaria vectors. Please note that the entomology part of section 5.7 considers IRS impacts on entomology, including insecticide resistance. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 47 to develop a comprehensive insecticide mitigation plan. The draft plan recommends three strategies for resistance management: rotation of insecticides; monitoring of insecticide resistance at fixed sites; and avoiding IRS with pyrethroids. In 2012 TVCSP supported NIMR-Amani to conduct a cross sectional study on insecticide resistance in Mainland. Mosquitoes from 14 district sentinel sites including Geita in Mwanza, Ngara in Kagera and Tarime in Mara, were tested for susceptibility to insecticides. Permethrin and lambda-cyhalothrin resistance was detected in An. gambiae s.s. and An. arabiensis in Ngara and Geita. In Tarime, the vectors were fully susceptible to all insecticides. NIMR-Mwanza conducts monthly cone bioassays for six months post spray to assess insecticide decay rate on sprayed surfaces. A 2012 NIMR cone-bioassay report from sentinel sites in the lake region showed that bendiocarb was effective up to 12 weeks post spray and lambda-cyhalothrin for 25 weeks post spray. A 2013 paper indicates reduced susceptibility to bendiocarb in north western Tanzania (Protopopoff et al, Malaria Journal 12: 149). Conclusion common to Zanzibar and Mainland  TVCSP has a sound system to monitor and manage insecticide resistance in Zanzibar and in Mainland including the Lake Zone districts. Recommendations common to Zanzibar and Mainland  RTI and partners should finalize the insecticide resistance mitigation plan and its implementation plan.  TVCSP should ascertain vector composition and behavior in the Lake Zone districts. 5.9 Lessons learned The focus here is on the evaluation objective that asks: “Identify the lessons learned from the RTI TVCSP since the beginning of the program, taking into account the perspectives of the stakeholders, the donors and the beneficiaries and the evaluation question: ‘Are there specific lessons from the project that can be applied in the second phase of the project and to other PMI programs and countries?’”. Consideration of lessons learned requires attention to all six TVCSP components; this section should be read bearing in mind discussion of all other evaluation questions and objectives. Please note that discussion here is based on team members' interpretations of lessons learned, as well as on inputs from RTI and other stakeholders. Overall TVCSP lessons learned 1. There is an imperative need to establish and sustain epidemic surveillance systems to support IRS (see also 5.7).  A key lesson learned is the imperative need to ensure there are robust and sustainable surveillance systems, not least in districts and regions where blanket IRS is being phased out and before IRS begins in any new district(s); at the time of writing such systems do not exist in Lake Zone, as is acknowledged by the NMCP.  TVCSP/RTI has been active in promoting transfer of surveillance lessons learned in Zanzibar to Mainland; any delays cannot be adjudged due to project shortcomings. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 48  An absence of effective epidemic surveillance systems can only reduce opportunity for maintaining gains made in malaria control, to which IRS has substantially contributed.  TVCSP has not been able fully to capitalize on the major reductions in child malaria in Kagera - to do this requires effective partnership between all stakeholders, not least in terms of implementing effective systems, supporting health facilities and ensuring provision of key commodities such as mRDT and ACT, and is beyond the scope of any one project or intervention.  No one project can be expected to act as a sole guarantor of maintained reductions in malaria cases; this requires a whole health system approach. 2. It is better to take short-term action on setting up and sustaining surveillance systems, rather than awaiting optimal conditions. As a result of the IDSR delays, a further key lesson learned has been the need to make best use of existing systems, rather than focusing solely on progress towards optimal information gathering. Thus the decision has been taken to proceed with a MEEDS approach in the Lake Zone, focused on malaria, until such time as the IDSR is genuinely and widely functional, beginning with Kagera as a pilot region.  The electronic surveillance and reporting system MEEDS in Zanzibar is entirely separate from the HMIS, while the IDSR system is intended to integrate malaria as one of the diseases tracked; this integration has been discussed by PMI and government partners since 2010, with funding allocated in successive MOPs.  Due to the Mainland delays in developing and instituting IDSR, inclusion of the MEEDS approach is seen as the best way forward for the immediate future. 3. Effective surveillance is imperative in the scale down to focal spraying. Effective epidemic surveillance will be essential as IRS scales down from blanket to targeted spraying (see 5.7 above for discussion of this topic). 4. Attention to Zanzibar best practices is necessary. TVCSP and its malaria partners in both Mainland and Zanzibar have worked to address the serious issue of failure to develop the IDSR sufficiently quickly, by adapting components of MEEDS as an interim measure. Because Zanzibar is further along the road to pre-elimination, consideration should be given to any further lessons learned/best practices that might have emerged and that might be adaptable to a Mainland environment. One such might be M&E feedback to health facility level, addressed further as point 6. 5. Attention has been given by TVCSP to leveraging Public-Private Partnership opportunities. TVCSP has been successful in developing a PPP with Geita District Council, Geita Gold Mine, AMREF and Plan International, resulting in spraying being undertaken in February 2012 in Kalangalala (urban) and Mtakuja (peri-urban) wards. This operation resulted in more than 18,000 structures being sprayed (> 90%) and protection of 111,563 people (U5 22,547; pregnant women 4,821). In addition, SELCOM has supported rollout of MEEDS on Zanzibar and will similarly contribute to its piloting in Lake Zone. Information from RTI is that there are no current arrangements for support from the private sector for the Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 49 mainland IDSR/MEEDS. In short, there are no additional examples of TVCSP PPP. RTI is in discussion with the Kilombero Sugar Company, regarding possible IRS operations in the Kilombero district; this is a future activity outside the project area. 6. The importance of continuous attention to BCC cannot be overstated. Continuous attention to the potential need for evolving BCC is important and an approach that TVCSP appears not to have fully addressed over time. As communities receive and accept IRS and as the type of spraying changes to targeted, people are likely to require information that supports them to be aware of mosquito behavior, the importance of continuing to using nets and seeking early diagnosis and treatment. 7. It is never too early to begin the development of an exit strategy and succession planning. A clear lesson learned is that it never too early to consider exit strategy and succession planning, including attention to IRS cost assessment; such focus is at best in its earliest stages at regional and district levels on Mainland. A number of donor-funded programs and projects now require these activities to be initiated during the inception phase, so as to assist longer term planning, specifically on sustainability and handover/ownership. RTI staff members indicate that an exit strategy will be developed during project year 4, as set out in the just-approved implementation plan. This document was not seen by the evaluation team. Related to this is the potential to include a results framework with timeline and milestones agreed during the inception phase of any future project - linked to exit strategy and succession planning. 5.10 Potential improvements and future projects This section reviews the evaluation objective: ‘Identify areas for implementation improvement and propose key activities for the follow-on project’. This is an over-arching objective and as such discussion will, where appropriate, address any of the TVCSP components. Furthermore, it is clearly not an objective to be discussed and managed by RTI/TVCSP alone, as some decisions as to short-term and longer term direction would be dependent on funding allocations and wider priorities within the malaria environment on Mainland and in Zanzibar. The lessons learned discussed above in 5.9 are all potential areas of enhanced focus for a new project and have been addressed to a greater or lesser extent by RTI/TVCSP and its malaria partners. Please also refer to 6.1 (cross-border collaboration) and 6.2 (BCC) for consideration of both TVCSP implementation improvement and activities for future projects. The discussion in 5.6 (cost saving/analysis) and recommendations made should also be closely considered, primarily in terms of planning and budgeting for a new project. Issues discussed in 5.7 under epidemiology should contribute to any discussions as to future direction. One absolutely key activity for any future project is to support the existence of an effective and sustainable epidemic detection and response system. As already discussed in detail in 5.7 and 5.9, significant challenges remain on Mainland, while gains made in Zanzibar Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 50 should not be assumed to be entirely without risk. It would be advisable for any future project to work with partners to ensure there is an adequate epidemic detection and response system in place and effective before commencing any IRS operations in new districts/regions. Evidence from this mid-term performance evaluation indicates the need for constant attention to how best to balance budget realities and optimal IRS, so as to ensure gains are sustained; TVCSP activities have been curtailed due to budget reductions and changes have been made to the ways in which thresholds for IRS are calculated, based at least in part on financial criteria. It is essential that epidemiological data continue to be the mainstay of IRS planning, for all malaria stakeholders, both for the remainder of TVCSP and in any follow-on project. Evaluation findings indicate the need for continuous monitoring of and responsiveness to community perceptions with regard to malaria control, not least in terms of sustainability of gains in reduction of malaria cases. A potential lesson here is speedy responsiveness to community members' beliefs and behaviors, however technically incorrect these might be considered to be. Another area of focus in any future project should be ensuring that feedback to all levels of partnership is effective and timely. RTI has undoubtedly developed a strong, coherent and effective monitoring and evaluation (M&E) system, with detailed attention to Performance Monitoring Plan indicators in quarterly and annual reports and a similarly coherent knowledge management system whereby district and sub-district data are collected, collated by data entry clerks at district level and passed upwards to RTI for further data cleaning and analysis. However, evidence from fieldwork discussions on Mainland, e.g. with data entry clerks, with DMOs and DMIFPs, is that while data are passed to RTI, feedback is apparently less comprehensive, thereby less effective. An example was given by an RTI employee of Biharamulo District Hospital, which neighbors Muleba where the recent outbreak occurred (it was not made clear why RTI would be collecting data in a non-IRS district), and other health workers' concerns. ”No explanation had been given by RTI to the senior nurse as to why the hospital should provide all the data, the use to which it would be put and how it would benefit the working of the hospital. So she refused to give the data entry clerk the information until she received a proper explanation. RTI came and talked to the doctor in charge and after that, data was given. But staff still feel that they don't get feedback. Several dispensary clinical officers and nurses in Kagera and Mwanza have told me that they provide all the information, but never get any feedback, unless the DMIFP calls and they have a discussion.” In this context, and given the fact of major problems with the IDSR, it is important that all levels of health workers and relevant administrators be given feedback, for several reasons: to foster partnership and ownership, to assist in evidence-based planning for malaria control Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 51 and overall public health care, and to acknowledge the vital part played by all IRS stakeholders, whatever their position and role. 5.11 Sustainability Discussion here is on the final evaluation question: ‘Is the program [TVCSP] addressing sustainability and how?’ Sustainability represents the third of the three areas the evaluation team was requested to consider in depth. A number of the findings described below have been discussed elsewhere in this report. This section is linked to 5.6, 5.9, 5.10 and to section 7, as well as to points raised in various other parts of the report, e.g. under 5.3 and 5.7. As such it is a crosscutting issue. Sustainability in this report refers to the broad topics: financial and epidemiological/technical sustainability and community acceptance and support. Financial sustainability 1. RTI/TVCSP has been diligent in prompting discussion with national, regional and district authorities regarding sustainability. While government structures and decision-making mechanisms differ between Zanzibar and Mainland, the evidence from the evaluation fieldwork indicates that RTI/TVCSP has dedicated time to discussing and/or facilitating debate with key stakeholders about project exit planning and the need for post-funding financial allocation and budget planning. While indications are that this process has developed to an extent in Zanzibar (see 5.6), this appears not to be the case on Mainland. One potential reason why debate on government sustaining of IRS budgets is somewhat limited may well be that there is a hope/expectation that PMI will continue its funding. An indication of the extent to which further discussions are required on financial contributions and potential for national sustainability of IRS on Mainland is given by a review of the Misungwi District (Mwanza region) Comprehensive Council Health Plan (CCHP) for FY 2013-14. The CCHP what was described by local government staff as an adequate contribution to IRS activities; Table 8 provides details of proposed district expenditures on malaria control, including IRS. 7 Table 8: Misungwi District CCHP FY 2013-14 allocations to malaria control (in USD) IRS 22,255 ACT 86,741 ITN 6,046 Total budget allocations to malaria 105,042 Total CCHP budget 3,662,216 % allocation to malaria < 3% % allocation to IRS 0.6% 7 It was not possible to examine other districts' CCHPs, or to discuss in any detail Unguja districts' financial plans. One limitation was that the evaluation team was able only to meet acting DEDs, which restricted access to CCHPs. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 52 2. Insecticide costs are considered on Mainland to present an insuperable barrier to government-funded IRS. This perception was mentioned several times by district administrators in Kagera and Mwanza and was referred to by RTI employees. See 5.6 for more details. No mention was made of any district having conducted a cost analysis of IRS, e.g. in connection with annual planning. It is apparent that considerable further advocacy is required, ideally not only by RTI/TVCSP but by PMI (as is already the case) for forward sustainability planning, so that if and when PMI no longer funds IRS, mechanisms are in place for government take-over. One recommendation could be that districts do undertake cost analysis specific to IRS. 3. RTI/TVCSP should provide more information on cost-savings and government-funded IRS operations in other African countries. As discussed elsewhere in this report, there are examples from other countries, e.g. Zimbabwe, of government being able to finance IRS without external support over a considerable number of years. In addition, the example of Mozambique should be discussed. This is where a comparison was made between Nampula and Zambezia provinces, one whose IRS was funded by PMI, the other by government, and where the government-funded operation was found to be more cost-effective and of no less quality. RTI should be more proactive in its presentation of potential for Tanzania to fund more of its IRS operations. 4. There is a need for PMI to provide clarity on budgets in the short/medium term. It appears that the reduction in the TVCSP budget led to the decision not to scale up to six regions on Mainland, but to concentrate on Kagera, Mwanza and Mara. As discussed in 5.7 and elsewhere, evaluation indications are that the reduction in budget contributed to changes to the threshold criteria for determining which communities to spray. It is conceivable that such decisions may have epidemiological consequences. 1. Epidemiological and technical sustainability 1. One absolutely key activity for any hope of sustainability is to support the existence of an effective and sustainable epidemic detection and response system. As already discussed in detail in 5.7 and 5.9, significant challenges remain on Mainland, while gains made in Zanzibar should not be assumed to be entirely without risk. Sustainability of IRS gains will require all malaria stakeholders working together to ensure there is an adequate epidemic detection and response system in place and effective before commencing any IRS operations in new districts/regions, and indeed before leaving any district that has previously received IRS. 2. Technical capacity has been developed to a standard able to manage and (it is hoped) sustain IRS. It does seem from the evaluation findings that there is now adequate technical capacity at regional and district levels on Mainland and at ZMCP and district level in Zanzibar to manage and sustain IRS - with the proviso that there needs to be more attention to the institutionalization of such technical capacity, through a health systems approach that factors in regular training for all cadres, a supportive supervision structure and quality assurance mechanisms, so as to avoid Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 53 loss of such expertise, should key staff members be transferred. Clearly such technical expertise needs to be matched by an effective epidemic detection and response system. 3. There is a need to take action to support retention of spray operators and other IRS workers. This is linked to the point above; the technical capacity of spray operators is central to successful IRS. Working with spray operators from communities that receive IRS is one way to strengthen potential or sustainability. Another approach is to ensure that all trained spray operators' details are recorded, so that retention over several spray seasons is possible. The use of MPESA, as introduced in May 2013, to pay spray operators and other temporary/seasonal IRS workers, may also contribute to retention of these key personnel. 3. Community acceptance and support 1. Fully informing community members is essential for ensuring sustainability. This mid-term evaluation of TVCSP has found that while community members receive adequate BCC about the actual process of IRS (before, during and after), there has been insufficient attention to emerging perceptions. As discussed in 5.3, the key issue is concern over infestation. Another is confusion over the shift from blanket to targeted spraying (on Mainland) and the rationale for continuing with focal spraying (in Zanzibar) and uncertainty as to what such change might mean in terms of mosquito resurgence and potential for malaria. A third issue is a minority view that IRS confers immunity or at least increased protection against mosquitos, leading to the expressed belief that LLINs need no longer be used. It is essential that community members' concerns be addressed throughout the lifetime of an intervention; perceptions will change over time and there may be different opinions. As was seen in South Africa, communities can begin to reject IRS due to infestation. Sustainability both of IRS per se and wider gains in malaria control might be jeopardized should communities not be seen as full partners and stakeholders in IRS. 6. Additional TVCSP components This section of the mid-term evaluation report briefly considers TVCSP components not specifically addressed in the mid-term performance evaluation SOW evaluation objectives and questions. Where appropriate, recommendations are made, both for the remainder of TVCSP and for any successor IRS intervention. 6.1 Cross-border collaboration Discussion Currently Tanzania (and by extension RTI in its management of TVCSP) is not collaborating on IRS activities with neighboring countries such as Kenya and Uganda, both of which are also in receipt of PMI funds for IRS. There is mention in a number of TVCSP quarterly reports that there might be collaboration with Rwanda, but this has not (yet) materialized. It appears that while RTI Tanzania has close links with its regional office in Nairobi, e.g. for access to technical expertise on environmental compliance and with RTI headquarters in North Carolina, it - and by extension TVCSP - has not made full use of opportunities for wider collaboration. Thus while the South Africa Medical Research Council has been a partner Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 54 in data management for improved surveillance, there is no documented discussion in TVCSP quarterly or annual reports, or in annual implementation plans, of specific efforts to learn from best IRS practices elsewhere in Africa. As has been previously discussed in this report, there is information available on how national governments have been able to sustain locally-funded IRS operations over a period of years, e.g. in Zimbabwe and Swaziland. In the context of reduced budgets and increased IRS costs due to the need for insecticide rotation, as well as potential for sub-optimal malaria risk management, it would be pertinent for RTI and indeed PMI to examine such approaches to see if any aspects could be adopted or adapted for use in Tanzania. The same is true of the costing exercise carried out in Nampula and Zambezia provinces in Mozambique, where management of a provincial PMI-funded IRS and a provincial operation funded by government was reviewed. One key finding was apparently that the government-funded IRS was more effective in budget management and economies of scale. Another potential area for cross-border collaboration in terms of possible lesson learning is with regard to bed bug infestation (see 5.7 above). IRS in South Africa almost halted due to severe infestations, with bed bugs having become resistant to the DDT used for IRS. The current solution in that country is to work with an additional spray operator per team, dedicated to spraying a carbamate in houses whose residents report infestation. Recommendations  TVCSP (and any follow-on project) should make more effective and proactive use of best practices and lessons learned elsewhere specific to effectiveness of IRS operations - these may have positive implications in terms of cost effectiveness and sustainability.  There should be discussion of possible study visits by NMCP and ZMCP staff to countries whose governments fund IRS, specifically to examine potential for both more fruitful cross-border collaboration/lesson learning and to investigate potential for cost savings. 6.2 Additional TVCSP area of focus: Behavior Change Communication This section of the report has been added in response to the fifth core TVCSP indicator: Number of people reached with IEC/BCC activities for IRS. Discussion In TVCSP years 1 and 2, the JHU Communication and Malaria Initiative in Tanzania (COMMIT) project was the Behavior Change Communication (BCC) and community mobilization partner (see also 5.3 for discussion of BCC inputs) on Mainland. Civil society organizations mobilized communities before IRS, initially in Kagera and subsequently in Mwanza and Mara. This partnership ceased in 2012, due to several reasons: the sheer proliferation of CSOs and CBOs made it difficult to supervise and quality assure BCC activities; several of the CSOs that were either national or international, had their own approaches and sometimes insufficient attention to local sociocultural mores; and cost considerations came increasingly into play. Site IEC Mobilizers (SIMs), already working as COMMIT volunteer Community Change Agents, received training in IRS and engaged in BCC activities at community level - a number of SIMs Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 55 are still active in such work. A JHU BCC specialist was seconded to RTI and mass media (radio spots) and interpersonal communication (IPC) materials were developed, informed by the findings of the qualitative study undertaken by RTI in Kagera in 2010. It appears that the decision to cease the TVCSP-COMMIT partnership was due to budget constraints. BCC for IRS on Zanzibar has been managed by the ZMCP BCC Unit, while the NMCP BCC Unit leads such activities for Mainland. Currently, there does not seem to be any dedicated BCC expertise within RTI Tanzania. One possible impact of the reduced support to BCC over the lifetime of TVCSP may be that there has been less than speedy attention to community concerns over infestation, the shift from blanket to targeted/focal spraying, etc. Moreover, there does not appear to have been adequate forward planning for BCC, e.g. how best to provide BCC that informs community members and leaders (and indeed health workers) about what they can best do to sustain gains made in malaria reduction and management, in the context of reducing numbers of rounds and types of spraying. Such focus is important, in that community support and informed behavior are essential to sustained malaria control. As noted above in 5.1, the percentage of community members who refuse spraying is used by TVCSP as a proxy indicator for the effectiveness of the BCC and interpersonal communication (IPC). It is necessary to point out that while all community members interviewed, without exception, accepted IRS as a public good, their opportunities to reject spraying are limited, due to community structures and potential sanctions. Therefore, it is advisable that the project devise another, more community-sensitive BCC indicator or indicator set that allows more finely grained understanding of community members' evolving needs and perspectives - see also 5.3 above, on community perceptions of IRS. Recommendations  RTI/TVCSP (and PMI) should consider budgeting for a focused, quick BCC review, in partnership with the MNCP and ZMCP BCC Units and JHU COMMIT, specific to assessing emerging/existing community concerns and to provide action-oriented recommendations to end of project.  Refresher training for SIMs and other key BCC workers should be considered, so as to equip people to provide IPC on relevant new topics, e.g. infestations.  Any future IRS project should ensure adequate and consistent funding for BCC.  Any future IRS project should have appropriate and disaggregated BCC indicator(s) reflective of community members' perceptions and needs. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 56 ANNEXES Annex 1: Scope of Work Annex 2: People Met Annex 3: TVCSP Evaluation Plan Annex 4: TVCSP Evaluation Schedule and Sample Annex 5: The current malaria epidemic detection and response situation in Mainland Tanzania Annex 6: References Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 57 Annex 1: Scope of Work African Strategies for Health Consultant Scope of Work: Team Leader/ Malaria/Vector Control Expert (International) Activity: Mid-term performance evaluation of RTI Tanzania Vector Control Scale￾up Project (TVCSP). Number of Days: 31 Dates: September 1, 2013 – October 15, 2013 Principal Work Location: Tanzania Travel Required: Yes Tanzania 1. Project Background The African Strategies for Health (ASH) Project is a five-year contract funded by the United States Government through the United States Agency for International Development Africa Bureau (USAID/AFR). ASH is implemented by Management Sciences for Health (MSH) in partnership with three Africa-based partners: African Population and Health Research Center (APHRC), Khulisa Management Services and Institut pour la Santé et le Développement (ISED), of Dakar University, Senegal. ASH also works in close consultation and collaboration with a variety of African public sector, civil society, and private sector health development institutions. ASH’s mandate is to assist the Africa Bureau of USAID to work with African institutions, other development partners and partners within the USG to provide a strategic vision for guiding investments to further the health of Africans. The project conducts reviews, assessments and a dialogue with partners working in the field to improve the understanding of constraints impeding the realization of the vision and the reaching of the Millennium Development Goals (MDGs) and the goals of the Global Health Initiative (GHI). 2. Activity Background a. Purpose of Assignment  The purpose of this evaluation is:  To determine the extent to which the RTI Tanzania Vector Control Scale-up Project (TVCSP) objectives and key result areas are being met;  To document lessons learned to inform the second phase of the program; and  Asses the various approaches used by for the TVCSP to determine what is working to achieve the intended results Background The US President’s Malaria Initiative The President’s Malaria Initiative (PMI) is a core component of the GHI, along with HIV/AIDS, and tuberculosis. PMI was launched in June 2005 as a 5-year, $1.2 billion initiative to rapidly scale up malaria prevention and treatment interventions and reduce malaria-related mortality by 70% in the original 15 countries by the end of 2015. This will be achieved by reaching 85% coverage of the most vulnerable groups — children under five years of age and pregnant women — with proven preventive and therapeutic measures, including artemisinin-based combination therapies (ACTs), insecticide-treated nets (ITNs), intermittent preventive treatment of pregnant women (IPTp), and indoor residual spraying (IRS). Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 58 Figure 1: Trends in Malaria Prevalence Percent children age 6-59monthstesting positive for malaria by RDT 20 18 11 10 THMIS 2007-08 THMIS 2011-12 7 3 Total Urban Rural The goal of PMI is to reduce malaria-associated mortality by 70% in Tanzania after full implementation of FY 2014 funding. PMI will assist Tanzania to achieve the following targets among persons at risk for malaria:  More than 90% of households with a pregnant woman and/or children under five will own at least one ITN;  At least 85% of children under five will have slept under an ITN the previous night;  At least 85% of pregnant women will have slept under an ITN the previous night;  At least 85% of houses in geographic areas targeted for IRS will have been sprayed;  At least 85% of pregnant women and children under five will have slept under an ITN the previous night or in a house that has been sprayed with IRS in the last 12 months. The Malaria Situation in Tanzania On the Mainland, 93% of the population lives in areas where malaria is transmitted while the entire population of Zanzibar is prone to malaria infection. Unstable seasonal malaria transmission occurs in approximately 20% of the country, while stable malaria with seasonal variation occurs in another 20%. The remaining malaria endemic areas in Tanzania (60%) are characterized as stable perennial transmission. Plasmodium falciparum accounts for 96% of malaria infection in Tanzania, with the remaining 4% due to P. malariae and P. ovale. The principal vectors of malaria on the Mainland are the Anopheles gambiae complex (An. gambiae sensu lato and An. arabiensis). In Zanzibar, high coverage of ITNs and IRS have changed the composition of the malaria vector population. Routine entomological data shows that An. arabiensis, which made up less than 4% of the population before scaling up of vector control interventions in 2005, represented almost 90% of the population in 2010, replacing the more efficient malaria vector, An. gambiae. The 2011/2012 Tanzania HIV/AIDS Malaria indicator survey (THMIS) puts malaria prevalence at 9.5 % (9.7% for mainland, 0.2% for Zanzibar). Compared to 2007/08, there was a 47% reduction in malaria prevalence from 18% to 10% (Figure 2). The malaria prevalence in Zanzibar reduced from 0.8 in 2007/2008 THMIS to 0.2%, giving a marked 75% reduction in malaria prevalence. The data shows varying regional prevalence (figure 1). Malaria prevalence ranges from less than 1% in Arusha, Kilimanjaro, Singida, and Iringa to 33% in Geita. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 59 There is also significant variation between urban (7%) and rural 20%). The same survey showed that malaria prevalence increased with age while anemia prevalence decreases with age. The regions that showed marked improvement were Kagera, Simiyu, Shinyanga, Mtwara, and Mwanza. IRS is currently taking place in the regions of Kagera, Mwanza, and part of Simiyu regions. Tanzania Mainland and Zanzibar Malaria Control Programs The United Republic of Tanzania is made up of the Mainland (43 million people in a total area of 881,000 sqm) and Zanzibar (1.2 million people in 2,500 sqm). Tanzania has two Ministries of Health, one for Mainland and another for Zanzibar. Both Ministries of Health have separate Health Policies and Strategic Plans. There are also two Malaria Control Programs. The National Malaria Control Program (NMCP) covers mainland and the Zanzibar Malaria Control Program (ZMCP) covers the islands of Unguja and Pemba in Zanzibar. Mainland: The NMCP 2008-2013 NMMTSP target is to scale-up IRS to 60 of the Mainland’s 113 districts over a five-year period. These include districts/regions with high malaria prevalence and the 25 epidemic prone districts. The plan includes setting up systems for entomological monitoring and epidemic preparedness and response. The NMCP’s 2008-2013 Medium-Term Strategic Plan targets indoor residual spraying (IRS) in areas of high malaria prevalence and unstable transmission. The NMCP used 2007/08 THMIS data (Figure 3) to target IRS in the Lake zone that showed the highest burden of malaria among all 21 regions of the Mainland. Data from 2010 THMIS showed that Lake Zone has the highest under-five mortality rate of 109/1,000 live births, above the national average of 81/1,000 live births. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 60 IRS on the Mainland was launched in 2007 in Muleba and Karagwe Districts of Kagera region, which were experiencing malaria outbreaks at that time. To date, Muleba and Karagwe districts have had six and five rounds of IRS, respectively. In 2009, PMI supported the expansion of IRS to cover the remaining five districts of Kagera Region and in 2010 and early 2011, IRS expanded to cover all the two remaining regions of Lake Zone, Mwanza and Mara (Figure 3) . In total, 18 districts in the Lake Zone (7 in Kagera Region, 6 in Mwanza Region, 5 in Mara Region) are being sprayed. Zanzibar: Malaria has been controlled at least two times in Zanzibar (1960s and 1980s) using large scale IRS campaigns, only to come back in full force after spraying was discontinued because of lack of resources. Zanzibar’s pre-elimination strategy relies on continued universal and/or focal spraying, universal coverage of ITNs, and strong surveillance to maintain control and prevent a rebound of malaria. From 2006 to 2011, Zanzibar benefited from six rounds of blanket IRS with household coverage over 90% for all the rounds; more than one million residents are protected with each round. The seventh round was targeted at districts showing malaria transmission. The combined household spraying and use of insecticide-treated nets has reduced malaria prevalence to 0.2% (2011/12 THMIS) and propelling Zanzibar into a pre-elimination phase. 3. RTI Tanzania Vector Control Scale-Up Program (TVCSP) With funding from PMI, USAID/Tanzania awarded to RTI International (RTI) a five-year Cooperative Agreement (CA) #621-A-00-10-00015-00 from the U.S. Agency for International Development (USAID) Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 61 starting on March 9, 2010 and ending March 8, 2015. To date $56,082,101 has been obligated to the project from FY2010 through FY2012. Under the CA, RTI works with the NMCP, ZMCP, and the local governments to implement indoor residual spraying (IRS), entomological and environmental monitoring, and limited epidemic malaria surveillance and response activities in Mainland Tanzania and Zanzibar through the Tanzania Vector Control Scale-up Project (TVCSP). The goal of the TVCSOP is contribute to the reduction of the burden of malaria by interrupting malaria transmission though IRS; and prevention of adverse effects of malaria epidemics through focal spraying. The TVCSP was originally designed to address the following five objectives of:  Scale up IRS in Tanzania by expanding IRS coverage from one region to six or more regions on Mainland and maintain required IRS, services as required in Zanzibar.  Establish an epidemic detection and a focal spraying response strategy for Zanzibar and IRS districts on Mainland.  Develop an environmental compliance strategy and monitoring plan for Zanzibar and Mainland.  Develop capacities of ZMCP, NMCP, and districts to plan and implement IRS, including logistics planning, environmental compliance, and epidemic response through focal spraying.  Establish a viable and sustained entomological monitoring system on Mainland and Zanzibar.  The program description was revised in the second year of implementation to add the seventh objective of “Develop an epidemic surveillance system on Mainland and Zanzibar.” The major components of the program are: IRS, environmental compliance; entomologic monitoring; epidemic surveillance and response; capacity building; cross cutting activities of gender and cross border collaboration; and monitoring and evaluation. The program targets are:  Over 85% of structures in IRS target areas sprayed.  85% of population protected through IRS in the targeted geographic area. Geographical coverage  IRS  Mainland: initially six regions but may be expanded to additional regions or different regions after the first two years.  Zanzibar: one full round of IRS in approximately 8 districts with evidence of malaria transmission and no universal coverage and use of ITNs.  Focal Spraying: all malaria transmission hot spots in Zanzibar and IRS districts on Mainland.  Entomologic monitoring: national for both Mainland and Zanzibar in both IRS and non IRS areas.  Epidemiological monitoring: Zanzibar and Lake Zone, with control areas in non IRS areas Indicators  Number of houses in the IRS program target areas sprayed with residual insecticide in the last twelve months with USG funds.  Proportion of houses in the IRS program target areas sprayed with residual insecticide in the last twelve months out of the total number of occupied houses in the IRS program target area.  Proportion of population protected through IRS in targeted geographic area (number of people residing in houses sprayed/Estimated total population in targeted district).  Number of people trained with USG funds to deliver IRS (disaggregated by sex).  Number of people reached with IEC/BCC activities for IRS. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 62 Expected Results  The expected outcome of the TVCSP are:  Increased geographical IRS coverage for IRS.  Strengthened local capacity of NMCP and ZMCP to plan and conduct IRS, including focal spraying.  Established system for epidemic response with focal spraying.  Established and functional entomological laboratory on Mainland and quality control for entomological monitoring for Mainland and Zanzibar  Established Environmental plan for IRS  Established viable sustainability plan for IRS.  Reduced malaria transmission in IRS areas  Reduced morbidity in IRS area (hospital attendances, admissions, MPR)  Established epidemic surveillance system in Zanzibar and mainland  Insecticide mitigation and management plan in place Detailed Project description Please see a full program description Attached 4. Objectives of the external evaluation This external evaluation shall:  Determine the extent to which the achievements have been made or not in relation to the expected results of RTI TVCSP since the beginning of the program;  Determine progress made in the capacity building and institutional strengthening of the Ministry of Health and Social Welfare (MOHSW)/NMCP and Zanzibar MOH ZMCP;  Establish how the project is perceived by the NMCP, ZMCP, Local Governments, and the Community  Determine the ways in which RTI TVCSP is addressing gender gaps  Assess the processes put in place to address human and environmental safety of the program, in relation to the US Government and local environmental laws.  Describe any cost savings and any interventions made to reduce the cost of the structures sprayed and people protected.  Review the documentation to assess the impact of IRS on the entomology and epidemiology in the IRS districts  Determine the extent to which RTI TVCTS has addressed insecticide resistance of the IRS program.  Identify the lessons learned from the RTI TVCSP since the beginning of the program taking into account the perspectives of the stakeholders, the donors and the beneficiaries; and  Identify areas for implementation improvement and propose key activities for the follow-on project. 5. Evaluation methodology The Evaluation team shall use facilitative methods and activities that will enhance collaboration and dialogue among counterparts particularly Ministries of Health NMCP and ZMCP, and the local governments. The Evaluation team shall work under the supervision and guidance of the Agreement Officer Representative (AOR) for the RTI TVCSP and the Mission Monitoring and Evaluation Officer of the Health Office. The AOR will organize all internal PMI meetings including linking the evaluation team with the Health Office Team Leader and other team members. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 63 This performance evaluation will address the following key questions:  What have been RTI TVCSP key achievements so far?  To what extend are the RTI TVCSP objectives likely to be achieved?  How is the project perceived and valued by the United Republic of Tanzania (MOHSW on mainland, MOH in Zanzibar, NMCP, ZMCP, Local governments in IRS areas of operation) and key malaria stakeholders in Tanzania?  In what ways has RTI project integrated gender considerations into its activities and how have gender issues been addressed in implementation of the program?  What approaches has RTI used and what is working towards the achievement of the intended results and improve acceptability of IRS?  Are there specific lessons from the project that can be applied in the second phase of the project, and to other PMI programs and countries?  Is the program addressing sustainability of the interventions and how? The Evaluation team shall develop an evaluation design and data collection methods; using a mixed methods approach to gather both quantitative and qualitative information that is based on sound social science methods and tools used in a manner to minimize potential biases. The proposed evaluation design data collection methods, tools to be used and work plan will be submitted to USAID/Tanzania and discussed during the team-planning meeting in Tanzania. The final evaluation design and work plan shall be presented to the AOR and relevant PMI and Health Team members for comments during the last day of the Team Planning Meeting (TPM) in Tanzania. The RTI TVCSP AOR will arrange for an initial introductory meeting with appropriate MOHSW and NMCP staff at the outset of the process. Where necessary the AOR may participate in meetings with the GOT representatives and partners. A general list of relevant stakeholders and key partners will be provided to the Evaluation team by the AOR at the time of arrival, but the Evaluation Team will be responsible for expanding this list as appropriate and arranging the meetings and appointments so as to develop a comprehensive understanding of the program and services offered through the RTI TVCSP Cooperative Agreement. The final methodology, together with evaluation tools and work plan, will be developed as a product of the Team Planning Meeting (TPM) shared and approved by the Mission, NMCP, and NIMR prior to application. 6. Evaluation process Document Review: SOW: Before finalization, the SOW will be shared with:  The NMCP and ZMCP to solicit their input and to solicit their input.  RTI to inform them of the scope of the external evaluation, and for comments.  ASH Project for comments and clarifications. Background Reading:  Prior to conducting field work, the evaluation team will review existing literature and data (Annex 1) that will include:  NMCP and ZMPC strategic documents and policies, medium-term strategic and business plans M&E strategic plans, and IRS BCC strategy; Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 64  PMI planning and strategic documents, including PMI strategic plans and Malaria Operational Plans.  RTI TVCSP Cooperative Agreement documents including:  RFA  application proposal  Annual implementation plans  RTI Monitoring and evaluation plan  quarterly and annual reports  IRS special reports  Environmental compliance Reports  Pesticide Evaluation and Safe Use Action Plans (PERSUAPs) and Supplemental Environmental Assessments (SEA) and Justification letters  Entomology monitoring reports  Baseline and other evaluation reports  Any other reports and documents reflecting RTI TVCSP work in Tanzania. All team members will review these documents in preparation for the initial team planning meeting. Team Planning Meeting in Tanzania  A two-day team-planning meeting will be held in Tanzania before the evaluation begins. The first meeting will:  review and clarify any questions on the evaluation SOW;  clarify team members’ roles and responsibilities;  establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion;  review and develop final evaluation questions;  review and finalize the assignment timeline and share with PMI,;  develop data collection methods, instruments, tools and guidelines;  review and clarify any logistical and administrative procedures for the assignment;  develop a data collection plan;  finalize the evaluation questions;  draft the initial workplan to share with PMI;  develop a preliminary draft outline of the team’s report; and  assign drafting responsibilities for the final report.  The second planning meeting will be at USAID and will be the initial briefing with PMI/Tanzania, AOR, the Mission M&E team, the Mission Gender Focal Person, and other members of the Health Team Health Team to allow PMI to present the team with the purpose, expectations, and agenda of the assignment. The evaluation team will present an outline and explanation of the design and tools of the evaluation.  The third meeting will be with USAID and PMI/Tanzania to present the preliminary findings and implementation challenges; and  The final meeting is for the evaluation debrief/summary of the data, draft recommendations and report. PMI Headquarter Consultations  The External Evaluation Team is encouraged to make consultations with PMI Headquarters before or during the evaluation process. Consultations may include:  PMI Washington (Tanzania back stop) Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 65  PMI CDC (Tanzania Entomology Expert)  Bureau for Environmental Management  Field visits/Key Informant Interviews: Key informant interviews at national, district, and community level will be conducted as required. The evaluation team will conduct interviews with key respondents identified during the planning meeting. Field visits will include travel within the Mainland in both Dar es Salaam and Lake Zone Region (northern Tanzania) and Zanzibar.  The Interview key informants will include PMI, RTI, ZMCP, NMCP, Ministry senior management on mainland and Zanzibar, RTI sub-partners, District Medical Officers, District Local government (DMOs and Administrators), the private sector working in vector control, and community leaders in IRS areas of operation.  The evaluation team should verify program achievements and collect both quantitative and qualitative data that demonstrate program performance. Gender representation must be taken into consideration.  A representative number of sites in Zanzibar and Mainland will be selected. in consultation with the PMI team. The evaluation team may solicit input from RTI, if needed.  The evaluation team shall arrange to visit selected sites supported through RTI TVCSP in consultation with the AOR and RTI in country leadership. The selected sites should include a representative sample of activity sites and should represent variety along the following dimensions: geographical location; technical focus of activities; sub-partner types and level of RTI TVCSP support.  The evaluation team will be accompanied by a member of staff from PMI/Tanzania and/or NMCP. The site visits will involve interviews with District Health Management Teams, and targeted beneficiary groups. The purpose of these site visits is to gain a better understanding of the technical competence of RTI TVCSP staff and sub-partners, the constraints encountered in the various categories of activity implementation, and key target audiences’ perception of their needs in order to provide quality services. The Evaluation Team may observe the spraying of houses if the evaluation coincides with the spray season.  The evaluation team should outline key meetings to coordinate post field visits in order to share findings and get final inputs before preparing the report. Data Analysis: The report should include both qualitative and quantitative analysis of the achievements in relation to the objectives and targets for the output indicators for the cooperative agreement. Wrap up and Debriefing: At the conclusion of the field visits/key informant interviews, there will be a debrief meeting at PMI/Tanzania, and also with NMCP and key malaria stakeholders. The purpose of the meeting will be to share preliminary findings and get final inputs before preparing the draft assessment report. 7. Team Composition The mission will seek the services of a four-person consultant team: one expatriate team leader, one expatriate team member, and two local technical consultant team members. Team Leader/ Malaria/Vector Control Expert/(International): One person with a minimum of 10 years of experience in public health, preferably with technical knowledge and experience of malaria control intervention preferably vector control. The team leader will be identified by USAID prior to the start of evaluation activities. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 66 2nd Malaria/Vector control Expert (Local): A second person with a minimum of 5-10 years of experience in Malaria control programs especially vector. This should include extensive technical knowledge and experience with interventions, policies and programs in malaria control programs. Monitoring and Evaluation /Disease Surveillance Expert: One person with relevant education and at least 5 years of experience in USAID project and organizational management. This person should have a minimum of 10 years of experience in designing, implementing, and managing international health programs at scale. This person should also have strong knowledge, skills and experience in qualitative and quantitative evaluation tools, as well as skills in environmental monitoring. This person should have excellent 10-15 years of experience in disease surveillance for infectious diseases, including knowledge and skills in entomological monitoring, epidemiological monitoring, and environmental monitoring, and gender programmatic integration. Capacity Building Expert: This person should have a post graduate degree in organizational development or health systems. S/he should have at least 5 years of experience with institutional capacity building and organizational development in developing countries. S/he should be knowledgeable in program assessment and evaluation methodologies in human resource and work plan development and training in information monitoring systems. S/he should have extensive experience, and demonstrate state-of-the￾art knowledge, in conducting programmatic evaluations/assessments. Two of the evaluation’s team members must be local Tanzanian with malaria and/or BCC experience. The Health Office M&E Officer may participate in the external evaluation. 8. Specific activities The team leader will be responsible for  Identification of specific team roles with individualized activities for each individual  Managing the team’s activities,  Ensuring that all deliverables are met in a timely manner,  Serving as the primary liaison with the Mission teams, and  Leading briefings and presentations. 9. Outputs and Deliverables The contractor deliverables shall include:  A proposed timetable for evaluation period with in two days of commencement of the in country work.  A written methodology plan and tools (evaluation design/operational work plan) during the pre￾evaluation meeting.  A proposed outline of the report on the findings and recommendations of evaluation with possible issues for discussion during the mid-evaluation meeting (within 2 weeks of arrival in-country.  A completed draft of evaluation report to PMI/Tanzania for presentation during the debrief meeting that will be held approximately 3 days before departure (see report format provided in Annex 2, Reporting Requirements). After the debrief meeting, the evaluation team shall incorporate oral comments received from PMI and stakeholders.  Draft report should be completed prior to the Team Leader’s departure from Tanzania. The written report should clearly describe findings, conclusions and recommendations (using the report format provided in “Reporting Requirements” below). PMI/Tanzania team will provide comment on the draft report within 5 working days of submission. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 67  A PowerPoint Slide Presentation of the key findings, issues, and recommendations before departure from Tanzania  A final report that incorporates the team responses to Mission comments and suggestions. The draft final report should be completed within 5 days after PMI provides its feedback on the draft report incorporating the comments received from the review of the draft and sent to the Mission. The report format should be restricted to Microsoft products and 12-point type should be used throughout the body of the report, with page margins 1” top/bottom and left/right. The report shall not exceed 30 pages, excluding references and annexes. After the final but unedited draft report has been reviewed by PMI, ASH will have the documents edited and formatted, and will provide the final report to PMI/Tanzania for distribution (2 hard copies and a CD ROM). It will take approximately 30 days for ASH to edit/format and print the final document.  List of all reviewed/cited sources in final report.  Annex 1 lays out reporting requirements and the report contents. 10. Qualifications  Minimum of 10 years of experience in public health, preferably with technical knowledge and experience with malaria control intervention preferably vector control.  Master’s degree or higher level of education in a relevant technical area, experience working in vector control program is desirable  Knowledge, skills, and experience with USAID contracting and reporting requirements; policies and initiatives; and tools, such as performance monitoring plans (PMPs) and results frameworks  Advanced written and oral communications skills in English  Expertise working in developing countries with decentralized health systems  Strong quantitative and qualitative analysis skills  Excellent skills in planning, facilitation, and consensus building;  Demonstrated experience leading an evaluation team;  Excellent interpersonal skills;  Excellent skills in project management  Excellent organizational skills and ability to keep to a timeline. 11. Level of effort PMI/Tanzania anticipates that the period of performance of this assessment will be approximately 31 days. This would include preparation days, in-country work in Dar es Salaam and the regions, and report writing and finalization. The in country assessment will take place TBD. The following is a sample timeline. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 68 Task/Deliverable Team Lead Malaria/Vector Control Expert (Local) Monitoring and Evaluation /Disease Surveillance Expert Capacity Building Expert(local) LOE LOE LOE LOE Review background documents & offshore preparation work. 4 4 4 4 Travel to Tanzania 2 TBD* TBD* TBD* Team planning meeting – including meetings w/ PMI/Tanzania and Health teams 2 2 2 2 Information and data collection/key informant interviews/site visits 17 17 17 17 Draft assessment report in-country- discussion, review of information, and analysis. 6 6 6 Report preparation and writing (in country) (preliminary draft report due to mission before departure from country) 2.5 2 2 2 Debrief meetings with PMI teams and key stakeholders 0.5 0.5 0.5 0.5 Depart Tanzania/travel 1 PMI provides comments on draft report (team out of country) – 5 days Team reviews comments and revises report – final report 5 2.5 2.5 2.5 PMI completes final review (10 working days) ASH edits/formats report (3-4 weeks) Total LOE days 40 35 35-38 35-38 * Travel time LOE will be determined whether international or local recruit Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 69 12. Logistic support The ASH Project will be responsible for all international travel and travel logistics. A six day work week is authorized when team is working in country. Local holidays are not authorized. Annex 1: Selected List of Background Materials  2007/08 Tanzania HIV/AIDS and Malaria Indicator Survey  Request for Applications (RFA) Number USAID-TANZANIA-07-005-RFA: IRS Scale-up in Tanzania (has Program Description)  CA # 621-A-00-10-00015-00: Scale up of IRS in Tanzania  RTI TVCSP Technical Application (volume 1): Scale up of IRS in Tanzania  RTI TVCSP revised Program Description  RTI TVCSP Annual Work plans  RTI TVCSP Quarterly and Annual Reports  RTI TVCSP IRS Reports  Entomology Special reports  Economic Analysis reports  Zanzibar and Mainland PERSUAPS and Justification Letters  Environmental management Ware House Inspection Report  PMI/Tanzania and Health Team strategy documents  Project products (deigned materials such as fliers, posters, publications etc.) Annex 2: Reporting Requirements  The findings from the evaluation will be presented in a draft report at a full briefing with PMI/Tanzania and possibly at a follow-up meeting with key stakeholders. The format for the evaluation report is as follows:  Executive Summary: concisely state the most salient findings and recommendations (not more than 2 pages);  Table of Contents (1 page);  Introduction: purpose, audience, and synopsis of task (1 page);  Background: brief overview of malaria and behavior change communication in Tanzania, PMI’s strategies and priorities, brief description of the TVCSP, purpose of the evaluation (2-3 pages);  Methodology: describe evaluation methods, including constraints and gaps (1 page);  Findings/Conclusions/Recommendations: for each objective area (15-20 pages);  Issues: provide a list of key technical and/or administrative issues identified (1-2 pages);  Future Directions/Recommendations (2-3 pages);  References (including bibliographical documentation, meetings, interviews and focus group discussions);  Annexes: evaluation methods, schedules, interview lists and tables – should be succinct, pertinent and readable, list of documents consulted, and SOW The report’s contents: The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 70  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females; children and pregnant women.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions.  Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 71 1. Dar es Salaam Annex 2: People Met National Malaria Control Program Dr Renata Mandike: Deputy Program Manager Yusufu Mwita: IRS Focal Person Dr Anna Maherudeka: M&E Officer Wilfred Mwafongo: M&E Officer/IDSR Focal Point Charles Dismas Mwalimu: Head, Integrated Malaria Vector Control Unit Linda Nakara: Head, BCC Unit Zudson Lucas: Program Administrator USAID Tanzania Dr. Jessica Kafuko: Senior PMI Technical Advisor Alisa Cameron: USAID/Tanzania Health Office Director Moses Busiga: M&E Specialist Angela Mwaikambo: M&E Specialist Gabriel Batulaine: Senior Environmental Management Specialist Ludovicka Taraimo: Gender Specialist Centers for Disease Control Tanzania Dr. Lynn Paxton: PMI Resident Advisor Chonge Kitojo: Program Specialist RTI International: Dar es Salaam Dr. Mahdi Ramsan: Chief of Party Mubita Lifwatila: Director of Finance and Administration Jeremiah Ngondi: Senior Epidemiologist Dr Stephen Magesa: Director, Vector Control Operations Shabbir Lalji: M&E Director Juma Ruta: Environmental Compliance Manager Amina Abdullah Awadh: Compliance Manager Zainab Aladina: M&E Officer (Knowledge Management) Vera Brown John: Senior Office Manager Emmanuel Likabaya: Logistician JHU COMMIT Jacob Macha: BCC COMMIT BCC Program Advisor 2. Zanzibar Zanzibar Ministry of Health Dr. Mohammed Jidawa: Permanent Secretary Dr. Mohammed Dahoma: Director, Preventive Services Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 72 Zanzibar Malaria Control Programme Dr. Abdullan Ali: Program Manager Juma Mcha: Head, Vector Control Unit Mohammed Y Mkanga: Assistant Head BCC Unit Riziki Suleiman: BCC Officer Jokha Salum: BCC Officer Amina Juma Faki: Assistant BCC Officer Mohammed Haji Ali: M&E Officer RTI International: Zanzibar Dr. Issa Garimo: RTI Zonal Manager Said Manmoud: Logistician Jaffar Abudalla: Logistician Nassor Said Nassor: Site Manager, North B District District administrators and health workers, Unguja Dr. Kalam Abdushir: District Medical Officer, West District Sabaah Salleh: District Administrative Officer, West District Ameir Ali: Acting Planning Officer, West District Habiba Suleiman Seif: District Malaria Surveillance Officer, West District Saida Hassan Faraji: Community Health Nurse, Shakani Health Center, West District Siti Mbarak Mohamed: Public Health Nurse 'B', Shakani Health Center, West District Hamisi Ame Saloum: District Administrative Officer, North B District Abdoulwahid Ahmed Mohammed: District Medical Officer, North B District Hamisi Amesalo: District Malaria Surveillance Officer, North B District Ramadhan Makame Khamis: Facility-in-Charge, Mahonda PHC Clinic, North B District Rehena Radhan Abdalla: SN/MO, Mahonda PHC Clinic, North B District Zamzam Makame Juma: Lab Technician, Mahonda PHC Clinic, North B District Bisiti Mohamed Ame: Orderly, Mahonda PHC Clinic, North B District Community Members, Unguja Female community members, Shakani Mitondo, West District Mwachum Juma Umanzi Isa Mariam Haji Hamisi Salma Mohamed Haji Mariamu Idrissa Hisani Suleiman Hatibu Male community members, Shakani Mitondo, West District Musli Amir Mikuri Ramadhani Musa Kombo Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 73 Haji Hassan Juma Khatibu Suleimani Khatibu Salim Amir Makuri Hussein Mohamed Bakari Female community members, Mahonda, North B District Nuhami Wanu Hamed Masiti Suleiman Shudu Hamis Kombo Khadija Alhaji Bakarai Ambiya Abdalla Msevu Male community members, Mahonda, North B District Musa Masanja Ussi Alhaji Bakari Hassan Haji Fumu Ally Othman Ally Mahabali Ally Issa Kinole Spray Operators, Unguja Mixed FGD, West District Maborouk Khamis Hamad (M) Hasi Abeid Hassan (M) Arif Alawi Makame (M) Hassan Hija Hassan (M) Mwalim Mwembamba Kwaom (M) Zainab Khamis Maalim (F) Alama Hasi Khaib (F) Talu Ngombo Juma (F) Zainab (F) Mixed FGD, North B District Maryam Rajab Mkinyi (F) Fatima Suleiman Iddi (F) Mklashamba Khatib Omar (F) Amisan Yussuf Haji (F) Nznhali Rajab Nassour (M) Mohammed Suleiman Juma (M) Juma Chui Mshenga (M) Bakar Fadhil Juma (M) Maulid Othman Khamib (M) Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 74 3. Lake Zone RTI International Lake Zone Joshua Mutagahaywa: Zonal Manager Dr. Michael Munishi: Epidemiologist, Lake Zone Ally Nyanga: Technical Specialist, Kagera Anold Mutafungwa: Data Specialist, Kagera Enos Emmanuel Mathias: Storekeeper, Kagera Eva Chanika: Office Manager Edwin Benedict: RTI Data Clerk, Mwanza Lorna Namata Mbalilaki: Storekeeper, Zonal warehouse Julieth Monary: Procurement and Logistics Assistant Lilian Charles: Site Manager, Bukoba Rural Alexander Tibenderwa: Site Manager, Muleba Philip Kuuli: Site Manager, Magu Emmanuel Benedict Tibalila: Site Manager, Koromite, Misungwi National Institute for Medical Research, Mwanza Dr Safari Kinunghi: Principal Scientist, Parasitology Dr Fabian Mashauri: Principal Research Scientist, Parasitology Jackline Lyino Martine: Research Scientist II and Coordinator IRS Epiphany Michael: Lab Technician, Insectary Michael Sadataley: HO/VC Kagera Region Regional administrators and health workers, Kagera Region Dr Khairoonisa Pathan: Acting Regional Medical Officer Andrea Rubwa: Regional Malaria and IMCI Focal Person Herman Kabirigi: Regional Environmental Health Officer District administrators and health workers, Kagera Region Kisanga Makigo: Acting District Executive Director, Bukoba Rural Dr. Hamza Mgula: District Medical Officer, Bukoba Rural Deodart Ngaiza: District Malaria and IMCI Focal Person, Bukoba Rural Paulina Mgunda: District Vector Control Officer, Bukoba Rural Leticia Mkome: District School Health Officer, Bukoba Rural Aisha Rugusha: District RH and Child Health Officer (IRS IEC Focal Person), Bukoba Rural Rehema Masubugu: District Data Clerk and IDSR Focal Point, Bukoba Rural Shela Kalaza: Nurse midwife, Kyamalange Dispensary, Bukoba Rural John Nnko: Acting District Executive Director, Muleba District Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 75 Dr. Joseph Kisana: District Medical Officer, Muleba Auxilius Mathew: District Health Secretary, Muleba Ernest Luumba: District Malaria and IMCI Focal Person, Muleba Mdabivu Kachwarimba: District Environmental Health Officer, Muleba Aristides Kyobya: District Vector Control Focal Person, Muleba Paula Paulo: District Data Clerk, Muleba Prayeda Nestory: Clinical Officer and Facility in Charge, Mabunda Dispensary, Muleba Community members, Kagera Female community members, Kyamalange village, Bukoba Rural Evelina Baraba Happiness Cyprian Najiat Francis Dorethea Joas Monica Charles Graciana Benjamin Male community members, Kyamalange village, Bukoba Rural Rangitoni Telepholi Tutalwa Faustin Angelo Katuali Petro Felix Charles Ndyebonela Female community members, Nyakasyeni village, Muleba District Sandelina Martin Elivia Dunstan Lourdes George Esther Donart Eudes Dominic Luciana Edmund Arodhia Anthon Cathalena Gaspari Appolonia Philippo Maria Saloum Edward Madelina Benedict Male community members, Nyakasyeni village, Muleba District Gastoni Stephani Ernest Edward Innocent Gasper Frederick Jeparus Gabriel Bwakafula Basil Tuoman Donath Mwaro Cleofas Baltazar Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 76 Spray Operators, Kagera Mixed FGD (Bukoba Rural) Gideon Kallemba (M) Audex Fidelis (M) Benjamin Kasilingi (M) Godbelta Albert (M) Hidaya Mikidadi (F) Severina Ngaiza (F) Renatus Tibakwa (F) Mixed FGD (Muleba) Regina Edward (F) Revima K Dionise (F) D. Biamanyilwohi-Mazige (F) Veronica Justiniani (F) Deoratha Clemenze (F) Hilda Mulokozi (F) Emmanuel Projesus (M) Johnson Kamuntu (M) Elson M Angelo (M) Vedasto Mutusi (M) Site IEC Mobilizers Edward Mwemesi: SIM, Bukoba Rural Mwanza Region Regional Administrators and Health Workers, Mwanza Region Ndaro Kylwijila: Acting Regional Administrative Secretary Dr. Valentino Francis Bangi: Regional Medical Officer Paula Beichuma: Regional Malaria and IMCI Focal Person Charles Simon Bundu: Regional Health Officer District Administrators and Health Workers, Mwanza Region Dr. Athumai Pembe: District Medical Officer, Magu District Joseph Mandago: District Malaria and IMCI Focal Person, Magu Kaisaria Mambea: District IEC Officer, Magu District Environmental Health Officer, Magu District Storekeeper, Magu Dotta Isaafa Ideloi: Clinical Officer, Lumesi Dispensary, Magu Sabina James: Enrolled Nurse, Lumesi Dispensary, Magu Pastory Mkaruka: Acting District Executive Director, Misungwi District John Nyorobi: Acting District Medical Officer, Misungwi Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 77 Dismas Dotto: Malaria and IMCI Focal Person, Misungwi Grace Mgeta: District IEC Officer, Misungwi Lucas Alex: M&E Officer, Misungwi Raphael Matalu: Facility in Charge, Koromije Health Center, Misungwi Community Members, Mwanza Female community members, Lumije village, Magu District Mariam Ferdinand Paulina Buswelu Happiness Ezekiel Gaudensia Joseph Georgia Francis Regina John Male community members, Lumije village, Magu District Samuel Iseko Enos Yohana Charles Mathayo Charles Samandali William Kashirima Philip Makoye Kawa Female community members, Koromije village, Misungwi District Venera Misango Kabula Saanane Rosemary Kengwa Leah Paulo Rachel Matthia Male community members, Koromije village, Misungwi District Michael Kahema Lukija Maahola Wilson Maningu Chitobela Nyembe Kinga Benjamin Kahema Nestory Joseph Kilasila S Nyanda Spray Operators Mixed FGD (Magu District) Paul M Lugwiha (M) Naabu Samuel (M) Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 78 Ngholombi Yangio (M) Luca M Buganda (M) Paolo Lywobgu (M) Masalu Mazoya (M) George P Msoga (M) Pauline Kulwa (F) Sophia Kazimili (F) Valentina Paulo (F) Helena Daudi (F) Joyce Bukelebe (F) Scholastica Mabula (F) Mixed FGD (Koromite, Misungwi) Kulwa Mathelus (F) Elizabeth Geluas (F) Helena Hyeji Gwafaya (F) Anna Wilson Remda (F) Esther Kamembe Madaha (F) Juliana John Nicodemo (F) Mussa J Nghologoto (M) George Mayige Kamhayo (M) Timothy Misuneku (M) Mathias Manyanza (M) Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 79 Annex 3: TVCSP Mid-Term Performance Evaluation: Evaluation Plan TVCSP MID-TERM PERFORMANCE EVALUATION EVALUATION PLAN (Final) Ikupa Akim John Govere Janet Gruber Billy Ngasala Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 80 TABLE OF CONTENTS 1. Introduction 2. Definition of a Performance Evaluation 3. Evaluation Objectives 4. Evaluation Methodology 5. Evaluation Limitations 6. Evaluation Work Plan and Time Line 7. List of People to be Met 8. Evaluation Report Structure and Team Writing Responsibilities 9. Evaluation Data Collection Tools Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 81 TVCSP MID-TERM PERFORMANCE EVALUATION: EVALUATION PLAN (Final) 1. INTRODUCTION The Tanzania Vector Control Scale-up Project (the TVCSP) is managed by RTI and funded by USAID, to the value of $56 M to end Financial Year (FY) 2012. RTI works in partnership with the National Malaria Control Program (the NMCP: for Mainland) and the Zanzibar Malaria Control Program (the ZMCP), with regional and district governments and health structures and with communities to implement indoor residual spraying (IRS), entomological and environmental monitoring and limited epidemic malaria surveillance activities in Mainland and Zanzibar. The background to the TVCSP is that RTI assisted ZMCP with its IRS program from 2006, contributing to a substantial drop in new malaria infections. RTI assisted the ZMCP to establish a malaria early epidemic detection system (MEEDS). RTI began supporting IRS in Mainland Tanzania in 2007, assisting the Ministry of Health and Social Welfare (MoHSW) to control malaria outbreaks in Karagwe and Muleba districts in Kagera Region. In 2009, operations were scaled up to spray in the remaining stable and high transmission areas. The TVCSP goal is to contribute to the reduction of the burden of malaria by interrupting malaria transmission through IRS and the prevention of adverse effects of malaria epidemics through focal spraying. The TVCSP originally addressed five objectives in 2010: 1. Scale up IRS in Tanzania by expanding IRS coverage from one region to six or more regions on Mainland and maintain required IRS, services as required in Zanzibar. 2. Establish an epidemic detection and a focal spraying response strategy for Zanzibar and IRS districts on Mainland. 3. Develop an environmental compliance strategy and monitoring plan for Zanzibar and Mainland. 4. Develop capacities of ZMCP, NMCP, and districts to plan and implement IRS, including logistics planning, environmental compliance, and epidemic response through focal spraying. 5. Establish a viable and sustained entomological monitoring system on Mainland and Zanzibar. A 6th objective has subsequently been added: 6. Develop an epidemic surveillance system on Mainland and Zanzibar. The major components of the TVCSP are: IRS, environmental compliance; entomologic monitoring; epidemic surveillance and response; capacity building; cross cutting activities of gender and cross border collaboration; and monitoring and evaluation. 2. DEFINITION OF A PERFORMANCE EVALUATION The TVCSP mid-term evaluation will be a performance evaluation. Its overall purpose will be: 1. to determine the extent to which the RTI Tanzania TVCSP objectives and key result areas are being met; 2. to document lessons learned to inform the 2nd phase of the project; and 3. to assess the various approaches used by the TVCSP to determine what is working [in order] to achieve the intended results Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 82 The 2011 USAID Evaluation Policy defines performance evaluations as focusing on ‘descriptive and normative questions: what a particular program or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring and other questions that are pertinent to program design, management and operational decision-making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual [as would be the case for an impact evaluation]’. Thus a mid-term performance evaluation focuses on process and implementation fidelity as it relates to achieving program goals, objectives and activities as they were envisioned, in addition to looking at some intermediate outcomes. The evaluation will assess the relevance of the program objectives, the extent to which these are being achieved at the mid-point and the efficacy, efficiency and relevance of the TVCSP. Another core aspect of a mid￾term performance evaluation is to provide action-oriented, evidence-based findings, conclusions and recommendations for the remainder of a project or program and to look forward to any envisaged successor intervention. Attention to the potential sustainability of program/project inputs and outcomes is another aspect of a performance evaluation. The UNAIDS M&E Reference Group (MERG) definition of performance is also relevant: the degree to which an intervention has operated according to specific criteria, standards or guidelines, and/or achieves results in accordance with stated plans. The intention of the evaluation is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. The evaluation will focus on the provision of information that is credible and useful, enabling the incorporation of lessons learned into planning and decision-making processes for the remainder of the project (until end February 2015), for sustainability of TVCSP activities and capacity development and in the design of future projects/programs. The TVCSP evaluation team will also be guided by the essential core evaluation activities as set out in the 2011 PMI Technical Guidance on M&E Activities [2011 PMI M&E MOP Guidance]: a. Estimating outcomes, coverage of malaria control interventions and associated behavioral outcomes b. Estimating the impact of malaria control interventions on malaria-related morbidity and mortality. The understanding of the TVCSP evaluation team is that because this is a mid-term performance evaluation, findings and recommendations cannot consider the higher-level impact of the project. Moreover, issues of direct impact attribution will not be addressed. The evaluation will provide findings and recommendations as to the way forward for the remainder of the TVCSP, to end-February 2015. Its findings and recommendations will also address post￾TVCSP programming. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 83 3. EVALUATION OBJECTIVES This external evaluation shall:  Determine the extent to which the achievements have been made or not in relation to the expected results of RTI TVCSP since the beginning of the program.  Determine progress made in the capacity building and institutional strengthening of the Ministry of Health and Social Welfare (MOHSW)/NMCP and Zanzibar MOH ZMCP.  Establish how the project is perceived by the NMCP, ZMCP, Local Governments, and the Community.  Determine the ways in which RTI TVCSP is addressing gender gaps.  Assess the processes put in place to address human and environmental safety of the program, in relation to the US Government and local environmental laws.  Describe any cost savings and any interventions made to reduce the cost of the structures sprayed and people protected.  Review the documentation to assess the impact of IRS on the entomology and epidemiology in the IRS districts.  Determine the extent to which RTI TVCTS has addressed insecticide resistance of the IRS program.  Identify the lessons learned from the RTI TVCSP since the beginning of the program taking into account the perspectives of the stakeholders, the donors and the beneficiaries.  Identify areas for implementation improvement and propose key activities for the follow-on program. 4. EVALUATION METHODOLOGY This part of the Evaluation Plan is guided by the points set out in section 4 of the Scope of Work (SOW), which describes the required approach of the TVCSP mid-term performance evaluation. As such, this performance evaluation will address the following evaluation questions, as set out in the SOW. These are the key evaluation questions as set out in the SOW:  What have been RTI TVCSP key achievements so far?  To what extent are the RTI TVCSP objectives likely to be achieved?  How is the project perceived and valued by the United Republic of Tanzania (MOHSW on mainland, MOH in Zanzibar, NMCP, ZMCP, Local governments in IRS areas of operation) and key malaria stakeholders in Tanzania?  In what ways has RTI project integrated gender considerations into its activities and how have gender issues been addressed in implementation of the program?  What approaches has RTI used and what is working towards the achievement of the intended results and improve acceptability of IRS?  Are there specific lessons from the project that can be applied in the second phase of the program?  Is the program addressing sustainability of the interventions and how? 4.1 Conceptual Methods and Approaches In order to address the evaluation questions, this performance evaluation will apply the following conceptual methods and approaches, guided by the points discussed in the SOW. Optimal balance of qualitative and quantitative data collection and analysis Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 84 The evaluation objectives and questions require a mix of quantitative, numerical data analysis and more in-depth descriptive and normative questions to be addressed through qualitative data collection and analysis. The evaluation team will use a mixed method evaluation approach, including desk review of existing quantitative and qualitative documentation (project and relevant national documents), gathering qualitative data through fieldwork and applying quantitative and qualitative data analysis. Quantitative data will primarily provide insights on outputs and potential outcomes of activities, while qualitative data will inform evaluation of the relative effectiveness of approaches, processes and activities employed to achieve project objectives. The key methods to be used throughout will be:  Key Informant Interviews with one person at a time (qualitative).  Focus Group Discussions (qualitative).  Direct Observation – at IRS sites, in health facilities, etc.  Review of TVCSP data (quantitative and qualitative).  Review of other relevant documentation, e.g. M/ZMCP strategies and statistics, GFATM reports. Because this is a mid-term, performance evaluation, with the USAID required focus on descriptive and normative approaches, the emphasis in data collection will be on qualitative methods. Semi-structured interview guides have been developed by the evaluation team: these allow respondents to reply individually, with probing questions as follow-up to ascertain detailed understanding and opinions. Semi-structured interviews also allow respondents to shape discussion as well as the moderator, which can result in richer data. The evaluation team is mindful of the fact that there are at best 15 days for fieldwork activities, which will include visits to Zanzibar and project locations in the Lake Region, in addition to interviews in Dar es Salaam. There will be 6 dedicated days for analysis of fieldwork data, while there will be ongoing, top line analysis while in the field. Focus will be on achieving the best data collection and analysis in the limited time available. The evaluation team will independently review the project/RTI database in order to provide an evidence base, and to triangulate and validate its own findings. The overall efficacy of the TVCSP Program Monitoring Plan (the PMP) as a tool to collect and manage monitor program process and progress will be assessed. The PMP performance indicators will be reviewed and the data source for each indicator considered; so too the management of that data collection, e.g. through application of the results framework. The extent to which the PMP and TVCSP data management support achievement of the N/ZMCP, USAID and PMI objectives and goals will be evaluated, as will progress towards achievement of Intermediate Results. Priority will be given to addressing the extent to which the TVCSP is achieving its targets and expected results at this mid-point and the degree of progress on indicators, all as listed in the SOW and the PMP. The evaluation team will triangulate its own findings against TVCSP quantitative and qualitative data. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 85 Avoidance of bias The evaluation team will maintain its independence of data collection, analysis and findings throughout the assignment. So as to avoid bias in evaluation data collection, the team has developed qualitative, semi-structured key informant interview (KII) and focus group discussion (FGD) guides for the discussions that will be conducted with TVCSP stakeholders, such as NMCP, ZMCP and RTI staff members, district health workers, community members, etc. Focus will be on collection of evidence-based data, balanced by individual respondents' perceptions and analyses, as is appropriate in a performance evaluation. See section 9 of this Evaluation Plan for the data collection tools. The use of standardized data collection tools will enable individual team members to be consistent in data collection, addressing the same questions across all interviews and with all respondents, thereby reducing the potential for variability in approach and discussion. The team will ensure coherence of unbiased and objective analysis by holding field evening meetings, where top line analysis will be done and any relevant issues discussed. In addition, attention will be given throughout fieldwork to any emerging limitations to the evaluation. Any such will be disclosed in the report, with particular attention to any limitations associated with the evaluation methodology (e.g. lack of opportunity to interview key stakeholders). As required by USAID, evaluation findings will be presented as analyzed facts, evidence and data and will not be based on anecdotes, hearsay or the compilation of people’s opinions. Findings and recommendations will be specific, concise and supported by strong quantitative or qualitative evidence. The evaluation team will seek at all times to provide sufficient information so that readers of the evaluation report can make an informed judgment as to the reliability, validity and generalizability of findings and recommendations. The team will evaluate the quality of secondary data (TVCSP and other sources) that have already been collected, in addition to ensuring the quality of fieldwork data collection. The quality criteria as set out in Appendix 1 of the 2011 USAID Evaluation Policy, will be applied throughout. Triangulation The approach of this evaluation will be to triangulate findings and analysis wherever possible, so as to minimize bias and provide as robust an evidence base as is feasible given the limited time available. Triangulation will be addressed through document review (including TVCSP data, MEEDS, etc as well as routine and non-routine HMIS), qualitative and/or quantitative data collection methods and participant observation. Triangulation requires the use of multiple methods and this approach will be applied will be applied in the TVCSP performance evaluation, through the use of Key Informant Interviews, Focus Group Discussions, Direct Observation and secondary data review. Triangulation also serves to reduce the potential for bias. As is noted in the 2010 USAID document entitled Performance Monitoring and Evaluation TIPS: Using Rapid Appraisal Methods, triangulation increases the validity of findings, thereby contributing to the efficacy of conclusions and recommendations. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 86 The OECD-DAC definition of triangulation is: ‘The use of three or more sources or types of information, or types of analysis, to verify and substantiate an evaluation’. Gender-sensitive data collection and analysis The evaluation team recognizes the importance of applying a gendered perspective in its evaluation of the TVCSP. The team is fully aware of the particular biological vulnerabilities of pregnant women, infants and children under five and acknowledges the relevance of gender norms and roles specific to vulnerability to malaria. All evaluation activities will be sensitive to gender aspects of the project in relation not only to its target populations, but in terms of the degree to which TVCSP activities have been sensitive and responsive to relevant gender issues, e.g. in capacity development. The evaluation team proposes that it will seek to review gender issues relevant to all TVCSP activities. Such review will be dependent on available project data as well as on through discussion during evaluation key informant interviews and focus group discussions, individuals’ and organizations’ perceptions as revealed in data collection, and also participant observation. All analysis of evaluation data will be informed by gender perspectives. 4.2 Analytical Approaches The over-arching approach to the mid-term evaluation data analysis is what has been termed a ‘general realist approach’, while applying the iterative methods set out below. The evaluation process will be guided by review of the TVCSP Performance Monitoring Plans (original and updated in December 2012), the Results Framework and the progress made to date by TVCSP towards the expected Intermediate Results. Attention will be paid to the implicit development hypothesis of the TVCSP, interpreted by the evaluation team as being the same as the RTI Strategic Objective: Malaria transmission reduced in IRS districts. In addition, the evaluation team will be attentive towards the Theory of Change underpinning the TVCSP: what the project intends to achieve and the results that project activities are expected to produce. All such focus will inform how the evaluation addresses the seven key questions, as set out in the SOW. The evaluation team will apply four distinct analytical steps: The design and implementation of data collection approaches and ongoing topline analysis; Separate analysis of qualitative and quantitative data; Triangulation of quantitative and qualitative results; and, Interpretation of the merged results This is a performance evaluation: as is described in the 2011 USAID Evaluation Policy, such evaluations are primarily qualitative and address descriptive and normative questions. Bearing this in mind, and with due attention to the realities of the time available for fieldwork and analysis, both qualitative and quantitative data analytical approaches will be applied, in parallel. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 87 Furthermore, the use of both qualitative and quantitative methods will enable validation and corroboration of the results. Parallel analysis will be applied, with a small degree of what is called multi-level analysis. Analytical Method Description Application Parallel analysis Several data sets (e.g. data from KII and TVCSP data sets) are analyzed separately, reviewed jointly and findings, conclusions and recommendations developed Triangulation approach key in this regard Qualitative data lead in parallel analysis, with quantitative data used to back up/substantiate or indeed qualify other analysis Multi-level analysis Both qualitative and quantitative analytical methods are applied This approach allows deeper emphasis on quality of interventions, through two-level data collection and analysis, e.g. of information provided by TVCSP implementers and community members The qualitative data from the various FGD and KII sessions may also be analyzed by applying a Thematic approach. In adopting this approach, the evaluation team will develop a list of potential themes during the fieldwork phase, corresponding to the evaluation questions. The FGD and KII data will be analyzed according to any emerging themes. Iterative analytical approaches will be used to triangulate and integrate data from TVCSP data, evaluation fieldwork data collection and other sources (potentially including N/ZMCP and other relevant policies, reports, DHS, MICS, research studies, etc) in the development of findings and recommendations. Data analysis will be determined by both the evaluation objectives (deductive) and multiple readings and interpretations of the raw data (inductive). Thus findings will be derived from both the evaluation objectives and questions and from analysis of the raw data. 5. EVALUATION LIMITATIONS At the time of writing (09/22 2013), the evaluation team has not observed any limitations. Should these emerge during the course of the evaluation, they will be discussed in the evaluation report. 6. EVALUATION WORK PLAN AND TIME LINE This the final work plan and time line. The choice of districts in Kagera was influenced by the wish to balance fieldwork in a relatively high performing district in Kagera (Bukoba Rural, to balance Muleba district, where a recent malaria spike has occurred. In addition Muleba is the first district where the TVCSP began its IRS operations in 2010. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 88 Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sept 2013 9 10 Review background documents and preparation work 11 Review background documents and preparation work 12 Review background documents and preparation work 13 Review background documents and preparation work 14 Travel to TZ (Janet G) 15 Travel to TZ (Janet G)/ OFF 16 Review background documents and preparation work 17 Review background documents and preparation work; travel to TZ (John G) 18 Team Planning Meeting (TPM) 19 TPM and Interviews/ fieldwork (day 1) Dar es Salaam RTI 20 Interviews/ fieldwork (day 2). Dar es Salaam USG RTI 21 TPM/ Interviews/ fieldwork (day 3) Dar Team work 22 OFF 23 Interviews/ fieldwork (day 4). Meeting 2 USG Dar NMCP USG Fly to Zanzibar 24 Interviews/ fieldwork (day 5) Zanzibar ZMCP MoHSW RTI TVCSP West District District staff RTI Health workers Community members & spray operators 25 Interviews/ fieldwork (day 6) Zanzibar TVCSP North B District District staff Health workers Community members & spray operators Fly to Dar 26 Interviews/ fieldwork (day 7) Dar NMCP and RTI 27 Interviews/ fieldwork (day 8) Dar NMCP and RTI; other meetings 28 Interviews/ fieldwork (day 9) Interview analysis 29 Fly to Kagera 30 Interviews/ fieldwork (day 10) Kagera RTI Regional staff Oct 2013 1 Interviews/ fieldwork (day 11) RMO 1st Kagera District (Bukoba Rural) District staff Health workers Community members & spray operators 2 Interviews/ fieldwork (day 12) 2nd Kagera District (Muleba) District staff Health workers Community members & spray operators 3 Interviews/ fieldwork (day 13) Fly to Mwanza RAS, RMO & other regional admin-iterators 1st TVCSP District (Magu) District staff Health workers Community members & spray operators 4 Interviews/ fieldwork (day 14) Mwanza NIMR, Mwanza 2nd TVCSP District (Misungwi) District staff Health workers Community members & spray operators 5 Interviews/ fieldwork (day 15) Mwanza RTI meetings Travel to Dar Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 89 Sunday Monday Tuesday Wednesday Thursday Friday Saturday 6 OFF 7 Dar Draft evaluation report: analysis. 8 Dar Draft evaluation report: analysis 9 Dar Preparation of outbrief 10 Dar Preparation of outbrief 11 Dar Outbrief at USAID Draft evaluation report: analysis 12 Dar Draft evaluation report: analysis 13 OFF 14 Dar Draft evaluation report preparation 15 Dar Draft evaluation report preparation 16 Dar Draft evaluation report preparation 17 Dar/travel Draft evaluation report preparation. Travel (John G and Janet G) 18 Draft report preparation and submission > by SOB 10/21 19 20 21 PMI provides comments 22 PMI provides comments 23 PMI provides comments 24 PMI provides comments 25 PMI provides comments 26 27 28 Final report (5 days) 29 Final report (5 days) 30 Final report (5 days) 31 Final report (5 days) Nov 2013 1 Final report (5 days) Date TBC: debrief to PMI DC Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 90 7. LIST OF PEOPLE TO BE MET: DRAFT 1 Please note the following:  This is a preliminary and extremely comprehensive list;  In order to achieve the best quality of detailed information for the evaluation, it is unlikely the evaluation team will meet each person/organization in each fieldwork location. The evaluation team will work towards achieving a representative overview, as per the requirements for an unbiased, independent, mid-term, performance evaluation;  The evaluation team discussed this list and fieldwork locations during the in-briefing meeting with USAID, PMI and CDC, and at the 1st and 2nd meetings with RTI. Further discussions will be held with USG, the N/ZMCP, RTI and other relevant stakeholders, as advised. As discussed in the SOW and the 2011 PMI Technical Guidance on M&E Activities, the evaluation will interview as many key TVCSP stakeholders as is possible during the time available for fieldwork. Friday 20th September 2013 08.00 USAID, PMI in-brief As advised Monday 23rd USAID and PMI discussion of evaluation plan, etc As advised Friday 20th and on additional days NMCP The following staff members who are responsible for: Program Manager: management of/liaison with TVCSP Program Administrator: Policy and Planning - specific to longer term integration of project activities with NMCP, plus sustainability focus Head Vector Control Unit Epidemiologist? Entomologist? IRS Focal Person Environmental Management Focal Person Capacity Development i/c - ? 1 or both the 2 SDC-funded technical Advisor/s - under NATNETS/NETCELL and Case Management Unit? Regional and district link - under program administrator Head of M&E Unit MEEDS Focal Person: if there is such a post Head Health Education/Behavior Change Communication (BCC)/IEC Unit Community mobilization/engagement Focal Person Ministry of Health and Social Welfare Where applicable, and based on the N/ZMCP and ministry division of labor specific to malaria, the following staff members who are responsible for: Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 91 Malaria programs/activities under the aegis of the MoHSW Liaison with the N/ZMCPs Liaison with the TVCSP M&E: e.g. to discuss integration of routine HMIS and project-specific data management Environmental Health Health Education Gender Etc CDC: To be advised RTI - national TVCSP CoP/Manager Dr Stephen Magesa: Vector Control Operations' Director Entomologist? M&E Officer responsible for TVCSP; if another person, also RTI officer resp. for MEEDS co￾ordination Environmental Control Officer (this must include waste management?) Officer resp. for warehousing RTI officer responsible for gender and malaria issues RTI officer resp. for Behavior Change Communication/IEC/community mobilization Training officer/Capacity Development - e.g. training of spray operators, clinicians, etc Appointments outside Dar es Salaam ZMCP As for NMCP ZMoHSW As for Mainland MoHSW Regional Level RMO Regional Malaria Focal Person Regional Health Officer Regional Health Promotion Officer District Health At district central level DMO Malaria/IMCI Focal Person If another staff member, person directly responsible for district oversight of TVCSP activities Environmental Health M&E Health Ed/BCC/IEC Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 92 Community Health Gender District Reproductive and Child Health Officer (DRCHCO) At health facility level Facility in Charge Nurses /clinicians at health facilities in districts where TVCSP spraying has been conducted ? Facility Management Committee - both for community perspective and to meet some of the community leaders ? ANC/MCH clinic staff member Other District Officers? RTI - regional and district In Dar or at the Lake/on Zanzibar Zone Managers (Mr. Joshua Mutagahywa, Lake Region ZM plus ?) - are these the same posts as Regional Team Leaders? IRS Co-coordinators Field Technical Managers for Mwanza and Zanzibar District Focal Person District/regional vector control officer District-level support staff RTI also has Team Leaders (called Site Managers?) and Malaria Focal Points at regional (and district?) levels. NB: it may be that these posts are covered by others on the list above; please advise. Environmental Officer Community Mobilization Officer Gender Focal Point Warehouse Manager National Institute of Medical Research NIMR head of Vector Control? Person who has led/is leading the National Survey of Malaria Vector Resistance to Insecticides (discussed in TVCSP QR April-June 2013) Plus? Private Spraying Companies partnering the TVCSP? Manager responsible Spraying operators (male and female) MEEDS Selcom Wireless rep MEEDS-trained health workers Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 93 Communities Male members of households that have received IRS Female members of households that have received IRS Male members of households that have not received/have rejected IRS Female members of households that have not received/have rejected IRS Community leaders Community Health Workers Community-based organizations active in project/IRS BCC activities Also? GFATM Other donors/national or international researchers active in malaria Ministry of Agriculture National Environmental Management Council 8. EVALUATION REPORT STRUCTURE AND TEAM WRITING RESPONSIBILITIES 8.1 Report format and contents The format for the evaluation report is currently as follows, as per the evaluation SOW. Changes may be made as a result of fieldwork and team deliberations during analysis. Any such changes will be communicated to USAID, PMI and CDC. During discussion with USAID and PMI on Friday 20th September, the following agreement was reached on two points: 1. The definition of 'each objective area' in the SOW regarding the section in the evaluation report on Findings/Conclusions/ Recommendations will be interpreted to mean that the team in its report addresses the evaluation questions as set out in the SOW, linking to the 6 TVCSP objectives and the evaluation objectives. We shall ensure that all objectives and questions are appropriately considered and discussed. 2. The evaluation team will look in depth at the three higher level questions of environmental concerns, how capacity has been built and whether/how the project and its partners and stakeholders are looking forward with regard to action on sustainability.  Executive Summary  Table of Contents  Introduction: purpose, audience, and synopsis of task  Background: brief overview of malaria and behavior change communication in Tanzania, PMI’s strategies and priorities, brief description of the TVCSP, purpose of the evaluation  Methodology: describe evaluation methods, including constraints and gaps  Findings/Conclusions/Recommendations: for each objective area (15-20 pages).  Issues: provide a list of key technical and/or administrative issues identified (including evaluation limitations/changes to the SOW);  Future Directions/Recommendations  References (including bibliographical documentation, meetings, interviews and focus group discussions)  Annexes: evaluation methods, schedules, interview lists and tables, list of documents consulted, and SOW Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 94 8.2 Team Report Writing Responsibilities (IA: Ikupa Akim; John G: John Govere; Janet G: Janet Gruber; BN: Billy Ngasala) 1. TVCSP Objectives Scale up IRS in Tanzania by expanding IRS coverage from one region to six or more regions on Mainland and maintain required IRS, services as required in Zanzibar. All Establish an epidemic detection and a focal spraying response strategy for Zanzibar and IRS districts on Mainland. BN and IA for epidemic detection and John G for focal spraying Develop an environmental compliance strategy and monitoring plan for Zanzibar and Mainland. John G to lead Develop capacities of ZMCP, NMCP, and districts to plan and implement IRS, including logistics planning, environmental compliance, and epidemic response through focal spraying. IA and Janet G to lead Establish a viable and sustained entomological monitoring system on Mainland and Zanzibar. John G to lead plus all Develop an epidemic surveillance system on Mainland and Zanzibar. IA and BN to lead 2. Evaluation Objectives This external evaluation shall:  Determine the extent to which the achievements have been made or not in relation to the expected results of RTI TVCSP since the beginning of the program; Janet G to lead, with team inputs  Determine progress made in the capacity building and institutional strengthening of the Ministry of Health and Social Welfare (MOHSW)/NMCP and Zanzibar MOH ZMCP; IA to lead with Janet G and others  Establish how the project is perceived by the NMCP, ZMCP, Local Governments, and the Community IA and Janet G  Determine the ways in which RTI TVCSP is addressing gender gaps Janet G to lead  Assess the processes put in place to address human and environmental safety of the program, in relation to the US Government and local environmental laws. John G and BN to lead  Describe any cost savings and any interventions made to reduce the cost of the structures sprayed and people protected. John G and Janet G to lead  Review the documentation to assess the impact of IRS on the entomology and epidemiology in the IRS districts John G to lead on entomology and BN on epidemiology)  Determine the extent to which RTI TVCSP has addressed insecticide resistance of the IRS program. John G to lead with BN  Identify the lessons learned from the RTI TVCSP since the beginning of the program taking into account the perspectives of the stakeholders, the donors and the beneficiaries; and Janet G to lead on both, with all team members contributing  Identify areas for implementation improvement and propose key activities for the follow￾on project. Janet G to lead on both, with all team members contributing Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 95 3. Evaluation Questions This performance evaluation will address the following key questions:  What have been RTI TVCSP key achievements so far? Janet G to lead, with all team members contributing  To what extent are the RTI TVCSP objectives likely to be achieved? Janet G to lead, with all team members contributing  How is the project perceived and valued by the United Republic of Tanzania (MOHSW on mainland, MOH in Zanzibar, NMCP, ZMCP, Local governments in IRS areas of operation) and key malaria stakeholders in Tanzania? IA and BN to lead, with Janet G and John G contributing  In what ways has RTI project integrated gender considerations into its activities and how have gender issues been addressed in implementation of the program? Janet G to lead  What approaches has RTI used and what is working towards the achievement of the intended results and improved acceptability of IRS? BCC: Janet G/IRS strategies: John G , etc - processes and crosscutting - each team member's lead questions will inform this question  Are there specific lessons from the project that can be applied in the second phase of the project, and to other PMI programs and countries? Each team member's lead questions will inform this question.  Is the program addressing sustainability of the interventions and how? Each team member's lead questions will inform this question. 9. EVALUATION DATA COLLECTION TOOLS Introduction: for all respondents I am part of a team conducting an evaluation of the Tanzania Vector Control Scale-up Project, which conducts Indoor Residual Spraying and other activities to interrupt malaria transmission. We have come to see you today to ask your opinion on a number of questions related to IRS and malaria and would welcome your contributions. Those contributions will be analyzed by the team to provide a report on the project activities, achievements and challenges since it began in March 2010. Our discussion will take at the very most 1 hour. Your comments and responses will be confidential - we are an independent team and will not disclose any person's name or other personal details. For community members and community spray operators: please ensure that each individual respondent gives his or her verbal consent before the focus group discussion begins. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 96 =============================================================== DATA COLLECTION INSTRUMENTS TOOL 1. KEY INFORMANT INTERVIEW (KII) FOR RTI EMPLOYEES WORKING ON TVCSP Part I: core questions for all employees and component-specific questions dependent on role and responsibilities Question 1 Please tell me about your role and responsibilities on the TVCSP [Probe on how long the individual has worked on the TVCSP]. Question 2 (Evaluation Questions: What have been RTI TVCSP key achievements so far? and To what extent are the RTI TVCSP objectives likely to be achieved?) Please tell me what you feel are the key achievements of the TVCSP to date and to what extent do you think the objectives have been achieved to date and will be achieved by the end of the project. [Probe on this question, to elicit in-depth response] Question 3 (Evaluation Questions: What have been RTI TVCSP key achievements so far? and To what extent are the RTI TVCSP objectives likely to be achieved?) Please tell me about any TVCSP challenges you have personally experienced in your role and any wider TVCSP challenges you have observed. Question 4 (Evaluation Question: What approaches has RTI used...) Tell me about the RTI /TVCSP approaches to its work on IRS; how do you think these approaches will contribute to the achievement of the intended results and improve acceptability of IRS? Question 5 (Evaluation Question: How is the project perceived and valued...) Please identify the partners with which you work [e.g. N/ZMCP] on IRS. Tell me about the engagement of the TVCSP with any national partners (the N/ZMCP, the MoHSW), with regional and district partners; please tell me about successes and also about challenges and the ways in which these have been addressed and if the TVCSP has changed its approaches and working practices as a result. Question 6 (Evaluation Question: How is the project perceived and valued...) Please identify the partners with which you work on IRS at regional/district level. Please tell me about the engagement of the TVCSP with any regional/district partners; please tell me about successes and also about challenges and the ways in which these have been addressed and if the TVCSP has changed its approaches and working practices as a result. Question 7 (Evaluation Question: How is the project perceived and valued...) Please tell me about the engagement of the TVCSP with communities; please tell me about successes and also about challenges and the ways in which these have been addressed and if the TVCSP has changed its approaches and working practices as a result. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 97 Question 8 (Evaluation Question: Are there specific lessons...) What do you feel are the key lessons learned by the TVCSP; how have such lessons been used to benefit the TVCSP and its partners? Can you identify areas for improvement and if you were designing a follow-on program, would you do anything differently? Question 9 (Evaluation Question: In what ways has RTI project integrated gender considerations...) Please tell me about TVCSP cross-cutting activities (gender and cross-border collaboration): have you had any engagement with either activity and if so, how and when? Question 9 Please tell me about your use if any of TVCSP M&E. How do you use the PMP in your work? How do you support data collection for project indicators? (How) have you used M&E to shape your work on TVCSP? Question 10 (Evaluation Question: Is the program addressing sustainability...) Do you feel the TVCSP has contributed to the sustainability of malaria prevention and control interventions in terms of national, regional, district and community capacity? [Probe on e.g. N/ZMCP capacity development, planning focus, capacity to deliver IRS, gender-sensitive approaches to malaria prevention, BCC capacity, MEEDS, wider M&E]. Question 11 (Evaluation Question: Is the program addressing sustainability...) What future activities can TVCSP contribute to, in order to help sustain IRS achievements to date? Question 12 Please describe any working relationship TVCSP has with any other malaria program or research activity in Tanzania (or regionally within East Africa/SADEC)? E.g. GFATM, Gates Foundation, Roll Back Malaria, NMRI, Ifakara [to elicit information on joint working/sharing, planning - for sustainability, for community of practice]. Part II: Tailored questions for RTI employees [in case these are not covered through part I of the KII] Question 1 Which of the TVCSP components do you contribute to? Question 2 Have your responsibilities changed over time, since you joined the TVCSP? Question 3 Please tell me more about your specific responsibilities for the TVCSP components or components on which you work. Discuss in depth. E.g. for the M&E Officer How are project data collected and by whom? Please describe the process whereby progress against targets and indicators is measured. Please let us discuss the Performance Monitoring Plan (the PMP): progress against individual indicators - milestones, process indicators. Why was the PMP amended in December 2012? Which other sources of data are used by the project Probe on the degree to which sex-disaggregated data are collected (for which program indicators/more widely on malaria, MNCH, whether these data are program-specific and/or routine HMIS, survey, etc) and used to inform planning Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 98 How evidence-based is the TVCSP? For the HR/Capacity Development Officer: has TVCSP trained any counterparts (e.g. N/ZMCP, District officials) and if yes, in which subject/s? Has the project contributed to training materials, to national policy documents, to research, to development of BCC materials?] For the RTI employee responsible for TVCSP BCC activities: please describe the BCC materials the project uses and the way in which these were developed (e.g. any community participation, field testing, picture messaging for illiterate audiences). Has the project provided training in interpersonal communication for any IRS-linked employees or health staff? Has the project reviewed the efficacy of the BCC materials; if so, with whom? For the RTI employee responsible for TVCSP gender activities: please describe your role and responsibilities specific to TVCSP gender focus. Does your work require engagement with spray operators, with community members? Does it require gender-focused work with other RTI employees working on TVCSP, with the N/ZMCP and other stakeholders? What sort of inputs have you made to TVCSP planning, implementation and data management? What sort of gender training have you received? Question 4 Who are the TVCSP stakeholders and partners? Please tell me about your relationship with other TVCSP stakeholders specific to the component/s on which you work [this might be district administrators, epidemiologists, entomologists, M/ZMCP M&E staff, MEEDS-trained health workers, private spraying companies, etc, depending on the individual's role and responsibilities]. Question 5 Please describe key achievements, challenges and lessons learned specific to the TVCSP component/s on which you work. Question 6 If you were to design the follow-up TVCSP components on which you work, what might you change and why? Question 7 Please describe any TVCSP cross-cutting activities on which you have worked: gender and cross￾border collaboration. TOOL 2. KII FOR TVCSP IRS COORDINATOR Question 1 Please tell me about your role and responsibilities as IRS Coordinator on the TVCSP Question 2 Can you please describe to me the organizational structure of the IRS program in the district? Question 3 What are the guiding operational documents which TVCSP developed and is using? Question 4 What are the data being collected and reported, by whom and how are those data used in planning and implementation of IRS? Question 5 Please tell me what you feel the program has achieved to date in terms of: Proportion of protected population in sprayed districts Proportion of sprayed houses in targeted districts. Changes in vector density, composition, infectivity and human biting rates in sprayed districts Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 99 Changes in malaria cases treated at health facilities in the IRS districts. Question 6 What measures have been put in place to ensure quality, efficiency, high coverage, community acceptance of the IRS program? Question 7 What measures/ interventions is the program implementing to reduce the cost of structures sprayed and people protected in targeted districts? TOOL 3: KII FOR ENVIRONMENTAL MONITORING OFFICER: RTI AND NMCP/ZMCP Question 1 Can you please tell me your role and responsibilities as an Environmental Compliance Officer with TVCSP? Question 2 Please tell me what you feel are the key achievements of the program to date in addressing human and environmental safety in relation to US government and Tanzania Environmental laws. Question 3 Can you please tell me the mechanisms/processes which TVCSP has put in place to address human and environmental safety before, during and after spraying? Question 4 Do you think TVCSP has put in place adequate environmental compliance measures for proper insecticide transportation, storage, application and disposal? Can you please tell me the measures that are in place? Question 5 What are the specific environmental and human safety measures TVCSP is implementing to protect IRS implementers, IRS beneficiaries and environment from contamination? Question 6 Can you please tell me how TVCSP deals with liquid and solid wastes that are generated during IRS operations? Question 7 Can you tell me the TVCSP system for insecticide monitoring/stock management? Question 8 Do you know of any TVCSP reports of insecticide poisoning, insecticide pilferage, vehicle accident (transporting insecticide, spray operators or sprayers)? If so what containment measures were taken? Question 9 Can you please tell me who the partners are that support TVCSP to implement TVCSP environmental compliance plan? Question 10 Are there some success stories or lessons learnt on environmental compliance you may want to share with me? Question 11 Are there any additional measures you feel the TVCSP should put in place to improve human and environmental safety? Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 100 TOOL 4: KII FOR THE MALARIA ENTOMOLOGIST Question 1 Can you tell me the purpose and role of entomological monitoring in malaria vector control Question 2 What are the malaria vectors that are found in sprayed districts and their preferred resting and breeding sites? Question 3 Is there a system of monitoring entomological indicators (describe the system, indicators collected and frequency of collection of indicators) and are there any changes in the indicators in relation to TVCSP? Question 4 Is there a system of monitoring insecticide resistance in the IRS targeted districts? If yes, can you indicate the current status of insecticide resistance in local vectors and how the information is used in IRS planning and insecticide resistance management? Question 5 Do you have protocols to carry out the following procedures? determining the susceptibility of adult mosquitoes to insecticides determining the residual effect of insecticides on sprayed wall surfaces determining the residual efficacy of insecticides on treated mosquito nets determining the susceptibility of mosquito larvae to insecticides Question 6 Do you have the technical and infrastructural capacities for: identifying mosquito species, testing for insecticide resistance testing, identifying insecticide resistance mechanisms, determining insecticide decay on treated surfaces, keeping mosquito colonies? TOOL 5: KII FOR MAINLAND/ZANZIBAR MALARIA CONTROL PROGRAMME STAFF MEMBERS Part I: core questions for all respondents Question 1 Please tell me about your role and responsibilities with the M/ZMCP [probe how long the individual has worked at the MCP and malaria background]. E.g. the M&E Officer: Strength of malaria-specific data collection - at all levels, from health facility up to national programme? Please describe how data analysis is fed back to district level. Probe on the degree to which sex-disaggregated data are collected. How are malaria data used to inform planning and N/ZMCP activities? For the N/ZMCP employee responsible for BCC activities: please describe the BCC materials the MCP uses and the way in which these were developed (e.g. any community participation, field testing, picture messaging for illiterate audiences). Has the MCP provided training in interpersonal communication for any IRS-linked employees or health staff, or has any external partner provided such training? For the N/ZMCP employee responsible for gender activities: please describe your role and responsibilities specific to N/ZMCP gender focus. Does your work require focus on operational and programmatic gender Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 101 activities? Have you contributed on gender issues to development of N/ZMCP strategic plans, M&E and data management, annual budgets? What if any has been your engagement with the TVCSP on gender issues? What sort of gender training have you received? Question 2 Please tell me whether you have any working contact with the TVCSP and whether you have a specific role and responsibilities linked to the TVCSP. Question 3 Tell me your view of the relationship between the MCP and the TVCSP. [Probe in depth on issues such as: how the two work together, whether and how each supports the other, if there is effective joint activity planning and delivery, if there is effective responsiveness from each side, whether there are any challenges and what you feel are the main achievements of any joint planning, working and support.] Question 4 In your experience, which partners does TVCSP work with at regional, district and community levels. From your perspective, please tell me about the relationship TVCSP has with regional, district and community partners (including CSOs/CBOs); please tell me about successes and also about challenges and the ways in which these have been addressed and if the MCP and/or the TVCSP has changed its approaches and working practices as a result. Question 5 How effective do you feel the TVCSP has been in meeting its objectives and addressing its key result areas? Question 6 What do you feel are the key lessons learned by the MCP from the TVCSP and vice versa; can you identify areas for TVCSP and MCP improvement Question 7 Are there any long-term benefits that you can identify for the MCP that have been supported by the TVCSP? Probe on e.g. capacity development, M&E, epidemiological and/or entomological capacity strengthening, program planning, community engagement, etc. Question 8 Do you feel the TVCSP has contributed in any way to the sustainability of malaria prevention and control interventions in terms of MCP capacity? Probe on e.g. MCP capacity development, capacity to deliver IRS, MEEDS, Wider M&E, greater gender focus, more cross-border collaboration Question 9 If you were designing a follow-on project/program to the TVCSP, would you do anything differently? Part II: Tailored questions for MCP members of staff [in case these are not covered through part I of the KII] Question 1 Are you responsible for contributing to one or more of the TVCSP components through your work at the MCP? Question 2 Have your responsibilities changed over time, since the TVCSP began its work in 2010? Question 3 Please tell me about your relationship, if any, with TVCSP/malaria stakeholders specific to the component/s on which you work [ this might be district administrators, epidemiologists, entomologists, Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 102 M/ZMCP M&E staff, MEEDS-trained health workers, private spraying companies, etc, depending on the individual's role and responsibilities]. Question 4 Please describe what in your view are the key achievements, challenges and lessons learned specific to the TVCSP. [Check if the respondent has any data/ report relevant to the evaluation and request copy]. Part III: Questions for NMCP/ZMCP on capacity building Objective: Develop capacities of ZMCP, NMCP, and districts to plan and implement IRS, including logistics planning, environmental compliance, and epidemic response through focal spraying. Determine progress made in the capacity building and institutional strengthening of the Ministry of Health and Social Welfare (MOHSW)/NMCP and Zanzibar MOH/ZMCP Question 1 Progress made in Capacity Building and Institutional Strengthening a. Before the start of the TVCP project, how would you rate the NMCP/ZMCP capacity to implement its IRS strategy? b. What were the specific challenges NMCP/ZMCP faced in relation to its capacity to implement the IRS strategy? c. How has the project improved the NMCP/ZMCP capacity to plan and implement IRS? d. What specific steps/activities has the TVCP Project taken to improve NMCP/ZMCP logistic planning? Have these led to an improvement? e. What specific steps has the TVCP taken to improve NMCP capacity in environmental compliance? Have these led to improvements? f. What specific steps has the TVCP taken to improve NMCP/ZMCP capacity in epidemic response through focal spraying? Have these led to improvements? g. Has TVCSP trained any N/ZMCP staff members, and if yes, in which subject/s? Has the project contributed to training materials, to national policy documents, to research, to development of BCC materials?] h. Overall, has the TVCP improved the capacity of NMCP/ZMCP to implement IRS? Question 2 Perception of TVCSP a. What is your opinion of this project? (is it a useful, effective, good for the community?) b. In your opinion how is the TVCP project perceived by local government authorities, the community? TOOL 6: KII FOR DIRECTOR/PERMANENT SECRETARY (and any other national/regional￾level senior MoSHW staff member) 1. View on IRS in the Country a. Rating of malaria as a priority for the interviewee (High; Medium; Low)? b. Description of the interviewee’s view on the organization and management of IRS in Tanzania 2. Political and Financial Commitment a. Indicate in what ways the office of the interviewee can contribute towards advocating for IRS b. Indicate opinion of interviewee on the current placement of IRS in the MoH c. Indicate whether IRS has adequate human resource at national, provincial and district levels d. Indicate whether IRS will be allocated resources from the government budget 3. Challenges government facing in IRS a. List key challenges the government is facing in sustaining IRS achievements in Tanzania. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 103 4. Engagement with TVCSP a. Engagement with project b. Opinion of its achievements c. Opinion of its challenges d. Has the work of the TVCSP supported the work of the MoHSW and the N/Z MCP? e. If yes, please describe f. Are project data used in MoHSW and N/Z MCP planning and operations? g. If no, please consider why h. How might TVCSP better support the MoHSW and the N/Z MCP to achieve its malaria targets? i. The best way forward for the project in its final 2 years, specific to working with the TZ government 5. Way Forward List ways in which IRS can further be strengthened and sustained in the country. TOOL 7: DISTRICT EXECUTIVE OFFICERS AND DISTRICT MEDICAL OFFICERS Question 1 Tell me your view of the relationship between the district of and the TVCSP. [Probe in depth on issues such as: how does the project work with the district, whether and how each supports the other, if there is effective joint activity planning and delivery, if there is effective responsiveness from each side, whether there are any challenges and what you feel are the main achievements of any joint planning, working and support.] Question 2 Has the TVCSP been of benefit to the district, community? Question 3 From your perspective, please tell me about the engagement TVCSP has with regional, district and community partners (including CSOs/CBOs); please tell me about successes and also about challenges and the ways in which these have been addressed and if the district and/or the TVCSP has changed its approaches and working practices as a result. Question 4 How effective do you feel the TVCSP has been in meeting its objectives and addressing its key result areas [we shall have to have these to hand, so as to remind the respondent]? Question 5 What do you feel are the key lessons learned by the district from the TVCSP and vice versa; can you identify areas for TVCSP and MCP to improve? Question 6 Are there any long-term benefits that you can identify for the district that have been supported by the TVCSP? Probe on e.g. capacity development, M&E, epidemiological and/or entomological capacity strengthening, program planning, community engagement, etc. Question 7 Do you feel the TVCSP has contributed in any way to the sustainability of malaria prevention and control interventions in the district? Probe on e.g. MCP capacity development, capacity to deliver IRS, the national health strategy, gender-sensitive approaches to malaria prevention and treatment, BCC development? Question 8 Is the district contributing to the TVCSP work on IRS [Probe on e.g. allocation of financial resources, support from district structures, seconding of staff, etc]. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 104 Question 9 Do you feel this is a project that the district could or should take over should RTI withdraw from your district? What arrangements are you making if any to continue IRS interventions in the long-term? TOOL 8: KII WITH HEALTH FACILITY STAFF MEMBER Obtain name, position and name and location of health facility Question 1 What is your primary assignment at this health facility and how long have you worked here? Question 2 Does your health facility provide any form of information or items for community members on malaria prevention - probe on what sort of information, if there are any materials, how illiterate people are given information. Does the facility provide ITNs/LLINs? Does your facility provide Intermittent Preventive Treatment in pregnancy (IPTp)? Question 3 What do you know about indoor residual spraying (IRS)? Has any been conducted in the past 1-3 years in the communities this health facility serves? Probe: has the health worker had any engagement with the project, e.g. any orientation in its goal, its activities, its BCC, its work with communities? Has the health worker had any training on IRS? Question 4 Does your facility/community accept routine IRS? What is the acceptance rate? Can you mention any reasons for not accepting the spraying? Question 5 In the time you have worked at this health facility, have you observed any changes in the number of people presenting with malaria and/or for presumptive treatment? If changes observed, probe: numbers up or down overall, any particular group or groups of people presenting more or less often (pregnant women, young women, mothers, infants, children, men), any observed changes in seasonal trends. Question 6 If any changes have been observed, what do you attribute them to (both negative and positive trends). Probe: greater public awareness, more BCC/IEC provided, IRS, ITNs, etc? Question 7 Please may we see your HMIS form/M&E form/record book/tally sheet where you record information on malaria. Probe: how often does the health worker submit the data; feedback (if any and from whom); does the health worker use the data. TOOL 8a CHECKLIST: Additional questions specific to Malaria Epidemiology for Health Facility staff 1. Name of health facility 2. Contact person name and title 3. Malaria morbidity: Is it an important problem in your community? If yes, which member/s of the population are affected, when and where? 4. How many patients sought malaria treatment during the most recent peak month for the malaria transmission? No Children < 5 years Pregnant women , Other age groups Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 105 5. What type of malaria diagnosis tests are available at the health facility? Probe; Do you encounter mRDT stock out? 6. What type of antimalarial medication is available at the health center? Do you encounter ACT stock outs? 7. What type of malaria medication is available in the village (market, pharmacies, CHW, other, without prescription etc)? 8. Has your facility noted change in no of cases of malaria morbidity or mortality after IRS rounds? Is it improving, deteriorating or unchanged? Are more or fewer men, women and children presenting? If deteriorating then what are the main obstacles to achieving a good level of control? 9. Do malaria epidemics occur in your District? When and where do malaria epidemics occur in your district, and why? Probe about (Unusual rainfall after prolonged drought, population movements, unusual rainfall, unusual temperature increase, numerous water pools after flooding, delayed or no control measures like IRS,) 10. What could be done to confirm the diagnosis of a reported epidemic of malaria? If the majority of cases of epidemic fever have patent parasitaemia, does that establish that the epidemic fever is malaria? If the existence of a malaria epidemic has been clearly established, what are immediate measures taken, in view of control? 11. . Do you have an established system of reporting a malaria epidemic; please describe the steps? Did your health facility receive any technical support from the NMCP, the TVCSP/RTI or other sources in reporting malaria epidemics? What type of support? TOOL 9. FOCUS GROUP DISCUSSION WITH COMMUNITY MEMBERS WHO HAVE RECEIVED IRS We shall have separate FGDs with male and female community members Introduction plus consent Question 1 What do you understand when we speak about IRS [we shall have to find our what the usual Swahili term is for this]. Allow people to discuss Question 2 Has your household ever received IRS? If yes, when, how many times, with what sort of frequency? Question 3 What is the reason for IRS? Question 4 Did you personally get an explanation of why IRS is considered important? If necessary, probe on: whether the reasons were explained, by whom, when and where (e.g. at the time of spraying, at the health center, if a woman from your husband) Question 5 How was IRS described to you and why is it considered important? Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 106 Question 6 Who made the decision to receive IRS - in your household, in your community (did you have a choice?) Question 7 Have you received any other information on malaria, either at the time of the IRS or at another time? If necessary, probe on e.g. health information on malaria transmission, prevention and treatment information. Question 8 Has the behavior of your household changed since you received IRS? If necessary, probe on e.g. sleeping under LLINs/ITNs, greater awareness of malaria transmission and prevention, household or community action to clear stagnant water, etc). Question 9 Have you seen any changes in the health of your household since you received IRS? If necessary probe, e.g. on benefits to individual members and the whole household, changes in patterns of illness - and be prepared to listen to any mention of perceived adverse effects. Question 10 Have you seen any changes in health-seeking behavior since your household received IRS - in other words, are you individually and/or other members of your household seeking medical help as often, less often or more often, and for which ailments? Question 10 Have you received any other health information and/or products in addition to the IRS? If necessary, probe on e.g., (free) provision of LLINs/ITNs). From whom: nurse, CHW, etc? Probe with women (also ask men): any information given during ANC? Any provision of intermittent Preventive Treatment in Pregnancy (IPTp)? Child health: any malaria prevention and treatment information given at CH/MCH clinics? Question 11 What is your opinion of the health care you, your household and your community receive for malaria? If necessary, probe on e.g. BCC, health worker responsiveness, ready availability of ACT, IPTp, referral, etc. TOOL 10. KEY INFORMANT INTERVIEW OR FGD FOR IRS SPRAY OPERATORS Introduction plus consent Question 1 Can you please tell me the number of times TVCSP has recruited you as a spray operator? Question 2 Can you tell the process TVCSP used to recruit you as a spray operator? Question 3 For how many days did TVCSP train you to be able to spray and what are the knowledge and skills you gained from the training? Question 4 Can you name personal protective equipment which you received and the use and care of each item? Question 5 What other personal measures were you told to do and not to do to prevent insecticide contamination? Question 6 Can you tell me the information you gave to community members before and after spraying? Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 107 Question 7 Were the community members happy to have their houses sprayed? Why were they happy or not happy? Question 8 In your opinion, what did the TVCSP achieve in this area? Question 9 What are the beneficiaries saying about the TVCSP after spraying? Question 10 Do you think spraying houses is useful and do you think the TVCSP should continue spraying this area? Question 11 What information did you record on the spray operator’s form? Question 12 How did you account for the insecticide sachets you received? Question 13 Can you describe the roles and responsibilities the community members play in spraying activities; could this participation be improved, to support the spraying activities? Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 108 Annex 4: Evaluation Schedule and Sample 4.1 Final Evaluation Schedule Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sept 2013 9 10 Review background documents and preparation work 11 Review background documents and preparation work 12 Review background documents and preparation work 13 Review background documents and preparation work 14 Travel to TZ (Janet G) 15 Travel to TZ (Janet G)/ OFF 16 Review background documents and preparation work 17 Review background documents and preparation work; travel to TZ (John G) 18 Team Planning Meeting (TPM) 19 TPM and Interviews/ fieldwork (day 1) Dar es Salaam RTI 20 Interviews/ fieldwork (day 2). Dar es Salaam USG RTI 21 TPM/ Interviews / fieldwork (day 3) Dar Team work 22 OFF 23 Interviews/ fieldwork (day 4). Meeting 2 USG Dar NMCP USG Fly to Zanzibar 24 Interviews/ fieldwork (day 5) Zanzibar ZMCP MoHSW RTI TVCSP West District District staff RTI Health workers Community members & spray operators 25 Interviews/ fieldwork (day 6) Zanzibar TVCSP North B District District staff Health workers Community members & spray operators Fly to Dar 26 Interviews/ fieldwork (day 7) Dar NMCP and RTI 27 Interviews/ fieldwork (day 8) Dar NMCP and RTI; other meetings 28 Interviews / fieldwork (day 9) Interview analysis 29 Fly to Kagera 30 Interviews/ fieldwork (day 10) Kagera RTI Regional staff Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 109 Sunday Monday Tuesday Wednesday Thursday Friday Saturday Oct 2013 1 Interviews/ fieldwork (day 11) RMO 1st Kagera District (Bukoba Rural) District staff Health workers Community members & spray operators 2 Interviews/ fieldwork (day 12) 2nd Kagera District (Muleba) District staff Health workers Community members & spray operators 3 Interviews/ fieldwork (day 13) Fly to Mwanza RAS, RMO & other regional administrat ors 1st TVCSP District (Magu) District staff Health workers Community members & spray operators 4 Interviews/ fieldwork (day 14) Mwanza NIMR, Mwanza 2nd TVCSP District (Misungwi) District staff Health workers Community members & spray operators 5 Interviews / fieldwork (day 15) Mwanza RTI meetings Travel to Dar 6 OFF 7 Dar Draft evaluation report: analysis. 8 Dar Draft evaluation report: analysis 9 Dar Preparation of outbrief 10 Dar Preparation of outbrief 11 Dar Outbrief at USAID Draft evaluation report: analysis 12 Dar Draft evaluation report: analysis 13 OFF 14 Dar Draft evaluation report preparation 15 Dar Draft evaluation report preparation 16 Dar Draft evaluation report preparation 17 Dar/travel Draft evaluation report preparation. Travel (John G and Janet G) 18 Draft report preparation; submission by SOB 10/21 19 20 21 PMI provides comments 22 PMI provides comments 23 PMI provides comments 24 PMI provides comments 25 PMI provides comments 26 Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 110 Sunday Monday Tuesday Wednesday Thursday Friday Saturday 27 28 Final report (5 days) 29 Final report (5 days) 30 Final report (5 days) 31 Final report (5 days) Nov 2013 1 Final report (5 days) Date TBC: debrief to PMI DC Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 111 4.2 Sample See also Annex 2 for the full list of people met. TVCSP Evaluation: Sample Dar es Salaam Zanzibar (West & North B) Lake Zone (Kagera - Bukoba & Muleba) & Mwanza - Magu & Misungwi) N/ZMCP officials ✓ ✓ Zanzibar MoH (Permanent Secretary & Director, Preventive Services) ✓ JHU- COMMIT ✓ RTI staff members ✓ ✓ ✓ (Mwanza Zonal & Kagera Regional) USG staff members ✓ Regional authorities: RAS/RMO/RMIFP/RHO ✓ (Kagera & Mwanza) District authorities: Acting DED (Bukoba, Misungwi)/(Acting) DMO/ DMIFP/DVCO/ DHO/DIEC/DM&E ✓ ✓ DMSO ✓ Health Facility staff ✓x 2 HF ✓x 4 HF Spray Operators (58 in total: 28M/30F) ✓x 2 (mixed groups) ✓x 4 (mixed groups) IRS Site Managers ✓ (1) ✓ (4) Community members (68 in total: 39M/29F) ✓x 4 (2 M & 2 F) ✓ x 8 (4 M & 4 F) SIMs ✓ (2) NIMR scientists & insectary staff ✓ (5) Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 112 Annex 5: The Current Malaria Epidemic Detection and Response Situation in Mainland Tanzania 1. Indoor residual Spraying in the Lake Zone to date and impacts on malaria Since 2007 the PMI-supported Tanzania Vector Control Scale-up Project (TVCSP) has implemented Indoor Residual Spraying (IRS) in the lake zone of mainland Tanzania, initially in two districts in Kagera region. The project progressively scaled up to cover the remaining five districts of Kagera Region and by early 2011, IRS expanded to cover all the districts in two remaining regions of Lake Zone; Mwanza and Mara. In total, 18 districts in the Lake Zone (7 in Kagera Region, 6 in Mwanza Region, 5 in Mara Region) have been sprayed. IRS is implemented in the Lake Zone concurrently with other malaria interventions: improved malaria diagnostics with Rapid Diagnostic Tests (RDTs) and improved malaria case management with ACTs; universal distribution with long lasting insecticide-treated nets (LLINs); and prevention of malaria in pregnancy. By September 2012, TVSCP had supported districts in the Lake Zone to spray 1,224, 095 structures (with 93% coverage) in all 18 districts of Kagera, Mwanza, and Mara Regions of Lake Zone, protecting about 6.5 million people. Due to the high universal coverage with LLINs and the reduction in the number of malaria cases over the past 3-4 years, TVCSP subsequently scaled down IRS in Lake Zone in financial year 2012/2013. Between October 2012 and June 2012, following this shift in spraying strategy from blanket to targeted spraying, 135,613 (91% of the eligible structures) were sprayed, protecting 659,697 residents. Together with the increased coverage with ITNs and other malaria interventions, IRS has contributed significantly to reducing the malaria prevalence by more than 50% in the Lake Zone. According to the 2011/12 Tanzania HIV/AIDS and Malaria Survey (THMIS), prevalence of malaria in children aged 6 to 59 months is 8% in Kagera (down from 41.1% in 2007/8); 19% in Mwanza (down from 31.4% in 2007/8); and 25% in Mara (down from 30.3% in 2007/8). 2. The Muleba outbreak During year 2 of TVCSP implementation, a dramatic increase in malaria cases and deaths was reported in Muleba District in Kagera Region. Prior to that, after the introduction of IRS and other malaria control interventions, the malaria positivity rate had stabilized at around 10%. TVCSP noted a dramatic increase in the malaria positivity rate in June 2011 and November 2011, over 40% and 36.4%, respectively. Following the NMCP investigation conducted between December 20–23, 2011, at Rubya and Ndolage hospitals in Muleba District, findings showed the positivity rate for November 2011 was 36.4% and 45.8% through December 22, 2011. In September-October 2013 The TVCSP mid-term performance evaluation team learned from district personnel in Kagera and Mwanza that after the switch to targeted spraying, villages that were not being sprayed were experiencing increased numbers of malaria cases. From this evidence it can be assumed that Kagera region and neighboring Lake Zone regions are vulnerable to dramatic increases in malaria cases and possibly epidemics, possibly due to reduced levels of immunity in the population due to relatively less malaria exposure. A surveillance system capable of detecting sudden increases in transmission that will trigger a response from district level malaria control personnel to avert possible high case-fatality rates in the community is therefore a priority. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 113 3. What is the status of the malaria surveillance system on mainland Tanzania? Mainland Tanzania adopted the Integrated Disease Surveillance and Response (IDSR) strategy in 1998 to strengthen surveillance of key infectious diseases. Electronic IDSR (eIDSR) will be implemented through the newly rolled out District Health Information System (DHIS2, an electronic version of the HMIS). The implementation of eIDSR is a partnership between the Ministry of Health and Social Welfare (MoHSW) Health partnership, the MoHSW Epidemiology Unit, the NMCP, the CDC Foundation, the University of Dar es Salaam (UDSM) and PMI/RTI. Each partner has clearly defined roles. The CDC Foundation is leading on the development of the electronic platform for reporting data. UDSM was contracted to develop the electronic system for reporting by the CDC Foundation. The role of RTI/TVCSP is to support implementation of eIDSR in the Lake Zone and Dar es Salaam. The key challenge to rolling out eIDSR has been the development of the electronic reporting platform, a process beyond the control of RTI/PMI/NMCP because it is led and funded independently by a different partner. TVCSP has provided technical support to NMCP to develop National Malaria Surveillance Guidelines; these are key for ensuring that surveillance standards are maintained across the board and to prevent duplication of systems. In collaboration with the MoHSW Epidemiology Unit and NMCP, TVCSP jointly developed IDSR reporting forms and undertook training of trainers on IDSR in October of 2012. However, the electronic platform was not available for implementation when expected. As a result, it has not been possible to scale-up training and supervision to the health facilities. In the absence of electronic reporting, TVCSP worked with the district malaria focal persons and the MoHSW Epidemiology unit to distribute IDSR reporting booklets to health facilities; Currently (late October 2013), progress in rolling out the eIDSR platform is slow and unsteady. TVCSP continues to engage constructively with the partnership and to report to PMI on important milestones. There are plans to pilot the system in November 2013 and TVCSP is providing support and funding for this activity. Despite the delays of the electronic component, RTI in collaboration with NMCP and MoHSW has accomplished the following: 1. Malaria weekly reporting was included in the IDSR tools and protocols 2. Training of trainers was undertaken to train district level staff on the IDSR reporting 3. IDSR reporting booklets were produced and distributed in all health facilities in the Lake Zone Progress made in training staff and providing tools notwithstanding, the lack of electronic reporting essentially means that MEEDS is not operational. It is not possible to implement MEEDS without timely reporting of weekly malaria data. 4. The way forward: the introduction of MEEDS in Lake Zone A functional epidemic detection and response system in the Lake Zone is critical, in view of the reported increases in cases in Kagera and anecdotal evidence of increases in Mwanza. As TVCSP moves from blanket IRS to targeted and then to focal IRS in the context of reduced malaria burden due to previous blanket spray rounds, it is essential that measures are put in place to detect malaria epidemics. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 114 While awaiting the eIDSR platform to become operational, TVCSP proposes to pilot a MEEDS which will collect the same malaria indicators agreed with NMCP and as specified in eIDSR, applying lessons learned from the successful Zanzibar roll-out of MEEDS. These data will be transferred to the eIDSR server so that it is available immediately to NMCP and MoHSW. Once the eIDSR is up and running, the MEEDS will be switched off (in the Lake Zone). The plan will then be to continue with the eIDRS platform, which will collect data on malaria and an additional 16 diseases in the IDRS framework. TVCSP is working on obtaining the requisite approvals to proceed with MEEDS reporting using mobile phones. Once this is done health facility staff will be able to report malaria data from the IDSR booklets to a dedicated server. At a malaria Technical Working Group meeting on 24 October, 2013 it was reported that the piloting of MEEDS will shortly begin in Temeke district in Dar es Salaam, to be followed by piloting in three additional districts (one district will be selected from each of Kagera, Mwanza and Mara regions in Lake Zone). 5. Potential challenges If the planned two-phase integrated approach does not work, then TVCSP, NMCP and other partners will need to make rapid decisions. Without proper surveillance systems it will be a challenge to make informed decisions on the optimal use of resources. Moreover, from the Muleba evidence, it will be impossible to prevent malaria epidemics - with potentially disastrous consequences. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 115 Annex 6: References Please note that all documents listed here have been reviewed for the purposes of the TVCSP mid-term performance evaluation. Alidina Zainab, Shabbir Lalji, Joshua Mutagahywa, Mahdi Ramsan, Jessica Kafuko, Uche Ekenna, Stephen Magesa, Jeremiah Ngondi and Rajeev Colaco. 2013. “Taking local ownership: government and household contributions to indoor residual spraying for malaria control in Zanzibar and Mainland Tanzania.” Abstract submitted to the 6th MIM Conference, October 2013. Bauch Julie, Jessica Gu, Mwinyi Msellem, Andreas Mårtenson, Abdullah Ali, Roly Gosling and Kimberley Baltzell. 2013. “Perception of malaria risk in a setting of reduced malaria transmission: a qualitative study in Zanzibar”. Malaria Journal 12: 75. Deus S Ishengoma, Bruno P Mmbando, et al. 2013. “Declining burden of malaria over two decades in a rural community of Muheza district, north-eastern Tanzania.” Malaria Journal 2013, 12:338. Fabian M. Mashauri, Safari M. Kinung’hi et al. 2013. “Impact of Indoor Residual Spraying of Lambda￾Cyhalothrin on Malaria Prevalence and Anemia in an Epidemic-Prone District of Muleba, North-Western Tanzania.” Am. J. Trop. Med. Hyg. 88(5), pp. 841–849. Kaufman, Michelle, Datius Rweyemamu, Jacob Macha, Robert Ainslie, Hannah Koenker and Fabricio Molteni. 2011. “My children and I will no longer suffer from malaria: acceptance and rejection of indoor residual spraying to prevent malaria in Tanzania.” Baltimore and Dar es Salaam: Johns Hopkins Bloomberg School of Public Health, Center for Communication Programs, University of Dar es Salaam and RTI International. 2011. Knols, Bart G. J. 2013. “1950s' strategy to control malaria on Zanzibar fails once more.” Malaria World blog post, March 29, 2013. Kweka Eliningaya, Beda Mwang'onde, Epiphania Kimaro, Shandala Msangi, Filemoni Tenu, and Aneth Mahande. 2009. “Insecticides' Susceptibility Status of the Bedbugs (Cimex lectularius) in a Rural Area of Magugu, Northern Tanzania” J Glob Infect Dis. Jul-Dec; 1(2): 102–106. Ministry of Health, Zanzibar and Zanzibar Malaria Control Programme. 2011. “Strengthening Monitoring and Evaluation activities for malaria control in Zanzibar.” Quarterly Report October - December 2011. Molteni, F., Wilson, K., Mugalura, F. Lalji, S., Mandike, R., and Mwita, A. “Shrinking the Malaria Map in Muleba District, North Western Tanzania: monitoring the impact of malaria prevention interventions scale￾up through hospital admissions.” National Institute for Medical Research (NIMR). 2009. “Entomological monitoring of IRS in Kagera region, north-western Tanzania: Baseline survey for the 5 new districts and follow up to Karagwe and Muleba districts.” Final report March 2009. NIMR Mwanza. NIMR. 2012. “Longitudinal entomological monitoring of IRS in the Lake Zone regions, Tanzania.” Annual Implementation Report. NIMR. 2013. “Detection and monitoring of insecticide resistance in malaria vectors in Mainland Tanzania.” NIMR Amani Medical Research Centre. March 2013. President's Malaria Initiative (PMI). 2013. “Seventh Annual Report to Congress.” April. PMI/Tanzania. Tanzania Country Profile. Dar es Salaam: PMI. Not dated. PMI/Tanzania. 2008. “Malaria Operational Plan: Tanzania FY 2009.” Dar es Salaam: PMI, USAID, CDC.. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 116 PMI/Tanzania. 2009. “Malaria Operational Plan: Tanzania FY 2010.” Dar es Salaam: PMI. USAID, CDC. PMI/Tanzania. 2010. “Malaria Operational Plan: Tanzania FY 2011.” Dar es Salaam: PMI. USAID, CDC. PMI/Tanzania. 2011. “Malaria Operational Plan: Tanzania FY 2012.” Dar es Salaam: PMI. USAID, CDC. PMI/Tanzania. 2012. “Malaria Operational Plan: Tanzania FY 2013.” Dar es Salaam: PMI. USAID, CDC. Protopopoff N., Matowo J., Malima R., Kavishe R., Kaaya R., Wright A., West P.A., Kleinschmidt I., Kisinza W., Mosha F.W., and Rowland M. “High level of resistance in the mosquito Anopheles gambiae to pyrethroid insecticides and reduced susceptibility to bendiocarb in north-western Tanzania.” Malaria Journal 12: 149. Research Triangle Institute (RTI) International. 2009. “Program description for additional activity to RTI CA # 621-a-00-10-00015-00. Activity: epidemic surveillance.” Dar es Salaam and Research Triangle Park, NC. (Not dated: 2009). RTI International. 2009. “Scale-up of Indoor Residual Spraying in Tanzania: Volume 1 - Technical Application.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2009. “Program Description: Scale-up of IRS in Tanzania. Section C: Description/Specifications/Statement of Work.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2010. “Tanzania Vector Control Scale-up Project (TVCSP). Performance Monitoring Plan: March 9, 2010 - March 8, 2015.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2010. “Tanzania Vector Control Scale-up Project (TVCSP). Annual Implementation Plan: Year 1 (March 9, 2010 - September 30, 2011).” Dar es Salaam and Research Triangle Park, NC. RTI International. 2010 “TVCSP. Quarterly Performance Report March 9 - June 30, 2010.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2010. “Tanzania Vector Control Scale-up Project (TVCSP). Assessment Report August 2010: IRS Warehousing Compliance for Indoor Residual Spraying.” Standardized Best Management Practices. Dar es Salaam and Research Triangle Park, NC. RTI International. 2010. “TVCSP. Quarterly Performance Report July 1 - September 30, 2010.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2010. “Tanzania Vector Control Scale-up Project (TVCSP). Supplemental Environmental Assessment: Tanzania Vector Control Scale-up Project Amendment to include Mwanza and Mara Regions in Tanzania, using Pyrethroids, Organophosphates, and Carbamates.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2011. “TVCSP. Quarterly Performance Report October 1 - December 31, 2010.” Dar es Salaam and Research Triangle Park, NC. RTI International. TVCSP. “Annual Performance Report March 9, 2010 - March 31, 2011.” Dar es Salaam and Research Triangle Park, NC. RTI International. TVCSP. 2011. “Annual Performance Report March 9, 2010 - March 31, 2011”; Annexes A-D. Dar es Salaam and Research Triangle Park, NC. RTI International (Environmental Compliance Unit). 2011. “Tanzania Vector Control Scale-up Project (TVCSP).” Solid Waste Disposal Report, April 26, 2011. Dar es Salaam and Research Triangle Park, NC. RTI International. 2011. “TVCSP. Quarterly Performance Report April 1-June 30, 2011.” Dar es Salaam and Research Triangle Park, NC. 2011. RTI International. 2011. “TVCSP. Quarterly Performance Report April 1 - July 30, 2011”; Annexes A-D. Dar es Salaam and Research Triangle Park, NC. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 117 RTI International. 2011. “Tanzania Vector Control Scale-up Project (TVCSP). 2011 Mid-Spray Environmental Compliance Inspection Report (Kagera, Mwanza, Mara, and Zanzibar).” Dr Gaspard Bikwemu. Dar es Salaam and Research Triangle Park, NC. 2011. RTI International. 2011. “Tanzania Vector Control Scale-up Project (TVCSP). Annual Implementation Plan: Year 2 (October 1, 2011 - September 30, 2012).” Dar es Salaam and Research Triangle Park, NC. RTI International. 2011. “Tanzania Vector Control Scale-up Project (TVCSP). Tanzania Vector Control Scale-up Project: Costs of Indoor Residual Spraying: Comparison of 2008 and 2009 Expenditures in Mainland Tanzania and Zanzibar.” J Sine and S Lalji. Dar es Salaam and Research Triangle Park, NC. RTI International. 2011. “TVCSP. Quarterly Performance Report October 1 - December 31, 2011.” Dar es Salaam and Research Triangle Park, NC. 2011. RTI International. 2011. “TVCSP. Quarterly Performance Report October 1 - December 31, 2011””; Annexes A-M. Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “TVCSP. Quarterly Performance Report January 1-March 31, 2012.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “TVCSP. Quarterly Performance Report January 1-March 31, 2012”; Annexes A-D. Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “TVCSP. Quarterly Performance Report April 1-June 30, 2012.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “TVCSP. Quarterly Performance Report April 1-June 30, 2012”; Annexes A-D. Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “TVCSP. Annual Performance Report October 1 2011 - September 30 2012.” Dar es Salaam and Research Triangle Park, NC.. RTI International. 2012. “TVCSP. Quarterly Performance Report October 1 - December 31, 2012.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “Tanzania Vector Control Scale-up Project: Costs of Indoor Residual Spraying, 2010.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2012. “Tanzania Vector Control Scale-up Project (TVCSP). Annual Implementation Plan: Year 3 (October 1, 2012 - September 30, 2013).” Dar es Salaam and Research Triangle Park, NC. RTI International. 2013. “TVCSP. Quarterly Performance Report January 1-March 31, 2013.” Dar es Salaam and Research Triangle Park, NC. RTI International. 2013. “TVCSP. Quarterly Performance Report January 1-March 31, 2013;” Annexes A-F. Dar es Salaam and Research Triangle Park, NC. RTI International. 2013. “Tanzania Vector Control Scale-up Project (TVCSP). Quarterly Performance Report April 1-June 30, 2013.” Dar es Salaam and Research Triangle Park, NC. Tanzania Commission for AIDS, Zanzibar AIDS Commission, National Bureau of Statistics, Office of the Chief Government Statistician and ICF International 2013. 2012. “Tanzania HIV/AIDS and Malaria Indicator Survey.” Dar es Salaam, Tanzania: TACAIDS, ZAC, NBC, OCGS and ICF International. Thomas, Matt. Cinderella Science [vector control]. 2012. Malaria World blog post, December 1. United Republic of Tanzania, Ministry of Health and Social Welfare and National Malaria Control Programme. 2008. “Malaria Medium-Term Strategic Plan 2008 – 2013.” Dar es Salaam. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 118 United Republic of Tanzania, Ministry of Health and Social Welfare and National Malaria Control Programme. 2008. “Monitoring and Evaluation Plan 2008 – 2013.” Dar es Salaam. United States Agency for International Development (USAID)/Tanzania. 2010. Cooperative Agreement. Dar es Salaam. March 2010. USAID Office of the Inspector General. 2011. “Audit of USAID/Tanzania’s ongoing activities under the President’s Malaria Initiative: audit report no. 4-621-11-007-P.” April 29, 2011. WHO & Roll Back Malaria. 2007. “Gender, Health and Malaria.” Geneva: WHO Dept. of Gender and Health & RBM. ZMCP Vector Control Technical Committee. 2011. “Entomological Monitoring and Insecticide Resistance Management in Zanzibar.” Report of the VCTC Meeting, March 29, 2011. In addition, a large number of BCC materials were made available for review by JHU COMMIT and the NMCP; these included radio spots, mid media and IPC materials. Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 119 For more information, please visit http://www.africanstrategies4health.org/resources.html Tanzania Vector Control Scale-Up Project | Midterm Performance Evaluation March 2014 1 The African Strategies for Health Project 4301 N Fairfax Drive, Arlington, VA 22203 Telephone: +1-703-524-6575 AS4H-info@as4h.org www.africanstrategies4health.org