June 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Socio-Economic Data Centre Limited for QED. Mid-term Evaluation of the Uganda Indoor Residual Spraying Project Phase II MID TERM EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PROJECT PHASE II June 2016 THE EVALUATION TEAM This evaluation was undertaken by Socio-Economic Data Centre Ltd (SEDC), with the following team: Dr. Denis Muhangi Social Scientist/Team Leader Dr. John M Govere Medical Entomologist/IRS Specialist Festus Kibuuka Capacity Building Specialist Joseph KB Matovu Behavioral Scientists/BCC Specialist Jimrex Byamugisha Statistician Cover photo: Spray Operators under the IRS Phase II project washing their pumps after a day’s work. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms/ abbreviations ...............................................................................................................................iv Executive Summary .........................................................................................................................................vi 1 Project Background .................................................................................................................................. 1 2 Evaluation Purpose & Evaluation QuestionS ...................................................................................... 2 2.1 Evaluation Purpose........................................................................................................................... 2 2.1 Evaluation Questions ....................................................................................................................... 2 3 Evaluation methods .................................................................................................................................. 3 3.1 Evaluation approach ......................................................................................................................... 3 3.2 Sampling design for the evaluation ............................................................................................... 3 3.2.1 Sampling of districts ................................................................................................................ 3 3.2.2 Sampling of sub-counties, parishes and villages ................................................................. 4 3.2.3 Sampling of household respondents .................................................................................... 4 3.2.4 Data collection methods and tools ...................................................................................... 5 3.2.5 Data management and analysis ............................................................................................. 5 3.2.6 Limitations ................................................................................................................................. 5 4 Findings, conclusions & Recommendations ........................................................................................ 7 4.1 To What Extent is the IRS program meeting the high quality standards set out in the program design? ............................................................................................................................................ 7 4.1.1 Introduction .............................................................................................................................. 7 4.1.2 Quality Standards ..................................................................................................................... 7 4.1.3 Inclusiveness standards ........................................................................................................... 9 4.1.4 Safety Standards .................................................................................................................... 11 4.1.5 Effectiveness Standards ........................................................................................................ 16 4.2 To what extend did the Behavioral Change Communication (BCC) campaign influence on desired behaviors in net use, prevention of malaria in pregnancy, seeking treatment early and other kinds of malaria prevention and management practices? .............................................. 19 4.2.1 Introduction ........................................................................................................................... 19 4.2.2 IEC/BCC Messages used in the campaign ....................................................................... 20 4.2.3 Target audiences ................................................................................................................... 21 4.2.4 Channels used ........................................................................................................................ 21 4.2.5 Implementation of the BCC Campaign ............................................................................ 21 4.2.6 Influence of IEC/BCC on IRS uptake ................................................................................ 23 4.2.7 Influence of IEC/BCC on desired malaria control behaviors ...................................... 23 4.2.8 Conclusions and lessons ..................................................................................................... 27 4.2.9 Recommendations ................................................................................................................ 27 4.3 How have structures and partnerships developed under NMCP been strengthened to continue implementing IRS interventions? ........................................................................................... 28 4.3.1 Introduction ........................................................................................................................... 28 4.3.2 Structures and partnerships in place for IRS work ....................................................... 28 4.3.3 Project quantitative achievements in strengthening IRS capacity of national and district level structures and partnerships ........................................................................................ 30 4.3.4 Strengthening the capacity of national level structures and partnerships ................ 31 4.3.5 Strengthening the capacity of PSO/NGOs in IRS .......................................................... 33 4.3.6 Strengthening District and lower level structures ........................................................ 34 4.3.7 Sustainability of IRS activities and outcomes .................................................................. 37 4.3.8 Conclusions and lessons ..................................................................................................... 39 4.3.9 Recommendations ................................................................................................................ 40 4.4 What are stakeholders’ perceptions on the effectiveness of IRS? ..................................... 41 4.4.1 Introduction ........................................................................................................................... 41 4.4.2 Perceptions about the effectiveness of IRS ..................................................................... 42 4.4.3 Benefits from indoor residual spraying ............................................................................ 44 4.4.4 Perceptions about the negative effects of the spraying ................................................ 45 4.4.5 Willingness to use IRS in future ........................................................................................ 47 4.4.6 Willingness to pay for Spraying ......................................................................................... 48 4.4.7 Conclusions and lessons ..................................................................................................... 49 4.4.8 Recommendations ................................................................................................................ 49 4.5 How has the program monitoring system, including environmental compliance and the CLA approach supported program management, learning and adaptation? ................................ 49 4.5.1 Introduction ........................................................................................................................... 49 4.5.2 Monitoring, evaluation and learning system in the IRS II project ............................... 49 4.5.3 Collaborative Learning and Adaptation ........................................................................... 52 4.5.4 Lessons learnt from the project’s work and how this has enabled adaptation ...... 53 4.5.5 Conclusions and lessons ..................................................................................................... 55 4.5.6 Recommendations................................................................................................................ 55 5 Annexes ................................................................................................................................................... 56 Record name and position of informant; duration in this position, date and place of interview, Note taker/recorder, interviewer ....................................................................................................... 125 List of Tables Table 1: Sample districts ................................................................................................................................. 4 Table 2: Distribution of sample households for the household survey by district ............................ 4 Table 3: Overview of Uganda IRS project II mid-term achievement of targets and core indicators .............................................................................................................................................................................. 9 Table 4: Susceptibility Results in six districts – September 2013 ....................................................... 11 Table 5: Examples of project use of monitoring data to improve program performance ............ 51 Table 6: Adaptations made by the project as a result of learning ...................................................... 53 List of Figures Figure 1: Composition of established IRS structures at district, sub-county, parish and village levels ........................................................................................................................................................................... 10 Figure 2: Project adherence to WHO safety standards ....................................................................... 13 Figure 3: Extent to which the project meets standards in disposal of liquid wastes ...................... 14 Figure 4: Effects of IRS on malaria incidence and indoor mosquito densities in Tororo district 15 Figure 5: Proportion of pregnant women that received the first dose of IPTp in IRS II project districts and non-project districts .............................................................................................................. 26 Figure 6: Structures for implementing IRS in Uganda ............................................................................ 28 Figure 7: IRS Project’s performance on PMP indicators by Dec. 2015 ............................................. 30 Figure 8: IRS capacity improvements in the new project districts between June 2015 and Dec 2015 .................................................................................................................................................................. 35 Figure 9: Household respondent’s perceptions about the effectiveness of IRS .............................. 42 Figure 10: Household respondents’ perceptions about the effects of the insecticide on men’s sexual drive and women’s fertility ............................................................................................................. 46 iv ACRONYMS/ ABBREVIATIONS AMELP ASSIST BCC CAO CBO CDC CDFU CHC CLA CoP CSO DHE Activity Monitoring, Evaluation and Learning Plan Applying Science to Strengthen and Improve Systems Behavior Change Communication Chief Administrative Officer Community Based Organization Centers for Disease Control and Prevention Communication for Development Foundation Uganda Communication for Healthy Communities Collaborative Learning and Adaptation Chief of Party Civil society organization District Health Educator DHIS DHO DHT DLG DQA EMCAB FAQs FGD FY GEMS GIS HH HMIS IEC IPC IPTp IQK IRS ITN LC LLIN MIS MoH MOP NDA NEMA NGO NMCP PDA PMI District Health Information System District Health Officer District Health Team District Local Government Data Quality Assessment Environmental Monitoring and Capacity Building in Vector Control Interventions project Frequently Asked Questions Focus Group Discussion Financial Year Global Environmental Management Support project Geographical Information System Household Health Management Information System Information, Education and Communication Interpersonal Communication Intermittent Presumptive Treatment during Pregnancy Insecticide Quantification Kits Indoor Residual Spraying Insecticide Treated Net Local Council Long Lasting Insecticide Treated Net Malaria Indicator Survey Ministry of Health Malaria Operational Plan National Drug Authority National Environment Management Authority Non-governmental Organization National Malaria Control Program Personal Data Assistant President’s Malaria Initiative v PMP PPE Performance Management Plan Personal Protection Equipment PSO RDC RDT SDS SEA SEDC SK SMS SO SOW Private Sector Organization Resident District Commissioner Rapid Diagnostic Test Strengthening Decentralization for Sustainability Supplementary Environmental Assessment Socio-Economic Data Centre Ltd. Store Keeper Short Messaging Service Spray Operator/Operation Statement of Work SPSS TWG UBOS UDHS UPCA USAID USG UVRI VCD VCO VHT WHO WHOPES Statistical Package for Social Scientists Technical Working Group Uganda Bureau of Statistics Uganda Demographic and Health Survey Uganda Pest Controllers Association United States Agency for International Development United States Government Uganda Virus Research Institute Vector Control Division (of Ministry of Health) Vector Control Officer Village Health Team World Health Organization World Health Organization Pesticide Evaluation Scheme vi EXECUTIVE SUMMARY PROJECT BACKGROUND The Uganda Indoor Residual Spraying (IRS) Project – Phase II started in June 2012 in 10 districts in Northern Uganda and is currently on-going in 14 districts in Northern and Eastern Uganda with support from the U.S. Agency for International Development (USAID)/President’s Malaria Initiative (PMI)1 and the Department for International Development–United Kingdom (DFID-UK). The major focus for IRS program in Uganda is to rapidly build and strengthen local IRS implementation capacity in the public sector, and to establish a system for regular and timely high quality IRS services that cover at least 85% of all targeted structures. Implementation of the IRS is in collaboration with the Uganda National Malaria Control Program (NMCP) EVALUATION PURPOSE AND EVALUATION QUESTIONS The main purpose of this midterm evaluation was to assess the extent to which the IRS project was achieving its objectives during the 2012 – 2015 period, and document emerging lessons that could be scaled up during the remaining period of the project. The evaluation sought to answer five questions, namely; (i) To what extent is the IRS program meeting the high quality standards set out in the program design? (ii) To what extent did the Behavioral Change Communication (BCC) campaign influence desired behaviors in net use, prevention of malaria in pregnancy, seeking treatment early and other kinds of malaria prevention and management practices? (iii) How have structures and partnerships developed under NMCP been strengthened to continue implementing IRS interventions? (iv) What are stakeholders’ perceptions on the effectiveness of IRS? (v) How has the program monitoring system, including environmental compliance and the collaborating learning and adapting approach supported program management, learning and adaptation? The potential users of the evaluation findings include PMI Uganda, USAID Uganda staff, other U.S. government agencies; USAID funded implementing partners (IPs), Ministry of Health (MoH) specifically the NMCP and other stakeholders with interest in malaria programing. EVALUATION METHODS The evaluators used a combination of qualitative and quantitative approaches. Data was drawn from both secondary and primary sources. Primary data was collected from community members, district and sub-county staff and political leaders, Ministry of Health (MoH) staff, staff from private sector agencies involved in IRS, and relevant staff of PMI. Project reports and other relevant documents were reviewed, and secondary data on key malaria indicators drawn from the Malaria Indicator Survey (MIS) 2013-2014, and the MoH’s Health Information Management System (HMIS) DHIS2 were analyzed. This evaluation was conducted in a sample of 13 IRS II project districts. Data was collected through a household survey with 1,412 households, 53 focus group discussions (FGDs), in-depth 1 The PMI is led by the USAID and implemented together with the U.S. Department of Health and Human Services’ Centers for Disease Control and Prevention (CDC). vii interviews with over 100 key informants, semi-structured interviews with spray operators and store keepers, and observation of IRS sites and stores. Quantitative data was entered using Epidata and analyzed using SPSS and STATA, while qualitative data was analyzed using Nvivo. FINDINGS AND CONCLUSIONS Adherence to the high-quality standards set at project design The IRS project has made considerable progress to date in the use of appropriate technologies and tools such as GIS, PDAs, smart phones and short message systems (SMS) which have enhanced cost effectiveness in program planning and management. The project has engaged the MoH/NMCP, districts, PMI and NEMA in the selection of insecticides, equipment and accessories for IRS, and in accordance with WHO and GoU guidelines. The project has conducted spray operations in a timely manner and adhered to set schedules. The project has adhered to inclusiveness standards by effectively engaged the MoH, NMCP, VCD, and relevant government agencies such as NEMA and NDA in planning, implementing, supervising and monitoring IRS operations. The project established district IRS teams and district and sub￾county task forces which play leading roles in micro/operational planning and mobilizing for and monitoring IRS implementation. Engagement of community leaders such as political, religious, security and opinion leaders in planning has contributed to community acceptance of the project. In accordance with safety standards requirements, the project conducted a Supplementary Environmental Assessment (SEA) that has guided environmental compliance activities in the districts. The project developed and implemented environmental compliance and mitigation monitoring plans and procedures in accordance with the PMI-issued best practices manual and Uganda environmental regulations. The project has established and maintained a wide network of district and parish stores and operational sites, provided personal protection equipment to spray and stores staff, incinerated solid wastes, and provided safety information to households before, during and after spraying. The project has conformed to effectiveness standards by recruiting and deploying adequate and competent staff for IRS management and implementation. In collaboration with the NMCP and other stakeholders, the project has also established infrastructure for implementation of IRS activities. The project, through its IEC/BCC partner, Communication for Development Foundation Uganda (CDFU), established systems and structures to provide information about IRS and other malaria prevention and control measures. Overall, the project has largely adhered to the standards set at project design. Influence of BCC on malaria control behaviors Abt. Associates partnered with CDFU, which provided expertise and support for IEC and SBCC activities and community mobilization. The IEC/SBCC campaign was designed based on the National Malaria Communication Strategy and the National Communication Strategy for Indoor Spraying (2008). The campaign initially focused on IRS rationale, benefits and procedures, but from late 2014 also focused on other malaria prevention practices including LLIN use, prevention of malaria in pregnancy and early care seeking for malaria. Key BCC channels used included radio talk shows, radio spots and announcements, fact sheets, later on incorporating other methods such as door to door mobilization, interpersonal communication (IPC), community dialogues, and viii use of SMS. Key actors in information dissemination included district IRS teams, district and sub￾county taskforce members, spray operators, LCI chairpersons, parish store mobilisers, VHTs, local community based organizations (CBOs) and non-governmental organizations (NGOs). The project also collaborated with the USAID-funded Communication for Healthy Communities (CHC) who provide support to intensify the reach of BCC messages. Most respondents in the household survey had heard about the messages on malaria prevention and majority of those who had received messages on malaria prevention reported that the messages had influenced their perception and actions on dealing with malaria prevention and treatment. However qualitative data indicated mixed results, with some community members reporting that they used LLINs after spraying, while others abandoned LLINs after spraying because they perceived the threat of malaria to be low. In some communities, continued use of nets was reportedly limited by lack of nets in some households despite the continuing distribution of nets to pregnant women during antenatal care visits at health facilities. Analysis of HMIS data shows no marked difference in the trends of use of IPTp12 between the new project districts, old project districts and non-project districts. FGD and key informant data show that most community members seek care promptly if they suspect malaria. Overall, while the evaluation methodology used could not allow adequate linking of BCC interventions to behavior change, the evaluation team concludes from the mixed results presented above that the BCC campaign has so far not substantially influenced malaria prevention behaviors partly due to the short time that the campaign focusing on other malaria prevention practices was implemented before this evaluation. The project has nevertheless intensified its BCC campaign and promises to achieve the desired outcomes. Strengthening structures and partnerships for IRS At national level, the IRS project II did not create new structures, but worked with existing structures, specifically the relevant Technical Working Groups (TWGs) and the individual public sector agencies, namely, MoH/NMCP, the MoH/VCD, NEMA, and NDA. The project’s participation in the TWGs provided an opportunity for the project team to share updates and results from the IRS activities, though there is no evidence of strengthened IRS capacity within the TWGs. The project has strengthened the capacity of MoH/NMCP mainly through mentorship and skills training of MoH/NMCP staff in areas of entomological and environmental monitoring, IRS planning and training of lower levels; infrastructural improvements of insectaries, and support to formulate an integrated vector control strategy. On the other hand, capacity building of MoH/NMCP and VCD has not been systematically guided by a capacity building plan, with the effort to do a capacity needs assessment and formulate a capacity building dating only as recent as 2015. Similarly, not all the capacity building activities listed in the project’s statement of work have been implemented. Structured sharing of IRS reports and results with the NMCP was also found inadequate. Private sector organizations (PSOs) in the context of IRS included small-sized companies under the umbrella of the Uganda Pest Controllers Association (UPCA), as well as the large companies such as tea and sugar estates. The project conducted IRS related trainings for members of UPCA, 2 First dose of the intermittent presumptive treatment with SP Fansidar during pregnancy. ix CBOs, and large private sector companies. These trainings did not adequately involve the trainees in practical spray operations. Trained PSOs also continue to lack specialized equipment such as pressurized pumps and the recommended insecticides to implement IRS to the required standards. There is also a lack of a policy and legal framework to regulate the work of private sector providers of IRS. The structures for implementing IRS at district and sub-county levels included IRS task forces set up by the project, and IRS teams made up of staff from relevant local government departments. The project trained these in the IRS core areas of competence. Whereas the districts have built up a critical mass of human resources and skills that can implement IRS, they lack the equipment, supplies and financial resources to continue IRS when needed. At the community level, the project has worked with VHTs most of whom were also recruited and trained as spray operators. It has also trained local political, cultural and religious leaders in social mobilization for IRS. The project has worked with the multiple stakeholders at national, district and community levels from both the public and the private sectors, in what may be termed as a broad partnership for IRS. However, this partnership remains weak, and financially dependent on donor funding. The Gou has not shown commitment to contribute resources to continue IRS partly because of the perception of IRS as being very costly. On the other hand, the project has done well integrating other malaria prevention and control interventions, such as continuous use of LLINs, early care seeking for suspected malaria, and prevention of malaria in pregnancy. These interventions are the cornerstone to sustaining the gains from IRS. Stakeholders’ perceptions of the effectiveness of IRS Majority of the household respondents in both new (89%) and old (95%) project districts contended that IRS is effective in killing malaria-transmitting mosquitoes and thereby reducing malaria. Qualitative data from the community and district stakeholders shows concurrence that indoor residual spraying against mosquitoes has been effective in reducing malaria burden in their families and communities. Similarly, stakeholders at national and district levels were unanimous that IRS has been effective in reducing the burden of malaria in the project districts, in concurrence with malaria surveillance data held by the MoH. Community members and key informants also cited a variety of other benefits from IRS. As a result of these perceptions this evaluation found a lot of demand for IRS especially in the old project districts in Northern Uganda. Concerns remain about nuisance mosquitoes and bedbugs especially in the Eastern districts. How the program monitoring system and collaborative learning and adaptation (CLA) has supported program management, learning and adaptation The project’s MEL system is comprehensive and included production and implementation of annual plans, routine data collection, and production of monthly, quarterly and annual reports. The project has formulated a learning agenda that is included in its Annual Monitoring, Evaluation, and Learning Plan (AMELP). The project has documented a number of success stories and shared them with stakeholders. The project has done considerably well learning from its work and using the lessons to make improvements as several examples cited in this report demonstrate. CLA is however still a new concept to many project staff. The MoH/NMCP is also yet to take ownership of the learnings and adaptions made by the project. x RECOMMENDATIONS Adherence to quality procedures  The project successfully piloted the use of GIS and PDA in three districts and should expand the use of the technologies to all IRS targeted districts.  The project should continue to adhere to all WHO, GOU and US guidelines and standards on IRS. Behavior Change Communication  Project and district IRS sustainability plans should include continuous sensitization of communities even after IRS phase-out to promote other malaria prevention practices.  Given the importance of other malaria prevention measures such as LLINs, in post-IRS periods, the MoH/NMCP and the project should work with other on-going programs to support communities to access nets, and offer support to households to hang and repair nets Strengthening of structures and partnerships for IRS sustainability  In the remaining period, the project should conduct all the capacity building activities for MoH/NMCP as stated in PMP updated in August 2015, and in consultation with PMI and NMCP support the implementation of the capacity building plan for NMCP developed in 2015.  The project should provide dashboard tool skills to MoH/NMCP. The project should work with MoH/NMCP to share ideas on how best to utilize the dashboard tool and to integrate IRS data into the DHIS/HMIS.  The MoH should advocate and lobby MOFPED, parliament and other decision makers to ensure allocation of funds to start off a government-funded IRS component that can be increased over time. As an advocacy tool, the MoH/NMCP should undertake a cost￾effectiveness analysis of using IRS.  As part of building sustainability, the project and MoH should develop continuous engagement with government institutions that have a large number of personnel such as the police, the prisons and the army as well as large PSOs to build their capacity in IRS, and provide support￾supervision to those providing IRS.  The IRS project should use the dashboard tool indicators for the six IRS project technical areas as capacity building outcome indicators so as to provide the true picture of the targeted districts’ and communities’ IRS capacity growth. Perceptions on IRS effectiveness  The project should continue the IEC/BCC campaign to address people’s questions and concerns about IRS, such as those relating to nuisance mosquitoes, bedbugs and need for continued LLIN use after spraying; and to dispel myths and misconceptions about IRS.  The GoU/MoH should take advantage of the high demand for IRS in Northern Uganda to promote private provision of IRS through private sector agencies for those who can afford it. MEL and CLA  PMI should provide CLA training and mentorship to the project staff.  The project’s learning processes should be opened up more to involve other stakeholders including the MoH/NMCP. 1 1 PROJECT BACKGROUND Uganda has the third highest number of P. falciparum infections in sub-Saharan Africa, and some of the highest reported malaria transmission rates in the world. There is stable, perennial malaria transmission in 90–95% of the country3 . Approximately 90 percent of Uganda’s 38 million people live in areas with high malaria transmission (>1 case per 1000 population), and the World Health Organization (WHO) estimates there are about 17,000 malaria-related deaths annually. Nearly half of hospital in-patient deaths among children under five are attributed to malaria and there are even more unreported malaria-related deaths that occur at home (WHO, World Malaria Report 20134 ). Malaria transmission is persistently high in some areas of Northern Uganda. For example, before the implementation of (Presidential Malaria’s Initiative) PMI’s indoor residual spraying (IRS) in the 10 districts, Apac district reported an entomological inoculation rate of 1,6005 . That is more than four infected mosquito bites every night and is among the world’s highest recorded. In most of Uganda, favorable temperature and rainfall allow intense vector propagation and perennial malaria transmission. Indoor Residual Spraying (IRS) is one of the Government of Uganda’s key interventions under the Malaria Reduction Strategy (2014-2020) focussing on both low and epidemic-prone areas (to prevent malaria epidemics) and high transmission endemic areas, accompanied by environmental management where feasible and effective. Indoor residual spraying is an effective method in malaria vector control as its purpose is to reduce longevity of adult mosquitoes and hence human/vector contact and its continuous use leads to significant reduction in malaria transmission levels. Malaria prevention and control are major foreign assistance objectives of the U.S. Government (USG). Under the United States President’s Malaria Initiative (PMI), USG supported the initial IRS project in Uganda between 2009 and 2012 in 10 districts of; Kitgum, Lamwo, Pader, Agago, Apac, Kole, Oyam, Amuru Nwoya and Gulu. The current project, Uganda Indoor Residual Spraying (IRS) Project – Phase II started in June 2012 continued in the 10 districts and is currently being scaled up in an additional 14 districts of; Otuke, Lira, Alebtong, Dokolo, Kaberamaido, Amolatar, Serere, Pallisa, Kibuku, Budaka, Namutumba, Butaleja, Bugiri and Tororo by 2017 (see map in Annex 1B). The major focus for IRS program in Uganda is to rapidly build and strengthen local implementation capacity in the public sector in the areas selected for this intervention. Additionally, it is to establish a system for regular and timely high quality IRS services that cover at least 85% of all targeted structures. Specifically, the purpose of Uganda Indoor Residual Spraying (IRS) Project is to achieve Presidential Malaria Initiative (PMI)/Uganda targets in IRS through: (1) implementing a high quality IRS program, (2) conducting comprehensive monitoring and evaluation of the IRS program and (3) developing the national capacity to conduct IRS. Implementation of the IRS is in full collaboration/ partnership with the Uganda National Malaria Control Program (NMCP). 3 World Health Organization (2014): World Malaria Report. Geneva: WHO 4 http://www.who.int/malaria/publications/world_malaria_report_2013/en/ 5 Okello PE, Van Bortel W, Byaruhanga AM, Correwyn A, Roelants P, Talisuna A, D'Alessandro U, Coosemans M. 2006. Variation in malaria transmission intensity in seven sites throughout Uganda.Am J Trop Med Hyg. Aug;75(2):219-25 2 2 EVALUATION PURPOSE & EVALUATION QUESTIONS 2.1 Evaluation Purpose The main purpose of this midterm evaluation was to assess the extent to which the IRS project was achieving its objectives during the 2012 – 2015 period, and document emerging lessons that could be scaled up during the remaining period of the project. 2.1 Evaluation Questions The evaluation sought to answer the following questions: 1. To what extent is the IRS program meeting the high quality standards set out in the program design? 2. To what extent did the Behavioral Change Communication (BCC) campaign influence desired behaviors in net use, prevention of malaria in pregnancy, seeking treatment early and other kinds of malaria prevention and management practices?6 3. How have structures and partnerships developed under NMCP been strengthened to continue implementing IRS interventions?7 4. What are stakeholders’ perceptions on the effectiveness of IRS? 5. How has the program monitoring system, including environmental compliance and the collaborating learning and adapting approach supported program management, learning and adaptation? 6 Initially, the BCC campaigns were focused on raising awareness around IRS, but this was subsequently modified to promote behavioral change in malaria prevention and management. 7 Structures and partnerships under NMCP are both public and private companies/ organizations at the national, district and community levels. 3 3 EVALUATION METHODS 3.1 Evaluation approach The evaluators used a combination of qualitative and quantitative approaches. A non￾experimental evaluative design was used based on comparisons of before and after situations. Data was collected from both secondary and primary sources. 3.2 Sampling design for the evaluation The evaluation team used a multi-stage cluster sampling technique to select sample districts, sub￾counties, parishes, villages. The clusters in this case were project districts (new and old), sub￾counties, parishes and villages. 3.2.1 Sampling of districts This evaluation was conducted in a sample of 13 IRS II project districts. The sample districts were purposively selected based on different criteria including location, date when IRS implementation started, the level of program uptake, whether spraying was going on at the time of data collection or not, and other district-specific characteristics. Based on these criteria, sample districts were selected from three categories as follows:  Five districts were selected from the 10 initial IRS project districts in Northern Uganda where IRS was phased out in 2014.  Five districts were selected from the new project districts in Northern and Eastern Uganda where IRS implementation started in 2014 and 2015 and the project is still on-going – but which were not scheduled for spraying during the months of data collection for this evaluation. This consideration was important to ensure that the household survey and other data collection activities do not get disrupted or biased by the spraying activities and vice versa. Selection of sample districts from this category also considered whether the uptake of IRS was good or poor in order to get a good mix of experiences.  Three districts were selected from the new project districts in Northern and Eastern Uganda where IRS implementation started in 2014 and 2015 and the project is still on-going – and which were spraying during the time of data collection for this evaluation. These districts were sampled specifically to undertake observation of IRS sites and stores (IRS parish stores are only functional during spray time), and to interview stores personnel. No household interviews or focus group discussions (FGDs) were held in these districts. In Otuke, an additional interest was to explore the persistent high level of malaria cases following rounds of spraying. Table 1 below shows the sample districts, while more details on the characteristics of the sample districts is provided in Annex IIA. 4 Table 1: Sample districts Category of district District Old IRS II project districts Amuru, Gulu, Kitgum, Apac, Oyam New IRS II project districts (not spraying at time of the evaluation) Amolator, Lira, Budaka, Serere, Tororo New IRS II project districts (on-going spraying at time of the evaluation) Dokolo, Kibuku, Otuke 3.2.2 Sampling of sub-counties, parishes and villages From each of the sampled districts, two sub-counties were selected, one rural and one urban/peri-urban. From each selected sub-county, two parishes were selected and in each parish, between one to three villages were selected, depending on the number of sample households needed to raise the district household sample quota computed for this evaluation. More details on the sampling procedures used to select these units are provided in Annex IIB. 3.2.3 Sampling of household respondents A sample of 1,412 household respondents were selected and interviewed in the household survey (see sample distribution by district in Table 2). Systematic sampling was used to select households from villages using lists of households provided by the village Local Council (LC) leaders and Village Health Teams (VHTs) as the sampling frames. In each selected household, only one adult respondent either a household head, spouse or another adult member was interviewed. The list of sample sub-counties, parishes and villages is provided in Annex IIC. Table 2: Distribution of sample households for the household survey by district Category of district District HH interviews conducted %ge Old IRS project districts Amuru 87 6.2 Kitgum 96 6.8 Apac 161 11.4 Oyam 175 12.4 Gulu 196 13.9 New IRS project districts Amolator 67 4.7 Budaka 85 6.0 5 3.2.4 Data collection methods and tools Data for the evaluation was collected through: (i) a desk review of relevant project and policy documents, (ii) a household survey among a sample of household, (iii) in-depth interviews with key informants selected from key stakeholders at community, sub-county, district and national levels, (iv) focus group discussions with community leaders, VHTs, male and female adult community members, youths, and pregnant women who had a child under the age of one, (v) semi-structured interviews with Spray Operators and Store Keepers, and (vi) direct observation at IRS sites and stores. More details on these methods, and a lists of the documents reviewed, key informants interviewed and FGDs conducted are provided in Annexes IIC - IIF. The data collection tools consisted of a household interview schedule (interviewer-administered questionnaire), a semi-structured questionnaire for Spray operators and Store Keepers, as well as key informant, FGD and observation guides. Copies of these are attached in Annex VII. 3.2.5 Data management and analysis All completed questionnaires were edited to ensure completeness and legibility of responses. Open-ended questions were post-coded, data was entered using Epidata Software, and analyzed using SPSS and STATA. MS Excel was used to produce charts and graphs. Quantitative data extracted from secondary sources was analyzed in MS Excel and Stata. Analysis was mainly done at descriptive, univariate level. Secondary data from the Malaria Indicator Surveys (MIS), Uganda Demographic Health Surveys (UDHS) and District Health Information System II (DHISII) were analyzed and used for comparisons purposes where relevant. For qualitative data, all notes were transcribed and typed in MS Word, and checked for completeness and originality. Qualitative data was analyzed using Nvivo software. Analysis followed identified themes such as knowledge of malaria prevention and control, exposure to IEC/BCC messages, attitudes and perceptions towards IRS, challenges and benefits experienced, and malaria prevention and control methods being practiced at household level. 3.2.6 Limitations One major limitation in this evaluation was in respect to addressing the evaluation question that focuses on the influence of BCC on other malaria prevention behaviors namely, use of LLINs, prevention of malaria in pregnancy, and early treatment seeking for malaria. BCC activities under the IRS project phase II initially focused on promoting IRS, and only started focusing on other Serere 110 7.8 Lira 199 14.1 Tororo 236 16.7 Total 1,412 100.0 6 malaria prevention behaviors in late 2014 to early 2015. As such the time period of implementation was too short to make sound judgements about the impact of such interventions. Besides, there were other on-going programs by the GoU/MoH, NGOs and other projects also promoting similar behaviors, including for instance routine health education and distribution of LLINs by health facilities. This made it difficult to attribute any observed changes in behavior to the project’s BCC campaign. Nevertheless, the evaluation team was able to make reasonable connections between the implemented BCC campaign and the reported malaria prevention behaviors, to enable drawing some insights and lessons for the project. 7 4 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 To What Extent is the IRS program meeting the high quality standards set out in the program design? 4.1.1 Introduction Effective IRS operations require adequate consultative planning that involves a broad partnership of stakeholders; sufficient skills, logistics and infrastructure; quality operations based on standard operating procedures and field supervision8 . The design of the IRS program entailed an elaborate set of standards that were grouped into four categories, namely, quality standards, inclusiveness standards, safety standards, and effectiveness standards. This section of the report discusses the extent to which the IRS program met the four high quality standards that were set at program design. 4.1.2 Quality Standards According to the IRS project Statement of Work (SoW) as well as the Request for Proposals (RfP) for this evaluation, the quality standards included use of appropriate technologies and tools for geographical reconnaissance, tracking and data management; working with the NMCP, PMI, districts and other relevant authorities to select appropriate insecticide, equipment and accessories; timeliness during the spraying exercise as recommended by the WHO technical standards; and cost-effectiveness and use of evidence in achieving the required targets and goals. The details of each of the quality standards and the extent to which the project met them are discussed below. (i) Use appropriate technologies and tools for geographical reconnaissance, tracking and data management Evaluation interviews and project documentation show that the project has made considerable progress to date in the use of appropriate technologies and tools such as GIS and PDAs. The project has used geocoding of household structures initially in three districts for quantifying IRS insecticide, spray operators, appropriate location of parish stores and for developing maps at district, sub-county and parish levels to enhance operations and monitoring. For example, in FY3, the project used GIS technologies to generate a total of 145 maps: 32 district level maps, 87 sub￾county level maps and 26 parish level maps (see Annex IIIA for sample maps). These mapping systems have enhanced cost effectiveness in program planning, implementation, monitoring and reporting. For instance, the project used the maps and the data to obtain a more precise number of sprayable houses to help in detecting and preventing fraudulent reporting by spray personnel. The project annual report for 2014 shows that 26.7% more houses were found when enumeration was done using new technologies than those found during previous spray rounds; the difference being attributed to some households not being captured in the manual enumeration due to being located in hard- to- reach areas (see Annex IIIB for comparison of sprayable houses 8 WHO (2013). Indoor Residual Spraying. An Operational Manual for Indoor Residual Spraying (IRS) for Malaria Transmission Control and Elimination. Geneva 8 enumerated before and after introduction of geo-coding). The geo-coordinates of the households were also used in reviewing the location of parish stores. In addition, the project has adopted the use of smart phones and short message systems (SMS) to transmit real-time spray data on key indicators such as spray coverage, insecticide usage rate and spray progress on a daily basis during the spray operations. The smart phones use the Open Data Kit (ODK) platform which enabled input of all the parameters captured on the daily spray card, in contrast to the SMS method which captured only a few of the parameters. Using the smart phones, the Store Keepers summarized data on spray performance on a daily basis and transmitted it to the Field Coordinators. The daily summaries helped the Field Coordinators to monitor key indicators like spray coverage, insecticide usage rate and the spray progress on a daily basis during the spray round, and immediately address any issues. As a result, there was improved quality in the monitoring of the spray operations resulting in quality IRS services in the districts of operation. The use of smart phones and SMS also reduced costs that would otherwise be associated with numerous physical monitoring visits and data collection. (ii) Work with the NMCP, PMI, districts and other relevant authorities to select appropriate insecticide, equipment and accessories The findings from the interviews with the national level staff including the entomologists at both the national and district level as well as the program documentation confirmed that there was stakeholder engagement in the selection of WHOPES approved insecticides, equipment and accessories for IRS, and this was done in accordance with WHO and GoU guidelines. Among the stakeholders engaged included NMCP, districts, PMI and NEMA. The Vector Control Division (VCD)’s capacity to routinely generate entomological data on insecticide resistance to guide selection of suitable insecticides for IRS was strengthened as a result. In addition, the project engaged with NEMA to approve the SEA reports to guide the selection of the insecticides in the 14 new project districts and appropriate location of soak pits (see Annex IIIC for copy of SEA approval). Through the strengthened stakeholder engagement, the project engaged the National Drug Authority (NDA) to register new insecticides as required by GoU guidelines. During the project implementation, the project has moved from pyrethroids to carbamates and recently to organophosphates on the basis of NEMA recommendation and reported insecticide resistance in the malaria vectors. (iii)Timeliness during the Spraying exercise as recommended by the WHO technical standards A review of the project records and interviews with the project staff revealed that the spray operations were timely conducted. Consultative planning meetings with district IRS team members discussed spray schedules and operational plans for upcoming spray rounds. The needs assessment in each district to estimate the supplies were conducted and procurement of the required spray logistics such as the insecticide and equipment needed for spraying, distribution and retrieval process of materials followed the set schedules and was done on time to ensure timely commencement and completion of spraying. All spray rounds were completed on schedule (see Annex IIID for a comparison of planned and actual spray schedules). 9 iv. Evidence-based and cost-effectiveness in achieving the required targets and goals Documentation and interviews with key informants show that the project and district stakeholders have developed and implemented annual plans and PMP and produced monthly, quarterly and annual reports on program implementation. Supervisory checklists and data quality audits have enhanced data quality and evidence from annual reports indicates achievement to date of all targets and core indicators on structures sprayed, populations covered and amount of insecticide used (Table 3). The reports on IRS performance have informed evidence-based decision for program cost-effectiveness. Table 3: Overview of Uganda IRS project II mid-term achievement of targets and core indicators Year Number of districts sprayed Houses sprayed (% coverage) Amount of Insecticide used Population protected (% coverage) Children <5 protected Pregnant women protected 2012/3 10 858,415 (92.3) 324,015 2,572,996 (93.1) 547,084 74,116 2013/4 10 870,943 (96.5) 322,052 2,581,839 (96.8) 538,264 81,218 2014/5 19 1,608,670 (92.6) 507,035 5,317,458 (95.5) 1,092,343 143,185 Conclusions and lessons  The project has successfully piloted the use of GIS and PDA for effective program planning and implementation.  The selection of insecticide for IRS is evidence-based and is done in accordance with WHO and GoU guidelines.  The adoption of smart phones and SMS technology has reduced costs, strengthened supervision, provided real-time data transmission and reporting for timely decisions. Recommendations  The project successfully piloted the use of GIS and PDA in three districts and should expand the use of the technologies to all IRS targeted districts.  The project should continue to adhere to the WHO, GOU and US standards and guidelines on IRS. 4.1.3 Inclusiveness standards The project design standards for inclusiveness included the requirements that the IRS program is planned, managed and implemented in collaboration with key Uganda Government ministries and departments; and that prior to commencement of spraying, appropriate planning is conducted in collaboration with the NMCP, districts and other relevant authorities. All three annual reports and evaluation interviews with key informants confirm that the project collaborated with the MoH, NMCP, VCD, Ministry of Local Government, and relevant 10 government agencies such as NEMA and NDA in planning, implementing, supervising and monitoring IRS operations (see for instance Annex IIIC – NEMA approval letter). This collaborative planning always started with extensive consultative meetings of COP and MoH/NMCP officials and ensured smooth and timely start, implementation and completion of spray rounds. MoH/ NMCP and VCD and Ministry of Local Government were engaged in planning, supervising, monitoring and capacity building. Other stakeholders who were involved in project activities included NDA and NEMA. NDA was called upon to inspect and license all stores and NEMA was involved in the approval of the SEA report and incinerators for disposal of contaminated solid waste. The engagement of NEMA and NDA significantly enhanced infrastructural capacity such as IRS operation sites and a network of district and parish storage facilities for safe IRS. The project established district IRS teams and sub-county task forces which play leading roles in micro/operational planning and mobilizing for and monitoring IRS implementation (see Figure 1). Engagement of community leaders such as political, religious, security and opinion leaders in planning is considered to have contributed to community acceptance of the project. Figure 1: Composition of established IRS structures at district, sub-county, parish and village levels Collaboration with stakeholders empowered district and sub county authorities to carry out micro planning for IRS. However, the project has not actively engaged other ministries such as that of Agriculture, Animal Industry and Fisheries; Ministry of Finance; and Ministry of Water and Environment. Conclusions and lessons  The project effectively collaborates with NMCP, districts, communities and relevant government agencies in planning IRS implementation.  The project engagement of community leaders such as political, religious, security and opinion leaders in planning is considered to have contributed to community acceptance of the project. •District Health Officer, District Malaria Focal Person, District Health Inspector, District Environmental Officer, District Vector Control, District Supplies Officer, and District Biostatistician District IRS Team •Local Council Chairperson, Sub-county Chiefs, internal security officers, Secretary for Health, Community Development Officer, Health Assistants Sub-county Task Force •Parish Store Mobilizers, LCI, Community and Opinion Leaders, Women and Youth Leaders Parish Task Force •Spray operators, Stoke Keepers, Supervisors, Security Guards, Wash Persons, Spray Team 11 Recommendations  The Project and PMI should discuss and consider the need to engage with other national level GoU ministries and departments, such as the Ministry of Agriculture, Animal Industry and Fisheries. 4.1.4 Safety Standards The project’s safety standards included vector susceptibility and behavior, safety for humans and environment, and efficacy and cost effectiveness. (i) Susceptibility and vector behavior Annual reports and evaluation interviews with the project Vector Control Entomologist and district Vector Control Officers revealed that the project in collaboration with MoH/MCP, VCD worked through six sentinel sites in six different eco‐epidemiological districts of Apac, Hoima, Kanungu, Kitgum, Tororo and Wakiso for monitoring entomological indicators including insecticide resistance. The project collaborated with CDC, MoH/NMCP, VCD and the Uganda Virus Research Institute (UVRI) to conduct two-week specialized training courses for district vector control officers on the use of the WHO tube and bottle bioassay for mosquito insecticide resistance testing and for monitoring transmission rates in IRS targeted districts. This training enhanced VCD capacity to conduct malaria vector surveillance. Over FY1, FY2 and FY3, the project has shown that Anopheles gambiae was fully susceptible to bendiocarb and pirimiphos-methyl and resistant to pyrethroids and DDT in varying degrees in all districts (Table 4). Table 4: Susceptibility Results in six districts – September 2013 Insecticide tested Apac Hoima Kanungu Kitgum Tororo Wakiso DDT 4% 95 34 39 81 41 10 Deltamethrin 0.05% 82 18 53 58 37 44 Etofenprox 0.5% 84 Lambdacyhalothrin 0.05% 56 27 40 21 Permethrin 0.75% 85 31 45 60 Pirimiphos-methyl 1% 100 100 100 100 100 100 Bendiocarb 0.1% 99.6 95 100 100 93.9 85 (ii) Safety for humans and the environment Evaluation interviews with key district environmental officers, project environmental officers and project documentation show that the Uganda IRS project phase II conducted SEA covering 14 new districts to guide all environmental compliance activities in the districts. The SEA and environmental assessments helped the project to relocate stores and soak pits that had been established at school premises. 12 The project developed and implemented environmental compliance and mitigation monitoring plans and procedures in accordance with the PMI-issued best practices manual9 and Uganda environmental regulations. For every spray round, the project in collaboration with stakeholders has conducted routine internal environmental compliance inspections of stores, washing bays and soak pits prior to, during and after spraying. The project collects environmental compliance data from observations of 30 percent of all the environmental soak pits in each district (see Annex IIIE for sample report from soak pit inspections) and uses the results to take remedial and mitigation actions to address safety. The project has engaged the Environmental Monitoring and Capacity Building in Vector Control Interventions project (EMCAB) and the Global Environmental Management Support project (GEMS) to conduct independent evaluations of the environmental compliance requirements for IRS. To ensure safety and security of IRS commodities such as the insecticides, the project established a network of storage facilities at district and parish/operational sites. By end of 2015, there was a total of 14 district stores and 432 parish stores in all the 14 districts and the project put in place best practices for stock management that uses stores records such as bin cards, store ledgers, vouchers that prevented pesticide pilferage and potential misuse and to promote safer use of insecticide to protect the environment and human health. Interviews with the project Environmental staff at the national and district level confirmed that the spray operators were provided with full personal protective equipment (PPE) and that there was close supervision to ensure proper use and care of PPE (see Photo 1). It was also noted that all the spray team members that are recruited are required to undergo medical examinations including pregnancy tests for women. Adequate information on IRS safety was provided to the spray team prior to the spaying and all house hold members were asked to remove beddings, clothes, food items, food utensils from the house before the spraying. The voice below from one of the Spray Operators echoes what many of them reported: Before spraying I inform them [household members] about the benefits of spraying and the day of spraying. On the day of spraying, I ask them to take out their property and instruct them on how to clean the house after spraying and where to put the rubbish. After spraying, I close the house and tell the members not to enter for about two hours. (SO, Railways Quarters Division, Lira District). 9 Best Management Practices (BMP): Manual for Indoor Residual Spraying (IRS) in Vector Control Interventions; July 2010; President’s Malaria Initiative; U.S. Agency for International Development, 1300 Pennsylvania, Washington DC 20523 Photo 1: Spray Operator fully dressed in personal protective gear 13 Other Spray Operators mentioned that they informed household members about the potential side effects of the insecticide, while others mentioned that they always called the adult household members to witness the mixing of the insecticide. Figure 2 shows results from the household survey regarding the extent to which the project adhered to selected aspects of WHO standards. Figure 2: Project adherence to WHO safety standards The data in Figure 2 show that in both new and old project districts, most standards were reported by household respondents to have been met, with the exception of spraying of all eligible rooms. This was most likely due to the discrepancy between the structures which the WHO and the project recommends to spray and those that the household members would like to have sprayed. Qualitative data shows that household members wanted kitchens, latrines and other structures also sprayed while the project guidelines exclude these. More specifically, majority of the household respondents in both new project old project districts reported that Spray Operators provided information to the household members before the actual spraying, and confirmed that the Spray Operators asked them to remove household property from the house before actual spraying. In addition, the project trained clinicians in emergency management of insecticide poisoning and provided atropine to health facilities in IRS targeted districts. Despite the mitigation, the evaluation team came across a few reports of insecticide pilferage, misuse and alleged deaths of cows and goats from bendiocarb poisoning in Paisho, Odele and 0% 20% 40% 60% 80% 100% 120% Did not spray utensils, water containers, wall hangings, food items, toys, etc Spray operator gave information on the day of spraying before spraying HH members waited for at least 2 hours before entering the house after spraying Household members properly disposed off dead insects & rodents Did not spray non-eligible rooms Sprayed all eligible rooms: sitting rooms, bed rooms, stores for non-fooditems, etc. Old districts (N=560) New districts (N=557) 14 Patigo sub-counties in Gulu. The project responded quickly to such reports and implemented appropriate corrective or mitigation measures. Pre, mid and post spray inspections were conducted to assess the continued adherence to GoU, USG and WHO standards. During the inspection storage practices, field operations and effluent waste disposal structures and sanitary facilities were monitored. The Post-spray environmental compliance inspections ensured proper decontamination and decommissioning of all IRS sites. The evaluation teams’ observation of a sample of the operational sites revealed that each site contained a storage facility, soak pit, washing area slab, and bathing shelters for spray operators. Figure 3 shows results from observation of IRS operational sites. Figure 3: Extent to which the project meets standards in disposal of liquid wastes The data in Figure 3 show that appropriate liquid waste disposal facilities were available in almost all the sites observed during this evaluation. The spray operators who were interviewed during this evaluation were knowledgeable on their duties and they demonstrated proper use and care of pumps including triple rinsing, inspection of pumps for leakage, disposal of washouts, care of solid waste and proper use of PPE. The findings from the interviews with the project staff and available documentation revealed that there was adequate supervision of all the parish stores to ensure that the insecticide contaminated waste was transported from parish stores to district stores in specialized trucks for temporary storage before collection by the Green Label Company trucks to Luwero Industries incinerator which was approved by NEMA. This task is overseen by the Project Environment Officer who ensures that the destruction certificate is signed. The non￾contaminated solid waste on the other hand is burnt at the Municipal Council dumping grounds. During the project FY3 of implementation, the project started sorting plastic wastes for recycling with a company named Gentex, thus reducing the project’s waste disposal costs; the project saved 8.8 Ug.shs. (about USD 2,600) in one year. (iii) Efficacy and cost effectiveness Evaluation interviews with vector control entomologists, vector control officers and review of project reports show that the project in collaboration with Vector Control Division and CDC 0% 20% 40% 60% 80% 100% Wash site available Soak pits available Wash site for PPE available Ablution blocks available Wash site fenced Location of soak pit appropriate Warning signs in place 15 performed WHO contact cone bioassays in the selected sentinel sites to assess the residual efficacy of bendiocarb and the quality of spraying. Project reports have consistently shown high quality of IRS application and a residual efficacy of bendiocarb-sprayed walls of more than 4 months against susceptible malaria vectors. The project has implemented pre and post IRS pyrethrum spray catches in 14 IRS districts and also in the previous 10 IRS districts to monitor changes in indoor resting vector densities as indicator of IRS effectiveness. Project annual reports show remarkable reduction in indoor resting densities in 14 IRS districts and increase in indoor resting densities in the 10 previous IRS districts. Figure 4 shows reduction in number of both malaria vectors and malaria cases with the introduction of IRS in Tororo. Figure 4: Effects of IRS on malaria incidence and indoor mosquito densities in Tororo district Source: MOH/VCD Report 2015 Conclusions  There is an effective insecticide resistance monitoring system  The project implements an effective environmental compliance system that ensures safety of humans and the environment.  There were limited reports of pilferage, misuse, poisoning and accidents, which the project responded to promptly and appropriately. “We used to have a lot of mosquitoes and you would not even sit outside, but right now, the population of mosquitoes has very much reduced”. Key Informant, Serere District. 16  There is no attempt by the project to use insecticide quantification kits (IQK) to monitor quality and decay rate of insecticides which may be cheaper than cone bioassays10 Recommendations  The project should maintain the implementation of an effective environmentally compliant IRS. 4.1.5 Effectiveness Standards Effectiveness standards included ensuring adequate human resources, office space, storage facilities, waste disposal logistics, and skills/ training; programmatic infrastructure for conducting the IRS program and related environmental and entomological activities – including systems to provide information and behavior change communication and appropriate IRS-specific information and education prior to, during and after any campaign; and preparation and execution of logistical plans for IRS-related commodities in compliance with Uganda national policies on IRS and WHO IRS guidelines - including timely procurement, and storage and distribution of all supplies. (i) Adequacy of human resources and skills training.11 The project recruited competent and adequate staff including Chief of Party, a Deputy Chief of party, a Senior Entomologist, an Environmental Compliance Manager, a Capacity Building Manager, an M&E Manager and Field Coordinators (1 Field Coordinator per two districts) to implement the IRS activities and partnered with CDFU to manage the BCC/IEC activities. At operational level, the project recruited and deployed store keepers, spray operators, sub-county supervisors, parish store mobilizers, based on the number of houses to be sprayed in each locality. The project has a specialist for each thematic IRS area supported by well-trained district officers and supportive staff. The project has equipped NMCP and district staff with IRS skills through specialized training, need-based refresher training, in-service training and international education visits. Interviewed district and sub-county key informants all acknowledged that the project established well trained and equipped district IRS teams for overall IRS program management and sub-county task forces for planning, implementing, supervising and monitoring IRS operations. Pre and post training skills assessments for all supervisors and store keepers has ensured adequate skills for IRS targeted districts. 10 Thawer NG, Ngondi JM, Mugalura FE, Emmanuel I, Mwalimu CD, Morou E, P et al. Use of insecticide quantification kits to investigate the quality of spraying and decay rate of bendiocarb on different wall surfaces in Kagera region, Tanzania. Parasites & Vectors; 2015; 242.: doi: 10.1186/s13071-015-0859-5 11 Plan and implement appropriate and necessary training activities for the range of skills needed to safely and effectively implement, monitor, and evaluate IRS including, but not limited to, refresher training for experienced Spray Operators, supervisors, clinicians, entomology technicians, environmental officers administrative and other support staff as well as training for new staff. 17 Project reports and evaluation interviews with district IRS focal persons show that the project operates a transparent process that involves sub-county authorities, the LC’s, the VHT parish supervisors/ health assistants to recruit spray team members. The project has used a cascading training approach to build local capacity in spray techniques, data capture, community mobilization, and environmental compliance. (ii) Adequacy of office space, storage facilities and waste disposal logistics The project collaborated with NMCP, districts, sub-county level structures, communities, NDA and NEMA to establish operation sites and a network of district and parish storage facilities for safe IRS delivery. The different informants we talked to at different levels in all districts confirmed the adequacy of stores and waste disposal facilities. The project has engaged the NDA to inspect and license all stores to ensure that the stores meet the recommended standards for insecticide storage. In some parishes, some community members and institutions (health centers) have offered free premises for use as stores which are then rehabilitated and inspected for suitability. In some districts, stakeholders expressed the need for more parish stores and more spray operators to cater for the big parishes and the hard to reach areas. (iii) Programmatic infrastructure for conducting the IRS program and related environmental and entomological activities As stated above, the project has established a network of storage facilities at district and parish/operational sites and put in place best practices for stock management. All operational sites visited contained storage facilities, soak pits, washing area slabs, and bathing shelters for spray operators. Photo 2: IRS wash areas and soak pits are properly fenced off to prevent public access Photo 3: Spray Operators washing their pumps in a designated and fenced wash area The site managers at the sites understood their roles and responsibilities in the management of effluents. All spray operators interviewed for the evaluation described the proper washing of pumps, disposal of washouts, care of solid waste and proper use of PPE. 18 Evaluation interviews with the district vector control officers confirmed that the project collaborated with CDC, MoH/NMCP, VCD and the Uganda Virus Research Institute (UVRI) and districts to establish six sentinel sites and provided specialized training courses for district vector control officers on malaria vector surveillance. The training has enhanced VCD capacity in malaria entomology. As part of capacity building, the project rehabilitated the Gulu University insectary and recruited and trained two insectary technicians to manage the insectary. Additionally, the project collaborated with CDC, PMI and Tororo district authorities to convert a former Tororo hospital theatre to an insectary for rearing susceptible An. gambiae s.s. for use in cone bioassays. This development has ensured sufficient mosquitoes for cone bioassays in the 14 new districts to monitor IRS quality and residual insecticide effectiveness on sprayed surfaces. The project established a pump maintenance workshop for routine maintenance and calibration of spray pumps which is critical for quality IRS. (iv) Systems in place to provide information and Behavior Change Communication (BCC) to safely and properly implement the program12 The project, through its IEC/BCC partner, CDFU, established systems and structures to provide information about IRS and about other malaria prevention and control measures. CDFU deployed appropriate staff, who work with relevant district staff to implement planned IEC/BCC activities. BCC field staff have also worked through the district IRS teams, sub-county task forces and the media to disseminate information to the communities using multiple channels. More detailed information on these aspects are included in Section 4.2 in this report, which focuses of BCC activities. (iv) Logistical plans for IRS-related commodities in compliance with Uganda national policies on IRS and WHO IRS guidelines13. The project worked with the NEMA, NMCP, and DHO to supervise a logistics system for procurement, distribution, retrieval of supplies and disposal of IRS waste. The logistics system incorporates adequate safeguards to prevent pilferage and misuse of insecticides. All empty insecticide sachets were tracked all the way through the disposal process as a means of minimizing or eliminating diversion of pesticide for unauthorized use. The project, in collaboration with the Vector Control Division (VCD) of MoH, trained casual workers to assess, repair and service spray pumps prior to each spray round. The spray operators who were interviewed during this evaluation reported that adequate equipment and supplies were available to facilitate their work. 12 Provide appropriate IRS-specific IEC/BCC prior to, during and after any campaign. The population to receive IRS shall be informed of all aspects of the IRS program through radio, community meetings, and other less costly and effective methods of communication. 13 This includes, but is not limited to, timely procurement, storage and distribution of all supplies, such as insecticides, spray pumps, protective clothing and any other supplies and equipment required to carry out the IRS operations in Uganda 19 Conclusions and lessons  There are functional district IRS institutional structures and sufficient technical skills, storage facilities and waste disposal sites for effective and safe IRS implementation.  Recruitment of spray teams is transparent and adequate training of the teams has enhanced IRS quality and effectiveness.  Pump maintenance and calibration system has improved quality of IRS.  The project put in place adequate entomological infrastructure to support malaria entomological monitoring.  The project implements an effective IEC/BCC strategy.  There is an effective procurement, storage, distribution and retrieval logistics system that engages relevant government agencies.  There is documented dramatic reduction and increase in vector density in 14 new districts and 10 old districts, respectively, indicating IRS quality and effectiveness. Recommendations  The GoU/MoH should mobilize and commit financial resources and utilize already project￾established technical, logistical and infrastructural resources to sustain IRS from 2018 in the 14 districts. 4.2 To what extend did the Behavioral Change Communication (BCC) campaign influence on desired behaviors in net use, prevention of malaria in pregnancy, seeking treatment early and other kinds of malaria prevention and management practices? 4.2.1 Introduction The IRS Project Phase II was designed to include an intensive IEC/BCC component to inform target communities about IRS before, during and after spraying. The project partnered with Communication for Development Foundation Uganda (CDFU), a specialized social and behavioral communication agency that provided expertise and support for IEC and BCC activities as well as community mobilization. This evaluation established that CDFU did not develop a new communication strategy for the IRS program, but used the National Malaria Communication Strategy and the National Communication Strategy for Indoor Spraying (2008) from the National Malaria Control Program to inform the design of the communication effort. Implementation of IEC/BCC activities was done in partnership with the District Health Teams, District Information Officers and other stakeholders in the districts and sub-counties. CDFU trained the District Health Education (DHE) officers and other members of the district IRS teams, sub-county taskforce members, spray operators, LCI chairpersons, parish store mobilisers, VHTs, local community based organizations and non-governmental organizations on social mobilization to disseminate information on IRS. 20 This evaluation found that the BCC campaign initially focused on raising awareness about IRS (such as IRS procedures, benefits, risks), but this was by late 2014 to early 2015 subsequently modified to promote behavioral change in malaria prevention and management, focusing on key behaviors, namely, proper and consistent LLIN use, prevention of malaria in pregnancy through use of intermittent presumptive treatment in pregnancy (IPTp), prompt testing and treatment seeking for suspected malaria, and male involvement in malaria prevention and treatment. This timing of BCC activities was partly in response to the resurgence of malaria in the old project districts in Northern Uganda. The timing also means that there was a short time between the commencement of BCC activities focusing on other malaria prevention activities and this evaluation. 4.2.2 IEC/BCC Messages used in the campaign Messages, broadcasted before and during the spray exercises were designed to remind the community about planned spray exercises and encouraged them to open their doors to the spray operators, stay at home and provide water for mixing insecticides. They also provided information about IRS, e.g. that IRS is safe, effective, free, and approved by MoH; the benefits of IRS; and information addressing insecticide pilferage, house ghosting, and misconceptions about IRS. The post-IRS messages, on the other hand, encouraged community members to continue using long lasting insecticide-treated nets after spraying, and provided feedback to the community on the last spray round. CDFU also developed leaflets containing frequently asked questions (FAQs) in order to respond to issues and concerns arising from the community. Other types of messages developed include gender-sensitive messages that encouraged both men and women to participate in the IRS program, and the integrated messages on health, which sought to promote other malaria prevention and control practices, besides IRS. Integrated messages encouraged: a) Continued use of LLINs after house spraying, b) Pregnant women to go for ANC for 4 times and take SP Fansidar, c) People to test for and receive treatment for malaria rather than treating themselves, d) People to destroy or clean the bushes and stagnant water around their homes; Messages and materials were translated and pre-tested with members of the intended audience. In collaboration with DHEs, materials and messages were translated into seven local dialects (Ateso Tororo, Ateso Soroti, Lango, Kumam, Dhopadhola, Lugwere and Lunyole). Communication materials were reviewed after spray rounds to incorporate new information and address community concerns as well as rumors and misconceptions that were bound to affect the program. 21 4.2.3 Target audiences The target audience for IEC/BCC included adult men, women and children, targeted as individuals and through groups. Other target groups included local leaders, religious leaders, political and cultural leaders, and other relevant groups of stakeholders identified. There was a deliberate effort to reach school children. Where specific groups of people were identified as resisting IRS, such as members of particular religious sects or organic farmers, these were also specifically targeted to address their concerns. 4.2.4 Channels used The channels used included radio talk shows, radio spots and announcements, and printed materials (fact sheets and posters) (see Photo 4). Based on the lessons learned from previous spray rounds, the project during year three incorporated additional methods of information dissemination and behavior change promotion, including interpersonal communication (IPC) and community mobilization through IRS committees and VHTs, community dialogues, call-in radio talk shows, use of important days (such as malaria day), as well as mobile short message service (SMS) as innovative and cost-effective method to improve outreach and mobilization. In some cases, the project used MoH film vans to mobilize communities for IRS at busy market places and trading centers. Most stakeholders and community members reported that community dialogues, radio talk shows and interpersonal communication (IPC) were the most appropriate and effective channels in disseminating IRS and other malaria prevention messages. Radios were reported to have been particularly good for reaching extensive and diverse audiences, while community dialogues and other interpersonal engagements were useful to address resistance to IRS. There were suggestions that other channels, particularly drama shows, could have been added to the channel mix. 4.2.5 Implementation of the BCC Campaign Campaign implementation started with planning meetings and training of the relevant agents on social mobilization. CDFU used community diagnosis to understand issues in the implementation process; and used the barrier analysis to identify key determinants driving the project to inform planning and design of the behavior change communication campaign. Photo 4: Poster on the household’s role in IRS 22 In general, most district staff and other stakeholders talked to during this evaluation reported that they were consulted and involved during campaign implementation. They also thought the campaign was implemented as planned and resulted into improved coverage of IRS in the target districts. The IRS project also collaborated with the USAID funded Communication for Healthy Communities (CHC) project – implemented by Family Health International (FHI360) – which supported the project in funding some SBCC activities, providing technical assistance to update materials and incorporate messages on duo protection using both IRS and other measures (LLINs and IPTp), reprinting IRS materials to increase reach, and support to orient, deploy and supervise community champions, media practitioners, VHTs, spray operators, religious leaders, teachers, cultural and traditional leaders to integrate malaria duo protection messages in their day-to-day activities and scale up mobilization for IRS. In the 10 Northern Uganda districts where IRS had ceased, CHC worked with partners such as SDS (Strengthening Decentralization for Sustainability), ASSIST (Applying Science to Strengthen and Improve Systems) and the districts to intensify SBCC interventions on correct and consistent use of LLINs, increase in IPTp uptake and timely diagnosis and treatment of malaria. CDFU also involved civil society organizations (NGOs and CBOs) in disseminating information and mobilization for IRS.14 These collaborations were important in increasing the intensity and reach of BCC messages amongst the communities. However, we found that sometimes, mobilization for IRS started late – due to reasons beyond the project’s making - allowing only a short time of SBCC before the spraying. In addition, while the project endeavored to involve stakeholders from MOH, NMCP and districts at all stages of planning and designing the communication campaign, there were concerns from some districts that they were not adequately involved in the planning of the IRS campaign. For instance, some DHEs reported that they were not involved in the selection of the radio stations to be used for talk shows and radio spots. Key informants also reported inadequate IEC materials to reach the communities and households. As already discussed, this challenge was later on addressed when CHC supported the project to print more materials. In some cases, Spray Operators were accused of not giving sufficient information to households. Another key challenge to the BCC campaign was the resistance and counter information from organic farmers of the shea seed crop especially in the districts of Dokolo and Lira who were concerned that their crop produce would lose its export market if they used IRS. The project addressed this challenge by engaging with the farmers and their leaders to explain that spraying was not being done in the gardens and crop stores. Similar resistance and counter information also came from some religious sects such as Kanyiriri in districts of Budaka, Kibuku, Namutumba and Pallisa and Issah Messiah in Serere district who stopped their members from allowing their households to be sprayed. There were also misconceptions about the insecticide that it causes cancer or could lead to impotence among men and infertility of miscarriage among women. 14 Communication for Development (CDFU). October 2012- September 2013. Uganda Indoor Residual Spraying Project- Phase II. Work Plan Narrative for Year 1 23 Others had heard rumors that the campaign was a deliberate government effort to reduce the population size. The project addressed this pockets of resistance through door to door visits, and inter-personal communication with the resistant individuals and groups. Community dialogues were also conducted in resisting communities to address their fears and concerns and provide accurate information. 4.2.6 Influence of IEC/BCC on IRS uptake The different methods and channels used, as well as the active involvement of district, sub-county and community structures in information dissemination has been noted to have contributed to the high acceptability and coverage rates of the spraying exercise. Community members who participated in the FGDs confirmed how they responded to the information received about IRS. IEC/BCC campaigns helped to overcome resistance to IRS that was initially experienced, arising from myths and misconceptions, and concerns about organic crop produce as already discussed above. The adoption of innovative methods of communication and social mobilization such as IPC and community dialogues and the targeting of resistant groups is noted to have helped to increase IRS coverage from 67.2% in the round of December 2014- February 2015 to 94.4% in the spray round of June-July 2015 in the new project districts Group A that include Dokolo and Lira where resistance to IRS was most prevalent. 4.2.7 Influence of IEC/BCC on desired malaria control behaviors From the household survey data collected during the current evaluation, most household respondents in the sample districts had heard about the messages on malaria prevention such as using IRS to kill mosquitoes (93%), need to sleep under LLIN by children and pregnant women (96%), need to continue sleeping under LLIN even after spraying (96%) and the need for the pregnant women to take anti-malarial medicine to prevent malaria in pregnancy (88%). There were no significant differences between the new and old districts. These high percentages indicate that the IEC/BCC campaigns had a good reach of the target groups. The most common sources of malaria prevention messages received by households were VHT/Health workers (45%) and Radio (40%). The VHTs here include mostly those who were working as Spray Operators and therefore had a responsibility of providing information to households about IRS and other malaria control practices just before, during and after spraying. Majority of the household respondents (97%) who had received messages on malaria prevention reported that the message received had influenced their perception and actions on dealing with malaria prevention and treatment. The BCC messages were reported to have influenced respondents to sleep under LLINs (80%), to seek early treatment for malaria (56%), to ensure children sleep under LLINs (52%), and to ensure pregnant women sleep under LLINs (31%). However, these self-reports of behavior change must also be interpreted and validated using data from other sources. Below we briefly discuss other findings with respect to each of the behaviors. Continued use of LLINs Majority (96%) of the respondents in the household survey had heard the message that they should continue using LLINs after spraying. Thus, most community members were aware of the 24 need to continue using LLINs after spraying. This was also confirmed by data from some of the FGDs: The best way to control malaria is using a mosquito net … Spraying has to go hand in hand with the mosquito nets because mosquitoes came back after some time. (FGD with men, Paicho, Gulu district) There was also qualitative evidence of change in malaria prevention behaviors as a result of the BCC campaign: … people are using mosquito nets more than before, almost all people in my place and nearby home all sleep under the treated nets to prevent malaria. …Am very observant on the signs of malaria, and I encourage all my family members to finish their treatment and continue sleeping under the nets. (FGD, Pregnant Women, Pece sub-county, Gulu district) …Yes, it did change [behavior], because people still sleep under mosquito nets and close their windows before dark to prevent the mosquitoes from entering the house. …We still clear bushes around our homes and drain all the water logged areas. This is all done even after the spraying exercise was carried out. (FGD, Women, Kadunguru sub-county, Serere district) The above voices tend to suggest that people were continuing to use LLINs even after IRS. This could partly be attributed to the information they received about the need to continue practicing other malaria prevention behaviors even after their houses were sprayed. However, it was also noted some community members continued to use LLINs after spraying, not necessarily because of the information they received but because they still saw mosquitoes in their houses as one respondent reported: For me in the first phase of IRS I did not sleep under the mosquito net because all the mosquitos were killed by the insecticide but after spraying in the second phase, I had to sleep under LLIN because the mosquitos from the house did not die at all. (FGD expectant mothers, Amolatar T/Council, Amolatar district) While this may not have been widespread, but it nevertheless must be recognized that visibility of mosquitoes and therefore perceived malaria threat is usually one of the drivers of net use15 . Qualitative data also suggest that after spraying their homes some people stopped sleeping under the mosquito nets because they did not see the mosquitoes or because they thought that they were now safe from mosquitoes. Indeed, different key informants and local leaders who participated in FGDs expressed worries that the success of IRS had generated some kind of complacency within the community with regard to the use of mosquito nets. 15 Strachan E.C., Nuwa A., Muhangi D., Okui A.P., Helsinki M., Tibenderana J.K. (2015): What drives the consistent use of long-lasting insecticidal nets over time? A multi-method qualitative study in mid-western Uganda. Malaria Journal 15 (1). 25 Not all people use mosquito nets after spraying their houses, once they don’t see mosquitoes or they reduce, nets are put aside for some time, till mosquitoes start coming back again. (FGD, Local leaders, Pabo, Amuru district) Actually IRS has de-motivated the use of ITNs because people feel there is no use as there is no longer malaria cases. Secondly, because mosquitoes are not there so the people have become complacent. But somehow that is the role of our communication department to say that much as malaria has gone down we need to continue using nets. (KII, Tororo district) But some community members also reported that they did not use nets because they did not have them. Thus some people complained lack of the mosquito nets even when they were aware of their importance in controlling malaria post IRS. The reported lack of nets must be considered in light of the fact that the last mass distribution of nets took place in 2012. But in addition, government has continued to distribute nets through antenatal clinics in health facilities. This was confirmed through interviews with health facility in￾charges and other key informants. Nevertheless, it’s true that some community members do not have nets as the data from the Malaria Indicator Survey shows an average number of less than three ITNs owned per household against the average household size of 4.9 people per household in the IRS districts regions Overall, the above discussion indicates mixed results about the effect of IEC/BCC on the continued use of LLINs for malaria prevention after rounds of spraying. Prevention of Malaria in Pregnancy BCC messages encouraged pregnant women to go for antenatal visits at least four times during the pregnancy, and to swallow the anti-malarials given to them during these visits in order to prevent malaria during pregnancy. Using DHIS2 data and taking the number of pregnant women coming for ANC1 as the denominator, we were able to compute the proportion of women that took the first dose of IPT in the old IRS project districts, the new IRS project districts, and the non-IRS districts from the same region as the new project districts. Figure 5 below shows this comparison. The results in Figure 5 show that the trend of IPTp1 uptake has been more or less the same for all the three types of districts between January 2013 and December 2015. While the uptake was slightly higher in the new IRS project districts, this trend is similar both before and after the BCC campaign focusing on malaria prevention through IPT and other methods was introduced around December 2014. This seems to suggest that the BCC campaign did not have a marked effect on the uptake of IPTp1 in the project districts during the short time between its commencement and this evaluation. For IPTp2 (see chart in Annex IVA) the proportion of women who used this service was slightly higher among both new and old IRS districts than in none-IRS districts. However, this trend remained more or less the same before and after the BCC campaign focusing on malaria 26 prevention through IPT and other methods was introduced around December 2014, also implying that the BCC campaign had not had any effect on IPTp2 uptake by the time of this evaluation. Figure 5: Proportion of pregnant women that received the first dose of IPTp in IRS II project districts and non-project districts Source: MoH DHIS2 data The above results may be attributed to the short time between commencement of SBCC focusing on other malaria control behaviors and the time of this evaluation. There were a few contextual issues around IPTp that need mention here. Some discussions with key informants at health facilities revealed that Fansidar is given to expectant mothers during ANC visits who are accompanied by their husbands. While this has been a policy by government to encourage male involvement in maternal health care, it is also true that not all expectant mothers go with their male partners, which essentially means that they may miss out on this preventive treatment for malaria. This situation was also confirmed by women who participated in FGDs in some districts. From the above, it can be noted that malaria prevention in pregnancy could be affected by other related policy practices such as requirements to go with a male partner. Another concern raised by the community members was that often times health facilities experience drug stock outs which affects not just adherence but also access to essential drugs such as Fansidar. Despite the challenges of drug stock out and the requirements to go with a male partner, this evaluation found that Fansidar is considerably available to expectant mothers as a preventive measure against malaria in pregnancy. Early treatment seeking for malaria Data from FGDs and key informant interviews indicate that people seek prompt treatment for malaria as some of the quotes below illustrate. 0% 20% 40% 60% 80% 100% 120% Jan to Mar 2012 Apr to Jun 2012 Jul to Sep 2012 Oct to Dec 2012 Jan to Mar 2013 Apr to Jun 2013 Jul to Sep 2013 Oct to Dec 2013 Jan to Mar 2014 Apr to Jun 2014 Jul to Sep 2014 Oct to Dec 2014 Jan to Mar 2015 Apr to Jun 2015 Jul to Sep 2015 Oct to Dec 2015 Jan to Mar 2016 New IRS districts Old IRS districts None IRS districts 27 … People of these days don’t play with malaria, they know the rightful dose and when they are not given the right dose, they make noise why they have not been given enough. People are aware of the dangers of malaria and want to prevent it. …Am very observant on the signs of malaria, … (FGD, Pregnant Women, Pece sub-county, Gulu district) People now visit the health facilities for testing whenever they are sick to confirm whether they have malaria or not, mostly the women who bring the children or themselves, the men don’t want follow lines in health centers; (KII, Mulanda sub-county, Tororo district) We have TTT which means Test Treat and Track. Under this, you don’t get malaria drugs if you have not been tested and you must have a positive malaria test called the RDT. Then if you have been treated, we track you to see where is your home, how is it, is it bushy, why are you getting malaria, do you have a net, are you using it etc.? That is what we call tracking. (KII, Serere district) To promote the test, treat and track approach, some key informants reported that they had also trained VHTs in using rapid diagnostic tests (RDTs) and dispensing Coartem. In some districts, they reported that they had even trained school teachers in the same practices. However, this evaluation could not clearly ascertain that proactive behavior for community members to seek early treatment for malaria is directly attributable to the role of BCC campaign which did equally emphasize such practice. In a few other cases, prompt care seeking was reportedly constrained by occasional shortages of antimalarial drugs and RDTs. 4.2.8 Conclusions and lessons Based on the above findings, the evaluation team concludes that: • SBCC focusing on malaria prevention practices started only recently in late 2014 and early 2015 and it is too early to clearly see its impact. • Observed malaria control behaviors may not directly be attributed to the implemented BCC activities due the fact that there have been other on-going health education programs such as those by government through health facilities. • The project has strengthened the SBCC component through collaboration with CHC, deepened the reach of messages on dual malaria prevention, and is thus likely to translate in improved practices 4.2.9 Recommendations  There is need for continuous sensitization of communities even after IRS phase-out. Project and district IRS sustainability plans should include continuous sensitization of communities even after IRS phase-out to promote other malaria prevention practices.  Given the importance of other malaria prevention measures (LLINs, IPTp, early care seeking) in post-IRS periods, the interventions to promote these behaviors should go 28 beyond IEC/BCC. The MoH/NMCP and the project should work with other on-going programs to support communities to access enough nets, and support other related tasks that enable continued net use such as net hanging and net repair.  PMI should in the near future commission a comprehensive evaluation of the BCC activities for IRS and malaria control 4.3 How have structures and partnerships developed under NMCP been strengthened to continue implementing IRS interventions? 4.3.1 Introduction The IRS project Phase II was expected to develop and strengthen structures and partnerships under NMCP involving public and private organizations at the national, district and community levels. Before discussing how the structures and partnerships were strengthened, it is important to first identify what these structures and partnerships were. 4.3.2 Structures and partnerships in place for IRS work The IRS project phase II worked with a multiplicity of structures both public and private, at various levels. These structures are summarized in Figure 6 below. Figure 6: Structures for implementing IRS in Uganda At national level, the project did not create any new structures but instead worked with pre￾existing structures. These included the MoH Technical Working Groups on Integrated Vector National level (public) TWGs under MoH, MoH, MoH-NMCP, MoH-VCD, NEMA, NDA, other relevant ministries District, sub-county & community levels District and S/county IRS Task Forces; IRS Committees; District IRS teams; VHTs; LCs; religious, political, & cultural leaders PSOs/NGOs Private pest controllers under UPCA; Large companies (tea estates, cement factories, Uganda Prisons); local NGOs & CBOs IRS Project II 29 Control, Malaria Case Management, Malaria in pregnancy, IEC/BCC, and M&E. Individual agencies at the national level that were involved in IRS included the MoH NMCP, the MoH VCD, and other government regulatory agencies namely, NEMA and NDA. Private sector organizations in the context of IRS included small-sized companies under the umbrella of the Uganda Pest Controllers Association (UPCA), as well as the large companies such as tea and sugar estates. The companies under UPCA were from both Kampala and the districts, while the large companies were mainly from outside of Kampala. The NGOs included a few local and national NGOs operating in the project districts. The structures established by the project to implement IRS at district and sub-county levels included IRS task forces at the district and sub-county levels, consisting of politicians, civil servants as well as IRS technical staff. The main purpose of the task forces was to mobilize communities for IRS and to oversee IRS implementation. The project also constituted district IRS teams consisting of technical staff from the relevant departments for health, vector control, supplies, environment, and community development. The district IRS teams were responsible for working with the project team to plan and implement IRS services. The structures at community level included the VHTs most of whom were also recruited and trained as spray operators, IRS parish store mobilizers, store keepers, and the local leaders including LCs, cultural and religious leaders. Partnerships for IRS The IRS project statement of work states that that sustainability of IRS would depend on, among others, “creation of strong partnerships with the GoU, the district local governments, other donors, private sector and communities as well as other health programs”. The project’s statement of work does not define what was meant by a ‘partnership’ but states that the Contractor shall … “Establish official guidance on how to engage in the national IRS partnership, …, which invites participation and provides clear guidance on how partners are to proceed as well as plan, and carry out IRS in a manner that is consistent with WHO guidelines.” The evaluation team did not find such guidance, and in its absence, have referred to other literature regarding what constitutes a partnership. In most literature, partnership is defined as a process involving two or more actors who pursue some shared goals. The key features of a partnership are identified to include pooling of resources and skills, sharing of responsibilities or joint action, trust, and expectation of mutual benefits (Bentley 2004; Jamali 2004; Wildridge et al, 2004; USAID 2016). The rationale for partnerships arises from the need to achieve goals that organizations would not achieve if they worked individually. The evaluators were therefore interested to know whether such a partnership has been created and is functional. The findings indicate that the project has worked with the multiple stakeholders at national, district and community levels from both the public and the private sectors, in what may be termed as a broad partnership. Interviews with stakeholders in MoH, PMI and the project also indicated that the partnership refers to the collaboration between the project and the different actors as shown in Figure 6. 30 4.3.3 Project quantitative achievements in strengthening IRS capacity of national and district level structures and partnerships According to the project revised statement of work (SOW) (2015), the IRS phase 2 project is supposed to strengthen the technical skills and capacity of the MoH/NMCP (at both national and district level), local governments and other relevant government bodies, and the private sector specifically large private companies with existing or planned IRS programs to conduct IRS operations and associated monitoring activities. This is to be done through provision of short￾and long-term technical assistance, basic and on the job field training and involvement in any or all aspects of the IRS campaigns. All the three annual project progress reports (2012-13, 2013-14, 2014-15) show tremendous levels of capacity building performance with respect to the PMP indicators by 2015 is as shown in Figure 7 below. Figure 7: IRS Project’s performance on PMP indicators by Dec. 2015 Source: Uganda Indoor Residual Spraying (IRS) Phase II Project, Annual Reports, (June 26, 2012 through September 30, 2013, October 2013 through September 30, 2014 and October 1, 2014 – September 30, 2015), Abt Associates Inc. From the above data, it can be noted that the project achieved or superseded 100% in most of the PMP indicators for capacity building. More elaboration of these achievements in provided in the sub-sections below. 0% 50% 100% 150% 200% 250% 300% 100% 100% 276% 217% 100% 67% Performance PMP Indicator IRS training manuals reviewed and approved by NMCP/MOH Dashboard charts populated with district data National and district staff receiving in-service training People trained in IRS from the private companies/ NGOs/ CBOs Large private companies trained on IRS to expand their ongoing IRS activities Capacity of three insectaries built to conduct entomological monitoring 31 4.3.4 Strengthening the capacity of national level structures and partnerships The structures at national level that were involved in IRS consisted of the Technical Working Groups under the Ministry of Health, specifically those for Integrated Vector Control, Malaria in Pregnancy, Malaria Case Management, IEC/BCC and M&E. In fact, the project’s Entomologist Chaired the TWG on Integrated Vector Control Management. The project’s participation in the TWGs provided an opportunity for the project team to share updates and results from the IRS activities. Apart from the IRS project staff attending and sometimes making presentations in the TWG meetings, this evaluation did not find evidence that the project strengthened the capacity of the TWGs to oversee IRS work. With regard to strengthening of the partnership for implementing IRS, this evaluation found that the partnership for IRS remains weak. While the ‘common goal’ of the partners is fairly clear, that of continuing to implement IRS in order to reduce malaria, the pooling of resources has been so far limited. The GoU has not contributed nor shown commitment to contribute resources to continue IRS. According to discussions with PMI, the expectation was that the GoU would increasingly take on IRS for instance by financing/spraying an increasing number of districts over the years, as PMI funding progressively reduces. This has however not happened. There is concern among national level stakeholders that IRS is very expensive and this seems to explain the low government commitment to put in resources. The IRS activities have remained financially dependent on donor funding, putting into question the capacity and interest of the GoU to continue IRS when and where needed. The PSOs and NGOs who were trained by the project also seem to have each gone its way, without clear mechanisms of what responsibilities they expect of each other and from the MOH as ‘partners’. On the other hand, the project has collaborated with other malaria focused organizations such as Malaria Consortium, as well as communication organizations such as CDFU and CHC, and so in this sense has contributed to the building of a partnership for malaria control, beyond just IRS, but through integration of IRS with other malaria prevention and control measures including continued use of LLINs, early care seeking for suspected malaria and prevention of malaria in pregnancy. The project has also contributed to the strengthening of the capacities of the individual government agencies such as the NMCP and the VCD of the MOH. The project annual reports and interviews with MoH staff indicate that the project has strengthened the capacity of MoH/NMCP mainly through mentorship and skills training of MoH/NMCP staff in areas of entomological and environmental monitoring, IRS planning and training of lower levels; infrastructural improvements, and, though on a limited scale; some support in policy formulation. MoH/NMCP and VCD staff have alongside district staff attended trainings organized and facilitated by the project in areas such as entomological monitoring and environmental compliance. The project’s second year annual report for instance shows that that in July 2014, the project conducted an in-service training for 27 VCOs from 13 district local governments (DLGs) who were affiliated the MoH/VCD and the NMCP with the purpose of improving knowledge and skills of the VCOs. As part of follow-up of the trainings, the project continued to utilize the VCOs in entomological monitoring activities such as wall bioassay studies and pre- and post-IRS PSCs. It 32 is also reported that in every spray round, the project called upon and assigned at least 10 VCOs to conduct these entomological studies in project and non-project districts. The project collaborated with CDC, MoH/NMCP, VCD and the Uganda Virus Research Institute (UVRI) to conduct specialized training courses for district vector control officers on the use of the WHO tube and bottle bioassay for mosquito insecticide resistance testing and for monitoring transmission rates in IRS targeted districts. This training has enhanced VCD capacity in entomology to conduct IRS-related monitoring and evaluation including vector bionomics and quality assurance of IRS treatment. The project also supported the MoH to formulate an Integrated Vector Control Strategy, though this is not yet completed. As part of capacity building, the project had, by December 2015 supported two of the three targeted insectaries. The project rehabilitated the Gulu University insectary and recruited and trained two insectary technicians to manage the insectary. Additionally, the project collaborated with CDC, PMI, Tororo district authorities to convert a former Tororo hospital theatre into an insectary for rearing susceptible An. gambiae s.s. for use in cone bioassays. The evaluation team however noted that there has been limited capacity building for the MoH in terms of integrating IRS data management systems. The dashboards used by the project to track progress in the capacity building of district personnel have not been shared with the MoH/NMCP, and IRS data is not integrated into the national data systems. While IRS-related information has been shared in TWG meetings, there seems to be a gap in terms of regular and a more focused sharing of reports and results between the project and NMCP. The evaluation team further notes that not all the capacity building activities listed in the project’s statement of work have been implemented. Specifically, capacity building thus far has not adequately covered aspects of logistics management, forecasting and procurement of the insecticide, and mapping and stratification of areas for spraying. The evaluation team found that the project’s efforts to build the skills of MoH staff have not been guided by a system analysis of the capacity needs and gaps, nor an elaborate capacity building strategy. Interviews with stakeholders and existing documentation show that a capacity needs assessment of the NMCP and a capacity building plan were only formulated recently, in 2015. These documents are yet to be used in the implementation of capacity building for NMCP and VCD. On the other hand, the NMCP had only skeleton staff on ground and so the efforts of the project to build capacity were probably not optimally utilized. On a positive note, as part of these efforts to address the capacity needs of the NMCP, there are efforts to raise the position of the NMCP from a program to a Division within the MoH, a move that would give it more visibility, as well as financial and decision-making authority. The evaluation team established that there exists a good working relationship between the NMCP/MoH and the IRS phase II project which has facilitated the capacity building activities outlined above, as well as joint undertakings such as celebrating the National Malaria Day. 33 Interviews with the Director of NMCP confirmed these good collaborative efforts which have so far yielded all the good results mentioned above. 4.3.5 Strengthening the capacity of PSO/NGOs in IRS The project conducted a technical skills training in IRS implementation and management for private sector pest control organizations brought together under the UPCA during year one, and a business skills training for the same target group during year two. In addition, CDFU, the IEC/BCC sub-contractor selected and trained ten representatives from six Community Based Organizations (CBOs) operating within the project area that agreed to partner with the project to improve social mobilization. During year three, the project conducted technical IRS skills training for large private sector organizations. As may be observed from Figure 7, by year December 2015, the project had trained all (100%) the people it had targeted from PSOs and NGOs. Interviews with the leaders of the UPCA indicated that the training conduced in 2014 would have been more effective if the IRS project had carried out a UPCA training needs assessment before conducting the training. The training would also have been more relevant if the project had involved more PSOs from the local levels and if during the trainees had been involved in the actual spray operations. The non-involvement of the local NGOs/PSOs could have been a result of the fact that no such NGOs have been involved in IRS activities before. This is reflected in the fears expressed by one key informant from Serere district in the following quotation: ...First of all, I have not seen any of those NGOs in Serere participating in IRS. I think that even their involvement would make things worse, because if people see that an NGO is involved, they will think of money coming from an NGO. They will think that the NGOs have a lot of money; so it is better that the program runs as part of government. And I think that is why the NGOs are fearing to implement this kind of a project… (KII, Serere district) The project also conducted technical IRS skills training for large private sector organizations during year II and III. The project targeted companies who provide IRS services or who are interested in carrying out IRS for their employees and surrounding communities. In collaboration with UPCA, the project trained 45 participants from five tea estates which included: Ankole, Mwenge, Kiko, Muzizi and Bugambe and staff from Hoima district in implementing safe and quality IRS in the tea estate labor quarters and surrounding villages. The training covered entomological monitoring, environmental compliance, and spray techniques. The project’s efforts in undertaking these activities were aimed at building capacity for undertaking IRS amongst such organizations, to ensure IRS is conducted to meet quality standards. The project also held discussions with large private companies namely: Kakira Sugar, Mwenge, Kiko, Bugambe, and Muzizi tea estates, Pilgrim Africa, Hima Cement factory, Tullow Oil, and CNOOC Oil to discuss IRS. As can be noted, the project’s engagement with private organizations included organizations even outside of the project’s target districts. The leaders of UPCA affirmed that the organization and its members were willing to continue implementing IRS activities after the project closes down because of their members’ strengthened 34 capacity resulting from the project support and because pest control is now part and parcel of UPCA’s mandate and vision. However, the evaluation team notes that only a small number of staff from each of the private sector organizations were trained raising questions about their capacity to undertake IRS on a large scale. Similarly, for the large companies, whereas their staff were equipped with skills to conduct quality IRS, they continue to lack the state of the art equipment such as pressurized pumps to conduct IRS in the recommended procedure. The insecticide used by the GoU and the IRS project is also not readily available on the local market. The large companies are therefore not yet performing IRS to the expected standards. While the provision of equipment was not a mandate of the project, but in the context of sustainability, this remains a challenge. In a bid to overcome this challenge, the project held discussions with a Ugandan company, Quality Chemicals Ltd., regarding the possibility of making World Health Organization-approved insecticides available on the Ugandan market where they can be accessed by accredited private organizations that offer IRS. Another limitation was that the neither the project nor the MoH has been called upon to practically supervise or support on-job training of the spray teams from these companies, and therefore guarantee the quality of IRS services that they are providing. This evaluation also found that there is no national legal provision for NGOs/PSOs involvement in insecticide control. As a result, there is fear among stakeholders that there are many quack spray operators in the country who cannot be easily regulated in absence of a comprehensive legal framework. 4.3.6 Strengthening District and lower level structures According to the project reports and interviews conducted with the project staff, the project strengthens the IRS capacity of districts in the six areas of: entomological competence, environmental compliance, IEC/BCC, logistics, M&E and spray operations which areas respectively fit in the already established district structures of District Vector Control Office (DVCO), District Environmental Office (DEO), District Health Education Office (DHEO), District Supplies Office (DSO), Bio￾statistics Office (BSO) and the IRS Focal Person within the District Health Office (DHO). The project implements a three-level cascade model of training whereby each group trained is responsible for training the subsequent lower level cadres: the project conducts training of trainers for national and district technical persons, who then train sub-county supervisors and parish store keepers who in turn train spray operators, parish mobilizers and wash persons under the supervision of district technical officers and project field coordinators. The project has used this cascading training to build local capacity in spray techniques, data capture, community mobilization, and environmental compliance. Apart from training other forms of capacity building have included on-job mentorship, support supervision and demonstrations. Measuring and monitoring district IRS capacity improvements: The capacity dashboard tool To assess the capacity transferred to the district personnel who are counterparts in IRS implementation, the project developed a capacity building dashboard tool. According to project documents, the tool serves as a ‘temperature gauge’ for ongoing capacity building transfer to relevant district staff in all the project districts. The project uses the capacity dashboard to measure the capacity being transferred to the district authorities to effectively and efficiently 35 manage IRS operations, specifically in the six IRS capacity areas of: entomological competence, environmental compliance, IEC/BCC, logistics, M&E and spray operations. The tool uses indicators for each of the project area which are scored accordingly on a scale of 0 to100 (see Annex VA for indicators). The project reports state that the scores are then filtered through objective analysis of district involvement in each indicator, to produce a visual dashboard that gives a ‘bird’s eye’ view of capacity building progress in each district (see more on this in Annex VB). The evaluation team has used the dashboard framework to review the implementation of the project’s capacity building component. Thus, the review considered an inventory of all the project’s capacity building activities implemented during the last three years. The dashboard tool has guided the consultants’ assessment of the existing IRS capacity of the project districts. IRS capacity performance in the districts Figure 8 shows the capacity improvements in the new project districts between June and December 2015. Figure 8: IRS capacity improvements in the new project districts between June 2015 and Dec 2015 Results from six16 of the fourteen new districts indicate that between the spray rounds of June 2015 and December 2015, except for M&E which registered a negative change, there was registered capacity improvement in each of the other project areas as shown in Figure 8. Entomological competence registered the highest score of 57. According the dashboard tool manual even entomological competence which registered the highest score of 57 still needs attention. In all the other project areas of environmental compliance, BCC/IEC, logistics, M&E and spray operations (SO), the districts showed slow progress and certainly they need more attention. Below we provide more discussion of the improvements in each of the project technical areas. The districts’ relative performance indicators in various technical areas were confirmed during this evaluation, by responses from the district staff interviewed. For instance, several staff were able to describe in detail the processes that they go through to conduct technical procedures related to their IRS work such as monitoring the susceptibility and resistance of mosquitoes to 16 Budaka, Bugiri, Kibuku, Namutumba, Paliisa and Serere -10 0 10 20 30 40 50 60 E/cpt E/cpl BCC Logs. M&E SO June 2015 Score 17 3 4 12 49 19 Dec. 2015 score 57 12 30 46 42 24 Capacity change 41 10 25 35 -7 5 36 the insecticide. Others could also describe what they are able to do with the knowledge and skills they acquired from the trainings: ...The trainings have actually helped me because if I go to any sub-county, I can be able to site a soak pit, I can give them the right dimensions and I can even explain to them what progressive triple rinsing means. Actually, I can even identify that this PPE is good and that is a bad one. So the trainings have really helped me. As far as environmental compliance is concerned, we have no problem. What the project has given us, we have also taught to the people at the sub-county... (KII, Serere district) Most of the sub-county staff interviewed also expressed improved capacity gained out of project training as evidenced below: …Yes. I was trained in store record keeping, how the chemical was supposed to be mixed and the precautions to take and the benefits of IRS… The training targeted district and sub-county level staff. Then we came and trained the spray operators. We also briefed the political leaders on importance of IRS and their roles and responsibilities in the ‘IRS… ...In this community, pregnant women feared having their houses sprayed. I talked to the women that women and children are the most vulnerable (to malaria) and (that) their households should participate in the spraying. Some of them then came to me to have had their houses sprayed. I also addressed the misconception that the land would be contaminated by the chemical... (KII, Agweng sub-county, Lira district.) …The capacity built was very effective since we had people who could do the spraying. In addition, we have an effective monitoring and supervision team of the project implementation from the district… (KII Kamonkoli Sub-county, Budaka district) Similar views were expressed across all the districts that were included in this evaluation. Overall therefore, the project appears to have registered some improvements the capacities of districts in the respective technical areas although there is a lot more work to be done. The districts’ negative change in M&E performance as seen in Figure 8 could have been due the non- involvement of some of the technical staff such as the District Planners. Our interviews with project staff indicate that the project worked more closely with the District Bio-Statisticians and Data Entry Clerks based in the District Health office, but not as much with the District Planners. In addition, the project’s engagement with the Bio-Statisticians and Data Entry Clerks was more focused on data entry and transmission, rather than the whole M&E cycle. The districts’ low improvement in performance (5% as shown in Figure 8) in conducting for spray operations is attributed to the unavailability of the district staff for these activities. According to the Project capacity building manager, the unavailability of district staff was caused by in other competing assignments. Data from the old project districts also shows significant improvements in capacity by year three (October 2014), from the status of April 2013 (see Annex VC). 37 Overall, the dashboard results presented above give a picture of the district capacity improvements as a result of the IRS project capacity building activities. The evaluation team however noted that these results are never given due prominence in project progress reports. As a result, the project PMP output indicators that are normally prominently reported on don’t give a true picture of the districts’ IRS existing capacity and growth. The evaluation has found out that the dashboard tool is a fast and easy way to understand the current landscape of capacity built in districts and across the project. The dashboard can be used both as a tool for monitoring and to measure the capacity growth across each target district and program area. The MoH/NMCP and other stakeholders may find the dashboard a very useful and innovative approach to track project progress in capacity building. However, according to the project Capacity Building Manager (CBM), by the time of the evaluation, very few people at the districts or the national level had the necessary skills required to use the tool effectively. The main reason given was that the tool’s software system is very expensive and may be out of reach for the districts given their meager financial resources. 4.3.7 Sustainability of IRS activities and outcomes The IRS project was expected to build the capacity of the MoH/NMCP, districts and PSOs/NGOs for IRS sustainability. In the context of IRS, sustainability can be conceived in the following ways: i) Maintaining the gains achieved from IRS, i.e. after IRS has helped to lower the malaria burden, ensuring that the levels of malaria are maintained at a low level through other malaria preventive measures such as continuous use of LLINs, early diagnosis and treatment of suspected malaria, and prevention of malaria in pregnancy. ii) Ability to continue implementing IRS to reach other parts of the country that may need it, as well as to address malaria epidemics if they occur. Ability to maintain low levels of malaria burden following IRS The project has ably integrated other malaria prevention and control interventions, namely, continuous use of LLINs, early diagnosis and treatment of suspected malaria, and prevention of malaria in pregnancy. Through the SBCC component, the project has educated communities about dual protection through IRS and nets, as well as prompt care seeking and prevention of malaria in pregnancy. These interventions have also been built into the sustainability plans of districts which is commendable. The remaining question, however, is where the GoU and the districts are able to finance the supplies, the SBCC and other program inputs necessary to sustain high net coverage and utilization, as well as use of other malaria prevention measures amongst Ugandan communities. Ability to provide IRS when needed With regard to the second aspect of sustainability, i.e. ability to continue proving IRS services when needed, the evaluation team notes as follows: 38 This evaluation found that the project has actively engaged all key stakeholders at different levels in IRS planning, implementation, monitoring and supervision. At national level, the NMCP and VCD have gained extensive skills in planning, implementing and supporting IRS activities. They may however need more capacity strengthening in the forecasting, estimation, procurement and management of IRS supplies. At district level, all components of the project were integrated and consistent with the district service delivery structure and systems. For example, supportive supervision, and training of staff in IRS implementation were all integrated in the district health system using available DHT staff. The different training activities which were conducted enhanced the capacity of the DHTs, health workers and VHTs, most of who have been hired and trained as Spray Operators. All these cadres of staff are now critical agents of change who can continue to facilitate access to information and delivering quality services if regularly equipped. The behavior change communication activities helped to increase people’s knowledge about the usefulness and availability of life saving interventions that have a dramatic impact on malaria. Additionally, individuals have been empowered to act on the acquired knowledge to take positive actions – to improve their own health and that of their families. To the extent that these human resources and skills are available in the districts, we can conclude that the districts have a good potential to run IRS activities. However, the project suffered from the inherent health system constraints such competing demands on the time of district staff, understaffing and lack of qualified health workers in lower health facilities. The GoU and district’s capacity to provide IRS when needed is largely determined by availability of financial, technical and logistical resources. Yet the availability of such resources is still not guaranteed. At national level, the GoU/MoH has not committed financial resources to continue to provide IRS when needed. IRS is perceived as costly and unattainable without donor support. Yet no cost-effectiveness studies have been done to determine what would be saved if malaria was controlled to almost zero levels and how this saving compares with the cost of IRS. Other key informants also remarked that IRS remains less prioritized by GoU, which instead has put emphasis for the period 2015 – 2020 on infrastructure such as roads, railways and hydro￾electricity. There is accordingly no evidence of government preparedness to take over the financing of IRS from the current donors. Whereas the districts have the human resources with skills, their readiness to sustain IRS activities, is also dependent on the availability of other resources, such as finances, insecticides and equipment as one of the key informants explains: “…The problem is that it [IRS] is a very expensive program … the equipment itself and then the insecticide that is supposed to be used. If it is left alone to us as a local government, we don’t even know where to buy it from. The procurement process can be hectic and most local governments are resource constrained. But since we have the VHTs who have been trained, and we hear that there is a plan to recruit community health workers who are VHTs and put them on the payroll, we hope and think that 39 there will be a way by using the local revenue to pay these people, buy the chemical and continue spraying every six months even if it is not the whole district. …And of course, continuing to sensitize because once the chemicals become available on the market and there are VHTs who have been trained, if you want your house to be sprayed and you manage to buy a sachet, then we can link you up with a VHT in your village to come and help you spray. …we also have the technical persons such as the vector control officers, data managers, and store keepers. Then of course on the expertise of use of information to cause change, we also have expertise on that ...” (KII, Serere district) Similar views were expressed by key informants from other districts as well. 4.3.8 Conclusions and lessons National structures and capacity building  MoH/NMCP staff have gained skills and competences in IRS through participation in IRS planning activities, trainings and support supervision.  However, the MoH/NMCP is yet to receive the requisite capacity building support from the project that was indicated in the statement of work (SOW) and project’s PMP.  Information sharing and integration between the project and the MoH/NMCP is still inadequate. District and community structures  A lot of efforts have been made towards strengthening the IRS capacity of district and sub-county task forces, and individual staff; the project had either achieved or superseded targets of all its PMP indicators by the end of the third year.  This IRS capacity growth of districts and lower level structures as observed above is not given due prominence in the project annual reports.  The project PMP output indicators prominently reported on in the annual project reports don’t give a true picture of the districts’ IRS existing capacity and growth.  While the dashboard tool is a useful source of information for managers working within Uganda IRS Project Phase II, very few people at the district or the national level had by end of 2015 understood the importance of the tool in IRS capacity building or the necessary skills required to use the tool effectively.  While the districts’ capacity has been built in terms of skills, these cannot be put to use to provide IRS when needed unless they are accompanied by other logistical and financial resources. Developed human capacities in the 10 previous IRS districts have remained largely under-utilized due to lack of accompanying resources. PSOs/NGO  The project has made good efforts to strengthen the IRS capacity of PSOs/NGOs. However, there is need for more support during actual spraying by these entities.  The project has strengthened the institutional capacity of the UPCA, hence laying a good foundation for building further the capacity of the members, for self-regulation among members of this association.  While the project has fulfilled its mandate with respect to building the competences of the private sector to conduct quality and safe IRS, many challenges remain in the context 40 of sustainability, in terms of having a critical mass of competent IRS service providers with access to recommended supplies and equipment.  There is no adequate national policy for private provision of IRS, which limits the full possibility of involving PSOs/NGOs in IRS and assuring a good quality service from them. Sustainability  The project is doing well integrating other malaria prevention practices (LLINs, IPTp, early diagnosis and treatment)  The project has set a stage for continued implementation of other malaria prevention and control activities including LLINs, prompt care seeking and prevention of malaria in pregnancy, but the capacity of GoU and the districts to finance such measures are still weak.  An exit strategy and building sustainability plans with districts is essential to the sustainability of the gains from IRS. 4.3.9 Recommendations National structures and capacity  In the remaining period the project should conduct all the capacity building activities for MoH/NMCP as stated in PMP updated on August 18, 2015.  The project in consultation with PMI and NMCP should support the implementation of the capacity building plan for NMCP developed in 2015.  The project should provide dashboard tool skills to MoH/NMCP or provide this support as part of the short term technical assistance (STTA) and share the dashboard manual. The project should work with MoH/NMCP to share ideas on how best to utilize the dashboard tool and to integrate IRS data into the DHIS and HMIS. The project should also share monthly entomological data with MoH (NMCP and VC).  The MoH should advocate and lobby MoFPED, parliament and other decision makers to ensure allocation of funds to start off a government-funded IRS component that can be progressively increased over time.  As an advocacy tool, the MoH/NMCP should undertake a cost-effectiveness analysis of what would be saved if malaria cases were controlled.  As part of building sustainability, the project and MoH (NMCP, VCD) should develop continuous engagement with government institutions that have a large number of personnel such as the police, the prisons and the army to interest them in IRS, build their capacity in IRS, and support them to spray their institutional houses and neighboring communities.  GoU should allocate a progressively increasing amount of funds towards malaria prevention through LLINs, prompt care seeking for suspected malaria and prevention of malaria in pregnancy in order to ensure that the gains from IRS are maintained. District and community structures  The dashboard tool indicators for the six IRS project areas of: entomological competence, environmental compliance, IEC/BCC, M&E, logistics and spray operations are highly ‘Specific, Measurable, Achievable, Realistic and Time bound (SMART) outcome indicators for the IRS capacity building component. The project should use them as capacity building outcome 41 indicators so as to provide the true picture of the targeted districts’ and communities’ IRS capacity growth due to project interventions  To improve performance and making the districts appreciate the importance of the dashboard tool, the project capacity building team should work with the respective districts to agree and set performance target (scores) which the districts should work to achieve; and train district staff in the dashboard and provide them with copies of the dashboard manual.  To improve performance in data reporting, the project team should design a user friendly M&E tracker in Excel which can be updated every quarter/reporting period to track both the output (numbers) and outcome indicators (scores). USAID/ACE (2006-2008) project17 successfully used this approach and designed a tracker for PEPFAR indicators, general output indicators as well as outcome indictors.  The district health office should plan for continuous training of staff in IRS given the high staff turn-over especially in rural sites. NGO/PSO  GoU should come up with a clear policy on pest control by PSOs/NGOs in order to effectively regulate their work and provide an enabling environment for increasing their involvement in IRS.  The IRS project Phase II should involve the PSO/NGO staff into spray operations during capacity building activities.  There is need to enhance the PSO/NGOs’ capacity in IRS M&E and quality assurance  The project in collaboration with MoH/VCD should accredit the PSOs/NGOs who have successfully attended the requisite IRS training as a way of regulation and quality assurance. This may be done through issuing of quality assurance certificates to those who meet the requirements for accreditation.  The project and MOH (NMCP, VCD) should ensure continuous engagement with large private sector companies to provide on-site support supervision in IRS, and IRS monitoring. 4.4 What are stakeholders’ perceptions on the effectiveness of IRS? 4.4.1 Introduction This section of the report presents the findings about perceptions of the various stakeholders on the effectiveness of the program. The evaluation team defined effectiveness as the ability of a program to achieve its intended objectives, i.e., the ability of the IRS program to reduce malaria in households and communities. We have considered stakeholder perceptions about 17 USAID/ACE project successfully strengthened the capacity of targeted Ugandan institutions (Uganda AIDS Commission (UAC), the Inter-religious Council of Uganda (IRCU) and its network of faith-based organizations, the Ministry of Health Resource Center (MORC), Hospice Africa Uganda (HAU) and National NGOs implementing HIV/AIDS programs) for improved program outcomes and sustained ability to deliver results in regard to HIV/AIDS prevention, care, and treatment. 42 effectiveness to also include perceptions about the negative effects of the program, such as side effects and other unintended consequences. 4.4.2 Perceptions about the effectiveness of IRS Household members who participated in the survey were asked a series of questions, one of which was a statement that required them to agree or disagree whether IRS was effective. Figure 9 below shows the proportion of household respondents in the survey in both old and new districts who agreed that IRS was effective in killing malaria transmitting mosquitoes, and thereby reducing malaria. Figure 9: Household respondent’s perceptions about the effectiveness of IRS “Spraying against mosquitoes is effective, i.e. kills malaria transmitting mosquitoes” As the data in the chart above show, majority of the respondents in both new and old project districts agreed that IRS is effective in killing malaria-transmitting mosquitoes and thereby reducing malaria. The proportion of respondents agreeing with this statement were slightly more in the old districts (95%), compared to the new districts (89%). This could partly be because those from the old project districts have witnessed both the drastic reduction of malaria after spraying, and its resurgence after a long period with no spraying, and this experience has demonstrated to them the efficacy of IRS. Data from the qualitative FGDs with community members, local leaders, and VHTs, as well as that from key informant interviews with district officials across all the sample districts also shows concurrence that indoor residual spraying against mosquitoes has been effective in reducing malaria burden in their families and communities. This view was based on the stakeholder’s 89% 95% 92% 6% 1% 3% 5% 4% 5% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% New districts Old districts Total Agree Neither Disagree 43 comparisons of the malaria situation before and after the spray rounds. Some of these reports are contained in the quotes below: IRS has helped us a lot. It has reduced malaria. It was a big problem here before spraying but now we no longer suffer from malaria as it was before. (FGD Men, Agweng sub-county, Lira district) … cases of malaria now as compared to 4 years ago have reduced a lot. …I can see that malaria is low as compared to 4 years ago as I can also see from my family whereby most of my family members used to fall sick because of malaria but as I am talking now my children are not suffering from malaria the way they used to do 4 years ago. (FGD, Pregnant Women, Amolatar district) Initially we had very high rates of malaria and I think Tororo was either number two or three in the whole country. Our malaria prevalence was very high around 56% and in a short time as a result of IRS it reduced to 36% and currently it stands at 16%. This is quite remarkable”. (KII, Tororo district) Stakeholders in the old set of districts in the Acholi region which were phased out in 2014 also referred back to the time when IRS took place and commented on how effective it was, compared to today when there is no spraying: …There is a change because during the time they had just spayed against mosquitoes, children who used to suffer from malaria were few … during the past when they used to spray against malaria, it helped to reduce on malaria but the moment they stopped indoor spraying for long the malaria increased rapidly. …the last time they sprayed against mosquitoes, malaria had decreased and after spending long without spraying, mosquitoes that cause malaria have multiplied leading to the high rates of malaria we experience today. (FGD, VHTs, Omiya Anyima sub-county, Kitgum district) Before IRS, the number of malaria cases was high. When IRS came in, the cases reduced, like in a day we could get only 7-10 positives. After IRS stopped the positives increased to about 10-20 cases. .Around April 2015, there was an outbreak and we could get 40-70 cases a day. (KII, Biashara HC II, Apac district) From the above voices, it is evident that even in the previous set of districts in Northern Uganda, where there is currently a resurgence of malaria, stakeholders acknowledge that IRS had helped to reduce the problem of malaria. Several district staff and leaders cited statistics from their HMIS showing a down ward trend in number of cases of malaria recorded at health facilities, which they attributed to IRS. However, some of the health workers such as in Serere district reported that the few cases of malaria that they receive now are of a severe nature and were not sure what the explanation for this might be. One possible explanation might be that after the spraying, the few people who get 44 malaria do not seek early treatment thinking it is another illness, leading the malaria to advance to a severe stage. Stakeholders at national and district levels were unanimous that IRS has been effective in reducing the burden of malaria in the districts that have been sprayed. They cited district and national figures demonstrating how malaria cases had reduced following rounds of spraying. There was also recognition among them that the effectiveness of IRS in the long term can be watered down if no adequate preparation is done to sustain the benefits through for instance continued of use LLINs and other malaria prevention and control measures. 4.4.3 Benefits from indoor residual spraying The perceived effectiveness of IRS was also reported in terms of the benefits that households, communities, health centres and local governments had realized from the spraying against mosquitoes. Community members talked about reduced health expenditures and using the savings to meet other needs, as well as more productivity on their farms and businesses. Local leaders and district stakeholders reported that the reduction in malaria has enabled families to be more productive because the time the parents would be sick themselves or nursing their children suffering from malaria is now spent at work without interruptions. Similarly, children are able to keep in school without much disturbance from malaria; Expenditures on malaria treatment reduced. Before indoor residual spraying, malaria was rampant and therefore we spent money on treatment in these private health facilities. There is malaria but its low and expenditure is low. (FGD, men, Kameruka sub-county, Budaka district) The spraying was very useful in the sense that malaria infection has reduced; the number of malaria patients has reduced. It is not like in the past when supplies of drugs like Coartem would get finished within a week. People are not spending much on treatment; children’s rate of school attendance has also improved. (KII, Namasale sub-county, Amolatar district) Several study participants also based their perception about the effectiveness of IRS on the reduced frequency of visits to health facilities to seek treatment for malaria, as well the patient load at the health facilities. There were several reports from stakeholders across all sample districts that the number of patients at health facilities usually reduced following rounds of spraying, which indicated that the spraying had been effective in reducing malaria. Health workers interviewed at several health facilities also confirmed the reductions in cases of malaria after rounds of spraying. The examples quoted below demonstrate these views: House spraying with the insecticide has reduced on the cases of malaria in the homes. Even when you go to the health centers, there are fewer people with malaria unlike before. (FGD, pregnant women, Namasale sub-county, Amolatar district) Before houses were sprayed, we had many cases of malaria but when the IRS program came in, ... the cases of malaria dropped. We have realized a very big change as far as malaria cases are concerned, 45 those ones who come and are tested positive are very few as compared to before the spraying. For the two months now, I have only gotten two cases of complicated malaria in children aged 5-10 years. (KII, Kadungulu HCIII, Serere district) Results further reveals that the IRS program has not only been effective in eliminating the targeted female anopheles mosquito that spreads malaria parasites but also other insects and rodents that are often nuances to people in the homes, especially termites, ants, bats and cockroaches. However, some community and district level stakeholders especially in the Eastern districts of Tororo and Budaka thought that the second round of spraying in 2015 was not as effective as the first round. They based this on their observation that mosquitoes came back quickly after the second round of spraying compared to the first. Some informants suspected that the insecticide could have been over-diluted, or improperly stored, but these claims were not based on any concrete information. However, project monitoring of insecticide efficacy indicated that the insecticide was effective in the second round as it was in the first round. Results further indicate that because of reduced malaria cases at health facilities, there has been reduced pressure on anti-malarial stocks because few patients are currently presenting with malaria in the program areas. Consequently, most facilities where the program has been operational were reported to be stocked with anti-malaria drugs and other related diagnostic supplies like RDTs, unlike in the past when they would be stocked out. As a result of the low cases of malaria, we found that some IRS project districts were donating anti-malaria drugs to neighboring non-project districts. For example, Tororo district gave stocks of Coartem to Mbale district because it was not being used and would expire if not given out. In the past if you come to our health center, you would find Coartem out of stock most of the time but now malaria has reduced due to IRS and Coartem is always in stock because people no longer frequent the Health center. (FGD, men, Abala Village, Agweng sub-county, Lira district) … we have registered very low stock-outs of anti - malarials, in fact we have out some of the anti malarials to some districts like Mbale. We gave Mbale some consignment of Coaterm …. (KII, Tororo district). District staff and leaders reported that as a result of the above, IRS had also helped to reduce district expenditure on anti-malaria drugs. The above scenario however points to the need for coordination between districts and the National Medical Stores (NMS) to adjust the volume of their antimalarial supplies accordingly. 4.4.4 Perceptions about the negative effects of the spraying In some cases, stakeholder’s perceptions about the effectiveness of IRS were closely intertwined with perceptions about the side effects of IRS on humans, animals, and aspects of the environment such as soils and crops. The data show that more than 75% of the respondents in both new and old project districts believed that the insecticide used to spray mosquitoes was safe to humans, animals, soils and crops. Of course, the safety referred to here is only guaranteed if the insecticide is used as prescribed in its application guidelines, and if all safety precautions are undertaken. 46 More specifically, household respondents were asked if they thought the insecticide used can make men impotent and women infertile. These specific questions were asked in light of the rumors that were reported to had been circulating in most communities about these possible side effects of the insecticide. Figure 10 shows the results from the household survey regarding people’s perceptions about these alleged side effects. Figure 10: Household respondents’ perceptions about the effects of the insecticide on men’s sexual drive and women’s fertility As the above data show, almost 60% of respondents from both new and old project districts disagreed that the insecticide used to spray mosquitoes can make men impotent, while more than 60% in both sets of districts disagreed that the insecticide can make women barren. These results show a positive picture that shows that the myths about IRS are not held by the majority. Indeed, those who thought these myths to be true were only less than 10% in both sets of districts. Qualitative data from the FGDs with community members indicate that those who talked about the risk of impotence and infertility were simply reporting what they had heard, but they did not necessarily believe these claims. Indeed, findings indicate that the earlier fears among community members that the insecticide could make men impotent and women infertile were slowly dying out. Following several rounds of spraying, community members were now realizing that those fears were unfounded and the claims were not true, as some FGD participants testified: …We have not heard or seen anyone complaining that the insecticide has made him impotent and the children you see being carried right now were given birth to after the spraying exercise. That means our men are perfectly functioning. (FGD, women, Kadunguru sub-county, Serere district) 47 It should however be a point of concern that 30-40% of the respondents neither agreed nor disagreed with the claims about the insecticide causing impotence and barrenness. This indicates an information gap, i.e. such people do not know what the truth about the effect of IRS is. This means that the more work needs to be done among communities to provide accurate information. Survey participants were further asked to agree or disagree with the statement that “spraying against mosquitoes’ conflicts with my religious beliefs”. The results show that an overwhelming majority of respondents (99.9%) disagreed that IRS conflicted with their religious beliefs. Only less than 0.1% reported that IRS conflicted with their religious beliefs. These were found in Budaka and Kitgum. 4.4.5 Willingness to use IRS in future Perceptions about the effectiveness of spraying were also reflected in people’s willingness to spray their houses in future. Many study participants expressed willingness to have their houses sprayed in future. This demand for IRS was especially strong in the previous Northern districts where IRS ceased in 2014. It [the spraying] should even start immediately because poverty and sickness is squeezing us. At least when there used to be house spraying, we used not to spend a lot of money in the treatment of malaria but now days the cases of malaria are too high. (FGD, men, Omiya Anyima sub-county, Kitgum district) …We have no objection to spraying again. …I think all people would welcome IRS and it should be done soon and it should not be for only five years, it should be forever. …This program had helped people in this community. …when our homes were being sprayed, mosquitoes were few and as a result cases of malaria had gone down. (FGD, local leaders, Pabbo sub￾county, Gulu district) It is clear from the findings that the demand for IRS in the Northern districts is based on past experience with its effectiveness as well as the current upsurge of malaria infestation in the Northern districts. On the other hand, there were a few community members who said they would not welcome spraying of their houses in future. Such views mainly came from those who had not realized the expected benefits from IRS. They cited cases of where the program did not go as anticipated, such as mosquitoes not dying and infestation with bed bugs. I don’t wish my house to be sprayed again because two days after spraying there was outbreak of bedbugs. And also after spraying, even if you keep out for some minutes as instructed, when you enter the house you get dizzy, I don’t know what type of insecticide they used. And when I decided not to use the net on the day of spraying to see whether mosquitoes had died, I found that the mosquitoes were there just as before spraying. So I don’t know whether malaria has gone down due to spraying or use of nets, but bed bugs are too much. (FGD, community members, Western Division, Tororo Municipality) 48 Overall, such sentiments which were against future spraying were few. 4.4.6 Willingness to pay for Spraying We asked participants who expressed willingness to have their houses sprayed again if they would be willing to pay or contribute some money to have their houses sprayed. Their responses were varied, but across all districts most participants observed that it would be difficult to contribute any money due to low incomes and general poverty amongst most households. Others argued that since they are already used to a free service, getting them to start paying would be difficult. Others yet said they already pay taxes to government and it should find the money to continue providing IRS. … I think it will be hard for us to contribute since some people are really too old and basically they even have no reliable sources of money to make them be in position to support the IRS program. … I think it is really hard for people of Uganda today to contribute some money towards the program because today we are already burdened paying for everything like school fees and medical expenses among others. (FGD, pregnant women, Amolatar town council, Amolatar district) In other FGDs, opinions were so divided that no consensus could be reached as seen from the FGD with local leaders in Gulu District below: … If the government came up such a measure, we would welcome it even though a tax is introduced for that purpose we would pay. … We already pay taxes, the government should get from our taxes and for IRS. … It should be like UPE where people cost share with the government. … The government should allocate a certain portion of our health budget for IRS, but the proposal of paying for spraying will be met with resistance by the community. … No one would hesitate to pay for a health program but they should first be sensitized so they know why they are doing so. … Collection of money comes with accountability issues, they would need to ensure transparency. …Not many people would want to pay for spraying; we have a lot of bills and taxes to pay; besides our sources of income are so limited. (FGD, local leaders, Pabbo sub-county, Gulu district) Some participants elsewhere, especially women, however argued that if informed early they would make a contribution while others were more ambiguous saying that it depends on the amount that one has to be asked to contribute towards the program. Others were of the view that since they pay taxes, it’s the responsibility of government to provide such services. Some of the local leaders were even more skeptical about such an idea being embraced by the people. They reasoned that since some of the people refused the program when it was for free, it might be almost impossible that people would consider paying for it. 49 4.4.7 Conclusions and lessons  Most stakeholders at all levels consider IRS to have been an effective program because it drastically reduced malaria cases in communities.  Experience from the previous rounds of spraying have helped to reduce fears and concerns about the insecticide causing impotence among men and infertility among women.  Stakeholders from the previous IRS project districts in the north acknowledge that even when malaria has reduced following rounds of spraying, there is a real danger of malaria cases increasing again if spraying stops and if other prevention and control measures are not adequately implemented.  There is a lot of demand for IRS to resume in the previous project districts in Northern Uganda.  There are mixed views regarding willingness to pay for IRS. 4.4.8 Recommendations  IEC/BCC should continue to address people’s questions and concerns about IRS, such as those relating to nuisance mosquitoes, bedbugs and need for continued LLIN use after spraying.  The project should through its IEC/BCC component continue to dispel the myths and misconceptions about IRS and to provide accurate information about the benefits and safety of IRS.  The GoU/MoH should take advantage of the high demand for IRS in Northern Uganda to promote private provision of IRS through PSOs for those who can afford it. 4.5 How has the program monitoring system, including environmental compliance and the CLA approach supported program management, learning and adaptation? 4.5.1 Introduction Program monitoring as well as the collaborative learning and adaptation (CLA) approach are essential components of all USAID funded programs. Strong monitoring, evaluation, and CLA are important to continuously improve interventions. This section of the report discusses the evaluation question that states: “How has the program monitoring system, including environmental compliance and the collaborating learning and adapting approach supported program management, learning and adaptation?” 4.5.2 Monitoring, evaluation and learning system in the IRS II project Components of the MEL System in the IRS Phase II Project The project’s MEL system consists of a number of inter-related elements all anchored in the project’s Performance Management Plan (PMP) now transformed into the Activity Monitoring, Evaluation 50 and Learning Plan (AMELP) which presents the strategy and indicators for monitoring, learning and evaluating progress in each of the three key results areas of the project. The PMP/AMELP is updated annually to incorporate any changes and new developments to the MEL and CLA processes. The project produces monthly, quarterly and annual reports on program implementation. The project collects operational data on structures sprayed, populations covered and amount of insecticide used which the supervisors summarize and transmit daily using SMS and smart phones to the district bio-statisticians and project data management staff. This is subsequently analyzed and compiled to generate reports. The project team also collects information during courtesy visits to the districts, district sensitization meetings, trainings, supervision visits, spray progress reports, data quality assessments, and other avenues. The project also monitors input, process and output non-core indicators in key thematic areas such as environmental and entomological compliance. The project manages a database of key outputs data. As a means of data quality assurance, the project undertakes data quality assessments (DQAs) during and after spray rounds and uses the results to address any errors and omissions discovered. The project has conducted operational research on selected topics including on women’s involvement in spray activities, role of VHTs in IRS, and the impact of giving sanitary pads to female spray operators on their involvement and efficiency. The studies on women’s involvement and on sanitary pads helped to identify critical gender issues relevant to the success and gender inclusiveness of IRS activities. The project has also documented a number of success stories including those on recycling and re-using plastic wastes, use of live fencing to protect communities and the environment, establishment of the Gulu University insectary, use of mobile money system to pay Spray Operators, and the empowerment of women to actively participate in IRS activities, among others (see Annex VIA for sample success story). These have been shared with partners. Data is used to produce reports which are shared with districts and other stakeholders, and is entered into the Partner Reporting System (PRS) managed by the Learning Contract, which enables partners to access it timely and when needed. However, interviews with the NMCP indicates that the project’s information sharing with MoH/NMCP has been rather inadequate. As part of the evaluation processes, the project conducts annual reviews with districts. The project has a learning agenda and has continuously learnt from its work and made improvements based on this learning. More about this is included in sub-section 4.5.3. Use of monitoring data in decision making and program management The MEL system as described above has generated useful data that has been used by the project to plan subsequent interventions and improve performance. The project staff respond to the information collected in various ways depending on urgency, availability of sufficient information, and the expected impact of the situation learned about. For instance, information on existing problems with spray operations may be acted upon immediately to improve spray performance. Other problems that require senior management meeting decisions may be referred to the SMT for appropriate action. Experiences from and information about previous spray rounds are also used to better plan up-coming spray activities as one member of the project team explains below: 51 Data on spraying from one set of districts is used to plan spraying in the next set of districts. For instance, when we establish the amount of insecticide used per square metre of the wall, this information informs planning for the next spray activities in terms of how much insecticide to procure. (KII with staff, IRS II project) Periodic reports are used as a point of reference to make decisions and inform future planning. Table 5 below also presents some examples of how the project based on performance monitoring data to design strategies for improving performance. Table 5: Examples of project use of monitoring data to improve program performance Level of performance Explanation and lessons learnt Remedial action taken to address under-performance Evidence of performance improvement after remedial action 4 districts (Kitgum, Apac, Nwoya and Gulu) achieved less than the targeted 90% IRS coverage in Round one ((Oct￾Nov 2012) Under-achievement in these districts attributed to hard-to￾reach areas that were not properly mobilized; inadequate on-ground supervision. Enhanced mobilization using rotational team leader approach18; and enhanced supervision at all levels. Coverages across all 10 districts increased during the next round of spraying (April-May 2013) to an average of 96.5%19. Under-achievement in the inaugural round of spraying in the new 14 project districts with IRS coverage of only 67.2% against a target of 90%. Low coverage mainly attributed to resistance to IRS amongst organic farmers in the Lango region The project reinforced its mobilization efforts: enrolled prominent politicians, district leaders, district staff and local leaders to advocate for IRS and respond to concerns; used door￾to-door mobilization, IPC and community dialogues; targeted households that were resisting IRS. Worked through the sub￾county task forces to conduct social mobilization in schools, religious gatherings, marketplaces, health facilities and other gatherings. Achieve 92.8 percent and 94.4 percent coverage during the subsequent spray rounds20 18 In this approach a resident SO from a specific village acts as the team leader on the day of spraying that village, and a group of neighboring villages, if these villages are not represented on the spray team. This arrangement enables better mobilization and sense of ownership in the village and the adjacent villages given the team leader’s familiarity and rapport with the communities. 19 Source: Abt Associates Inc. (2013): Uganda Indoor Residual Spraying (IRS) Project Phase II Annual Performance Report: June 26, 2012 through September 30, 2013. Uganda Indoor Residual Spraying Project Phase II. 20 Source: Abt Associates Inc. (2015): Uganda Indoor Residual Spraying (IRS) Project Phase II Annual Performance Report: October 1, 2014 – September 30, 2015. Uganda Indoor Residual Spraying Project Phase II. 52 4.5.3 Collaborative Learning and Adaptation Definition/conceptualization of CLA in the IRS project Based on interviews with IRS II project staff and the documents they produce, CLA is the context of this project understood as a process by which information generated during the course of project implementation is made available to users, shared with stakeholders, and used to improve performance. Existence of a learning agenda The project PMP updated in August 2015 included a ‘Learning Plan’ which described the learning activities or avenues, how learning and knowledge share would be integrated into the ongoing project activities, and how the lessons learned would be used strategically to maximize program results. The key learning avenues outlined include courtesy visits to the districts, district sensitization meetings, training of trainers, training of spray operators, refresher trainings, supervision visits, short message service (SMS), spray progress reports, and data quality assessments. The frequency of these activities and the persons to be involved are also specified. It is noted however that the project’s learning plan as included in the 2015 PMP focuses more on how information for learning would be generated and less on how learning would be facilitated, who is intended to benefit from learning, and the format in which information is to be shared. Environmental Compliance Monitoring The project developed and implemented environmental compliance and mitigation monitoring plans and guidelines in accordance with the WHO, USAID and Uganda environmental regulations. Interviews with project staff indicated that these plans and guidelines are being adhered to. The project has supported effective, safe and secure insecticide storage management. Interviews with the project team and review of key project documents indicated that pre, mid and post spray inspections are conducted in compliance with required standards. Operational staging sites appear to be well-managed and maintained, and solid waste disposal is managed through use of a standard incinerator. The project has since discovered that it can spend less through recycling of plastic wastes, and has accordingly enter into arrangements with a recycling company to this. There were few reports of insecticide pilferage, misuse, poisoning and accidents, but the project respondent promptly to take appropriate action, including apprehending persons involved, taking punitive action if they were among the staff or spray teams, reinforcing supervision and other remedial measures. As part of the efforts to combat insecticide pilferage and improve the accuracy of data, the project has introduced the stamping of each household’ spray card by LCI chairpersons as confirmation that the household have been sprayed and the punching all the empty sachets returned to the parish stores on a daily basis to minimize the risk of reusing the empty sachets in accounting for the insecticide used. In addition to tracking core indicators during the IRS implementation, Uganda IRS project phase II monitored input, process and output non-core indicators in all thematic areas that included environmental compliance and entomology. These non-core indicators have been included to provide an overview of additional activities conducted by the project and its malaria partners in 53 work towards implementation of IRS operations that are compliant with environmental standards and supported by effective entomological interventions. Results from environmental monitoring activities have informed program decisions regarding for instance the re-location of IRS stores previously established at schools and health centres. Entomological monitoring & evaluation Annual reports and evaluation interviews with the project Vector Control Entomologists and District Vector Control Officers show that the Uganda IRS project phase II in collaboration with MoH and VCD and districts established six sentinel sites in six different eco‐epidemiological zones for monitoring entomological indicators including insecticide resistance. The project has implemented pre-and post IRS pyrethrum spray catches in 14 IRS districts and also in the previous 10 IRS districts to monitor changes in indoor resting vector densities as indicator of IRS effectiveness. Project reports show that the project conducted one-month post spraying cone bioassays in selected sprayed houses in the sentinel sites to assess residual efficacy of bendiocarb and the quality of spraying. The results from these entomological monitoring and evaluation activities have informed the projects’ change of the insecticides used. The project has worked with other USAID-funded projects including CHC, SDS and ASSIST, which has also promoted cross-project learning. 4.5.4 Lessons learnt from the project’s work and how this has enabled adaptation The evaluation team found that the project has documented lessons learnt from its work. Some lessons and experiences have been successfully applied to improve quality of performance and cost reduction. The table below summarizes how the lessons learnt have enabled shifts to improved practice. Table 6: Adaptations made by the project as a result of learning Previous practice Issues and lessons learnt by project Adaptations made by project based on lessons IEC/BCC 1 IEC mainly through radio talk shows, radio spots Resistance to IRS from organic farmers; learnt that radio talk shows and other BCC were not effective in reaching the resistant groups Use of IPC and door-to-door mobilization to re-inforce BCC messages 2 Initial IRS communication focused on IRS Resurgence of malaria in Northern Uganda; learnt that for sustainable benefits, people should continue using nets and other prevention practices Integrated of other malaria prevention messages into communication campaigns (LLIN use, IPTp, early diagnosis & treatment) 54 Previous practice Issues and lessons learnt by project Adaptations made by project based on lessons Planning 3 Use of existing data of HHs from UBOS to determine sprayable households Data out of date and did not accurately indicate number of households in targeted location; led to under-estimation of insecticides and other materials needed. Used GIS mapping; enhanced accurate planning, efficient implementation, and comprehensive spray coverage Monitoring and data transmission 4 Relied on Spray Operator records to know which houses were sprayed Some scrupulous Spray Operators took advantage to inflate records; Incidents of insecticide pilferage Introduced stamping of spray cards by LCI chairpersons which improved community engagement, accountability and reduced insecticide pilferage 5 Motorcycle couriers delivered data from the field every 2-3 days during spraying Delayed transmission of data (taking 2-3 days); high costs using motorcycle couriers to deliver data Introduced use of Smart phones and SMS technology to transmit data; faster data transmission; reduced cost of using motorcycle couriers Safety standards 6 All solid wastes transported to incinerator in Nakasongola Costly to incinerate all solid wastes Started recycling of plastic wastes; saved costs 8.8m Ug.Shs. (about USD 2,600) in one year. 7 Used dry wooden poles to fence staging sites to ensure safety High cost to replace fallen poles and replace fencing Introduced use of live fencing reduced costs and time of staging sites repairs Administrative efficiency 8 Cash payments made to Spray Operators after spray rounds Costly and risky carrying cash to different parishes to pay Spray Operators Rolled out payment through MTN mobile money; saved costs, improved timeliness, and reduced risks IRS sustainability 55 Previous practice Issues and lessons learnt by project Adaptations made by project based on lessons 9 Phase out from Northern Uganda districts Resurgence of malaria; learnt that exit planning is very important Project currently engaging project districts to formulate sustainability plans The information in the table above demonstrates that the project has effectively learnt from its work and used this learning to make adaptations that have improved project performance. The evaluation team noted however that CLA was still a relatively new concept to the project team and as such not all members of the project team are acquainted with what CLA is, what it entails and what is required to effectively apply the approach. Some of the staff under the MEL unit have undergone trainings, yet others have not. Those that have attended some trainings also report that they would require more capacity building in this area. Similarly, the evaluation team found that the MoH/NMCP has not been fully abreast with the important learnings and adaptions made by the project. 4.5.5 Conclusions and lessons  The project’s MEL system is comprehensive and included production and implementation of annual plans, routine data collection, and production of monthly, quarterly and annual reports which are shared with stakeholders.  The project has done considerably well learning from its work and using the lessons to make improvements  CLA is still a new concept to many project staff  The MoH/NMCP is yet to take ownership of the learnings and adaptions made by the project. 4.5.6 Recommendations  The project and its partners should introduce the use of smart phones and SMS technology in the remaining IRS districts.  PMI should provide CLA training and mentorship to the project staff  The project’s learning processes should be opened up more to include other stakeholders including the MoH/NMCP 56 5 ANNEXES ANNEX IA: EVALUATION STATEMENT OF WORK REQUEST FOR PROPOSAL EVALUATION OF UGANDA INDOOR RESIDUAL SPRAYING (IRS) PROJECT – PHASE II 1. BACKGROUND Uganda has the third highest number of P. falciparum infections in sub-Saharan Africa, and some of the highest reported malaria transmission rates in the world. There is stable, perennial malaria transmission in 90–95% of the country21. Approximately 90 percent of Uganda’s 38 million people live in areas with high malaria transmission (>1 case per 1000 population), and the World Health Organization (WHO) estimates there are about 17,000 malaria-related deaths annually. Nearly half of hospital in-patient deaths among children under five are attributed to malaria and there are even more unreported malaria-related deaths that occur at home (WHO, World Malaria Report 201322). Malaria transmission is persistently high in some areas of Northern Uganda. For example, before the implementation of (Presidential Malaria’s Initiative) PMI’s indoor residual spraying (IRS) in the 10 districts, Apac District reported an entomological inoculation rate of 1,60023. That is more than four infected mosquito bites every night and is among the world’s highest recorded. In most of Uganda, favorable temperature and rainfall allow intense vector propagation and perennial malaria transmission. Indoor Residual Spraying (IRS) is one of the Government of Uganda’s key interventions under the Malaria Reduction Strategy focussing on both low and epidemic-prone areas (to prevent malaria epidemics) and high transmission endemic areas, accompanied by environmental management where feasible and effective. Indoor residual spraying is an effective method in malaria vector control as its purpose is to eliminate adult mosquitoes and its continuous use leads to significant reduction in malaria transmission levels. Malaria prevention and control are major foreign assistance objectives of the U.S. Government (USG) hence the commissioning of a flagship Indoor Residual Spaying (IRS) activity under the United States President’s Malaria Initiative (PMI) in Uganda in 2009. The major focus for IRS programs in Uganda is to rapidly build and strengthen local implementation capacity in the public sector in the areas selected for this intervention. Additionally, it is to establish a system for regular and timely high quality IRS services that cover at least 85% of all targeted structures. PMI’s initial IRS project was implemented between 2009-2012 in 10 districts of; Kitgum, Lamwo, Pader, Agago, Apac, Kole, Oyam, Amuru Nwoya and Gulu. The current project, Uganda Indoor Residual Spraying (IRS) Project – Phase II started in June 2012 continued in the 10 districts and is currently being scaled up in an additional 14 districts of; Otuke, Lira, Alebtong, Dokolo, Kaberamaido, Amolatar, Serere, Pallisa, Kibuku, Budaka, Namutumba, Butaleja, Bugiri and Tororo by 2017 (map at Annex 1B). The purpose of Uganda Indoor Residual Spraying (IRS) Project – Phase II is to achieve Presidential Malaria Initiative (PMI)/Uganda targets in indoor residual spraying (IRS) through: (1) implementing a high quality 21 World Health Organization (2014): World Malaria Report. Geneva: WHO 22 http://www.who.int/malaria/publications/world_malaria_report_2013/en/ 23 Okello PE, Van Bortel W, Byaruhanga AM, Correwyn A, Roelants P, Talisuna A, D'Alessandro U, Coosemans M. 2006. Variation in malaria transmission intensity in seven sites throughout Uganda.Am J Trop Med Hyg. Aug;75(2):219-25 57 IRS program, (2) conducting comprehensive monitoring and evaluation of the IRS program and (3) developing the national capacity to conduct IRS. Implementation of the IRS is in full collaboration with the Uganda National Malaria Control Program (NMCP). The total estimated cost of the IRS Phase II project is $78,836,757 implemented by Abt. Associates over a period of five years. 1. EVALUATION PURPOSE AND USE The main purpose of this evaluation is to provide an independent opinion on the extent to which the IRS project is achieving its objectives during the 2012 – 2014 period, document emerging lessons that can be scaled up in additional districts during the remaining period of the project. 2. AUDIENCE The primary user of the evaluation findings is PMI Uganda, USAID Uganda staff, other United States government agencies; USAID funded implementing partners (IPs), Ministry of Health specifically the National Malaria Control Program (NMCP) and other national and international stakeholders with interest in malaria programing. 3. EVALUATION QUESTIONS The evaluation will specifically answer the following questions: i. To what extent is the IRS program meeting the high-quality standards set out in the program design? ii. To what extent did the Behavioral Change Communication (BCC) campaign influence desired behaviors in net use, prevention of malaria in pregnancy, seeking treatment early and other kinds of malaria prevention and management practices? Initially, the BCC campaigns were focused on raising awareness around IRS, but this was subsequently modified to promote behavioral change in malaria prevention and management. iii. How have structures and partnerships developed under NMCP been strengthened to continue implementing IRS interventions? Structures and partnerships under NMCP are both public and private companies/ organizations at the national, district and community levels. iv. What are stakeholders’ perceptions on the effectiveness of IRS? v. How has the program monitoring system, including environmental compliance and the collaborating learning and adapting approach supported program management, learning and adaptation? These are illustrative questions and there will be an opportunity for these to be modified once the team of consultants is on board. 6. Methodology The evaluation will apply cross-sectional design using mixed method approach i.e. using both quantitative and qualitative methods. The evaluation team is expected to propose and use sound sampling techniques to determine districts, villages, households and respondents to be visited and which data will be collected. Suggested data collection methods include:  Review of Program Documents and Related Literature: There are several IRS related documents that have been produced. The evaluation team will be provided access to these documents for review. These documents may include solicitation documents, annual work-plans, Performance Management Plans (PMPs)/Monitoring, Evaluation and Learning Plans, progress reports (e.g. quarterly, semi-annual and annual reports), district and health sector planning, strategy policy and performance documents etc. In addition to these documents, evaluation team may access malaria 58 related data from the DHIS-2 and other sources as may be guided by the NMCP. The evaluation team will decide on the type of analysis that can be conducted using these secondary data sources in addition to the primary data in order to answer the evaluation questions.  Key Informant Interviews (KII): Using a structured questionnaire comprising primarily of open ended questions, the evaluation team will conduct in-depth interviews with respondents from Ministry of Health, USAID, local government, non-government stakeholders and health facility managers and service providers in implementation areas of the three projects.  Focus Group Discussions: Focus group discussions (FGD) will be another method that could be used to collect data from IRS stakeholders. These may include staff from local governments, partner organizations, CSOs, and, clients who have utilized services offered through these projects, etc.  Survey of Key Stakeholders: It is also suggested that the evaluation team consider conducting a survey of a sample of clients/beneficiaries in selected districts where these projects were implemented. This survey could be used to assess the extent to which the IRS approach was effective. The evaluation team will develop tools and detailed guidance for data collection and work closely with NMCP, District staff, PMI and USAID/Uganda to identify appropriate respondents. All data collection instruments and guides will be approved by USAID/Uganda and PMI prior to the beginning of fieldwork. The evaluation team will propose data analysis strategies and tools for both the qualitative and quantitative data. The team will be expected to conduct trend analysis; comparisons of performance and changes in the targeted districts over time as appropriate and any other comparisons that could highlight achievement or lack of achievement of positive health effects related to malaria from IRS implementation. Data disaggregation and analysis by gender and age to establish the differential effects of the project on men, women and different age groups will also be expected. The team will propose other analysis approaches. The evaluation team shall describe the type of software for quantitative and qualitative data analysis they propose to use. Additionally, the evaluation should establish the key lessons learnt from the referred to phase (2012-2014), how they can inform phase II and how the lessons may be replicated in other parts of the country. 7. KEY DELIVERABLES The evaluation team is expected to deliver the following products: a. Attend in-briefing meetings: The evaluation team will be required to attend in brief meetings to discuss the statement of work with key stakeholders including, The Learning Contract, USAID Uganda/ PMI and the NMCP. b. Inception Report: A report detailing the evaluation team’s interpretations of the assignment, proposed evaluation design, methodology, sampling (including districts, communities and households), analytical plans, tools for data collection and analysis, detailed evaluation plan with timelines and data collection tools. A filled evaluation design matrix following the template as attached as Annex 3 should be included in the Inception report. The report should also provide an overview of the methodology that will be used to select areas to be visited and respondents/participants. 59 c. Debriefing: Oral presentation by the evaluation team after completion of data collection and field work. The presentation should articulate the team’s readiness to begin report writing exercise i.e. availability of information to answer evaluation questions. d. Oral Presentation: Power Point presentation (including hand-outs) to: i. Learning Contract prior to the USAID presentation ii. USAID/Uganda PMI iii. NMCP and Key stakeholders. The main presentation will normally be for 60 minutes covering the major findings, conclusions, and lessons learned, and allowing for about 30 minutes of discussion and feedback. e. First Draft Evaluation Report: The content should cover all the main elements of the report including major findings, conclusions, lessons learned, and relevant annexes. The input from the oral presentation sessions should also be incorporated in the report. The first draft should be 15- 20 pages and any other annexes f. Final Draft Evaluation Report: A complete report presented in the agreed-upon format and incorporating comments from USAID Uganda and other stakeholders. The report should comply with the USG evaluation report standards set out in Annex 4 and 5 (USG as the funding agency). g. Cleaned labeled and ready to use electronic copies of datasets collected through fieldwork and cleaned ready to use electronic copies of FGD analyses if any. In addition, copies of all instruments used in data collection must be separately delivered to USAID. h. Final Report: The team leader will submit a final report within one week of receiving final comments from USAID/Uganda including those from other stakeholders. The core report should be less than 60 pages, including project specific pullouts but excluding annexes. 8. TEAM COMPOSITION The evaluation team must include one IRS specialist; Evaluation Specialist with statistical and data analysis competencies and one other senior Behavior change and capacity building expert with knowledge in district structures, community behavior, malaria monitoring and surveillance systems among others. A Team Leader, selected from among the consultants; will be responsible for assigning tasks to team members and supervising performance and coordinating the evaluation activities. S/he will be the main point of contact between the evaluation Activity Manager at USAID/Uganda, QED and the evaluation team. The Team Leader will review all plans and outputs and be responsible for submitting quality products to QED and USAID/ PMI. The team leader must have experience as an evaluation Team Leader and has played significant role in designing, managing or executing in a minimum of five evaluations. The IRS expert shall have:  A minimum of a Master’s degree in Public Health, Epidemiology, medicine and or related field.  Should have knowledge and experience in the design, management and evaluation of malaria control programs in Africa or similar environments.  Shall have at least seven years’ experience in the design and management of IRS programs including environmental compliance; logistics management, storage; insecticide chemicals for IRS; and capacity building for IRS in sub Saharan Africa or similar contexts. 60 Evaluation specialist  Must have played significant role in designing, managing or executing evaluations including data analysis.  Must have significant expertise in analyzing large datasets health datasets in developing countries  Will be an evaluation expert with at least five years’ experience in evaluating malaria control programs  Knowledge and experience in evaluating behavioral change communication programs is an added advantage.  Must be familiar with Health systems in Uganda or East and sub-Saharan Africa.  Excellent report writing and analytical skills Capacity building and BCC Specialist  Must have knowledge and experience in evaluating communication for behavior change and local government capacity building programs.  Must be familiar with Health and district systems in Uganda or East and sub-Saharan Africa.  Ability to analyze data and has report writing competencies will be of an added advantage. 9. Duration: The task is estimated to begin on or about the beginning of December 2015 and be completed no later than March 2016. Contract may provide for protracted timeline outside the performance time to provide for billing and closure. 10. LOCATION Kampala and regional field offices, and site visits conducted in the different beneficiary districts, health facilities and communities in districts where IRS was implemented in the years 2012 – 2014. 11. MANAGEMENT ROLES AND RESPONSIBILITIES The USAID Senior Strategic Information Advisor, will have primary administrative and technical responsibility for the evaluation process. This also includes making the necessary arrangements for USAID inputs and briefings. The Evaluation Team will liaise closely with the Contracting Officer’s Representatives (CORs) for IRS, USAID PMI staff, SI Team and the Program Office M&E Specialists on coordination and clarification of USAID requirements and standards for maintaining effective communication on what is to be learned. The IRS project will participate in the design and planning of the evaluation, support with logistics for implementation (avail copies of relevant documents, arrange for meetings and interviews, etc.), participate in the oral presentation and review the draft and final reports. GoU/MoH National Malaria Control and other stakeholders, will participate in the review of the statement of work, oral presentation and review of the draft and final reports. The evaluation contractor will provide own laptop computer(s) and other equipment necessary for performance of this activity; transport hire (within Kampala and upcountry), office supplies, communication costs, field work expenses (including survey, focus groups and dissemination). The evaluation contractor should anticipate meeting field costs for about 3 government staff who may participate in the evaluation upon approval by USAID. Implementing Partners, USAID and other stakeholders will not interfere with the evaluation team’s capability to collect objective information and to conduct independent investigation relevant for this evaluation, analyze data and make inferences, conclusions and recommendations. 61 12. EVALUATION CRITERIA 1. Technical Approach Sub criteria in order of importance  Extent to which the proposed technical approach is clear, logical, well-conceived, technically sound and accurately interprets the evaluation questions provided in this Statement of Work  Extent to which the proposed technical approach demonstrates an understanding of the implementation context 2. Key Personnel  Extent to which the proposed key personnel meet the required qualifications demonstrating the Offer’s ability to conduct the evaluation 62 Annex 1B: Map Showing IRS Project Phase II Coverage 63 Definition of terms24 Result 1: High quality safe and effective IRS program implemented High quality  Use appropriate technologies and tools such as Geographical Information System (GIS), remote sensing, and Personal Data Assistants (PDAs) for geographical reconnaissance, tracking and data management to ensure cost effective program planning and implementation.  Work with the NMCP, PMI, districts and other relevant authorities to select the appropriate insecticide, equipment and accessories to be used in the various geographic areas targeted for spraying. Selection shall be done in accordance with World Health Organization (WHO) and Government of Uganda guidelines, namely: insecticide.  Spraying exercise is conducted in a timely manner to meet WHO technical standards;  Evidence based and cost effective in achieving the required targets and goals. Inclusiveness  IRS programs planned, managed and implemented in collaboration with key Uganda Government ministries and departments25.  Prior to commencement of spraying, conduct appropriate planning in collaboration with the NMCP, districts and other relevant authorities to spray at least 10 districts per year (defined on the basis of 2011 administrative borders) during the appropriate timeframes as directed by PMI. Safety  Susceptibility and vector behavior; safety for humans and the environment; efficacy and cost effectiveness Effective  Adequacy of human resources, office space, storage facilities, waste disposal logistics, skills/ training.26  Programmatic infrastructure for conducting the IRS program and related environmental and entomological activities - Systems in place to provide information and Behavior Change Communication (BCC) to safely and properly implement the program - Provide appropriate IRS￾specific Information Education. Communication/Behavior Change Communication (IEC/BCC) prior to, during and after any campaign. The population to receive IRS shall be informed of all aspects of the IRS program through radio, community meetings, and other less costly and effective methods of communication.  Prepare and execute logistical plans for IRS-related commodities in compliance with Uganda national policies on IRS and WHO IRS guidelines. This includes, but is not limited to, timely procurement, storage and distribution of all supplies, such as insecticides, spray pumps, protective clothing and any other supplies and equipment required to carry out the IRS operations in Uganda. 24 Additional definitions and description of key terms and statements shall be derived from the IRS program description at the time of the evaluation. 25 MoH/NMCP, National Environment, Management Authority (NEMA), Ministry of Agriculture, Ministry of Water, Lands and Environment (MWLE),National Bureau of Standards (NBS), National Drug Authority (NDA), Ministry of Finance (MOF) and other relevant authorities in designated Uganda districts according to international standards. 26 Plan and implement appropriate and necessary training activities for the range of skills needed to safely and effectively implement, monitor, and evaluate IRS including, but not limited to, refresher training for experienced spray operators, supervisors, clinicians, entomology technicians, environmental officers administrative and other support staff as well as training for new staff. 64 ANNEX II: ANNEXES TO EVALUATION METHODS AND LIMITATIONS Annex IIA: Characteristics of Sample districts District IRS Phase II Project period and other district characteristics Scope of data collection activities Old IRS Phase II districts Gulu 2012- Nov 2014 HH survey, FGDs, KIIs Amuru 2012-May 2014 Kitgum 2012-May 2014 Apac 2012-May 2014; previously with highest malaria endemicity Oyam 2012- Nov 2014 New IRS Phase II districts Amolatar Dec 2014 – On-going; HH survey, FGDs, KIIs Lira Dec 2014 – On-going; Initially poor program intake but improved Budaka May 2015 – On-going; Serere May 2015 – On-going; good program intake Tororo Dec 2014 – On-going; good program intake Dokolo Dec 2014 – On-going; high resistance to IRS from organic farmers; spraying ongoing at time of data collection. Observation of IRS stores & sites; interviews with stores personnel Kibuku May 2015 – On-going; spraying ongoing at time of data collection. Otuke Dec 2014 – On-going; persistent high malaria cases despite IRS; spraying ongoing at time of data collection. Observation of IRS stores & sites; interviews with stores personnel; KIIs with district and sub-county key informants. Annex IIB: Sampling of sub-counties, parishes and villages From each of the sampled districts, two sub-counties were selected, one rural and one urban/peri-urban. Selection of sample sub-counties from each cluster (rura/urban) followed consultations with district officials. Where all the eligible sub-counties were considered more or less similar in terms of social-economic characteristics and response to IRS, random selection was done. Where some sub-sub-counties had peculiar characteristics of interest to IRS, purposive selection was done taking into account such unique considerations such as pockets of resistance to IRS. From each selected sub-county, two parishes were selected and in each parish, between one to three villages were selected using simple random sampling, depending on the number of sample households needed to raise the district household sample quota computed for this evaluation. Annex IIC: Data collection Methods Data for the evaluation was collected through desk reviews, a household survey, in-depth interviews with key informants, focus group discussions, and direct observation at IRS sites and stores. 65 Desk Review: The evaluators conducted a review of relevant documents including project documents such as annual workplans and annual reports; GoU policy documents such as the Malaria Operational Plans (MOPs), IRS manuals and guidelines; as well as national surveys, namely, Malaria Indicators Surveys (MIS), Uganda Demographic and Health Survey (UDHS), and District Health Information System II (DHISII). See Annex IIH for the list of documents reviewed. In-depth Interviews with Key Informants: The evaluation team conducted in-depth interviews with key informants selected from key stakeholders at community, sub-county, district and national levels. The stakeholders included community leaders such as LCs, opinion leaders and religious leaders; VHTs; district and sub-county staff and political leaders; health facility in-charges; IRS project staff from Abt. Associates; MoH/NMCP; MoH vector Control Division; government regulatory agencies including National Environment Management Authority (NEMA) and the National Drug Authority (NDA); CSOs and PSOs; and staff of PMI. A summary list of the key informants interviewed is provided in Annexes IIF and IIG. Household survey: A household survey was conducted among a sample of households using a hard copy structured questionnaire. The survey sought to collect information about target beneficiaries’ views on IRS, levels of satisfaction, exposure to BCC messages and their responses to these messages in terms of adoption of recommended malaria prevention practices. Semi-structured interviews with Spray Operators and Store Keepers: Interviews were held with Spray Operators and Store Keepers to assess issues of training, personal safety, information dissemination, and other aspects related to their work. Semi-structured tools were used for this purpose. Focus Group Discussions (FGDs): FGDs were conducted among program beneficiaries, specifically, male and female adult community members, pregnant women and women who have a child under the age of one, youths, VHTs as well as community leaders. Each FGD target between 8 to 12 participants and were conducted in the local language. A summary list of the FGDs conducted is provided in Annex IV. Observation: Direct observation was conducted at selected IRS sites and stores to assess aspects of environmental compliance and adherence to standards. The research team observed the type of equipment and materials used for personal protection; and presence, design and location of soak pits for disposal of biodegradable wastes, wash areas, and progressive rinse and evaporation tanks. Annex IID: Study sites sampled for the Evaluation District Sub-county Parishes / Wards Villages / cells Budaka Kameruka Lerya  Lerya  Bunamwera Bupuchai  Natoto  Bunyekero Kamonkoli Kamonkoli  Kamonkoli  Nyanza Sekulo  Kapulukuchu  Kosti Serere Serere T/C Kakusi  Kakusi 66 District Sub-county Parishes / Wards Villages / cells  Township Osuguro  Ajesa  Kikota Kadungulu Iruko  Alilimkipi  Adepi Kabulabula  Aturia  Aputon Tororo Western Division (Urban) Bison Maguria  Bison A  Bison B  Bison C (Juba) Agururu A  Agururu A1  Agururu A2 Mulanda (Rural) Lwala  Koi  Powino North  Poleko Mulanda  Atangala B  Sirongo A  Korobud B Lira Agweng Abala  Acanpii  Barkogo Augolocom  Teadwong  Corner Lwala Railways Division Railways Quarters  Railway Qtrs “A”  Railway Qtrs “B” Temogo  Market square  Okole Amolatar Namasale Izigwe Kikondo  Adagami ‘B’  Kipangani ‘B’ Amolatar Town council Amirimiri Apalepe  Oryamai  Amolatar  H/Quarters Apac Chegere Chegere  Anyambazi ‘A’  Anyambazi ‘B’ Ongica  Abwal  Ougica Apac Town council Eastern Ward  Upper Centre Akal Western ward  Biashara  Temogo Oyam Ngai Akuca  Baribule  Bar -Ryeko Aramita  Bar Rec ‘A’  Bar Rec ‘B’ Western ward  AlutKot 67 District Sub-county Parishes / Wards Villages / cells Oyam Town Council  Akwach Eastern Ward  Akatakata  Akaidebe Gulu Pece Division Pawel  Pudyek  Pawel Central  Cubu Acoyo Tegwana  Layibi Central  Cubu Paicho Sub County Kal Umu  Acut Omer  Kicike Kal Ali  Tee-Olam  Kinene  Laminto Amuru Amuru Town Council Pogi  Abongodang  Labila om Amoyo Koma  Kigoli  Atiti Pabo Kal  Oguru  Kal Central Parubanga  Payibi  Abera Kitgum Omiya Anyima Palwo  Abwore East  Mota Forest Pela  Kweyo  Omiya Anyima central Labongo Layamo Pagen  Pagen west  Pagen central Ocete Toke  Ocete Toke Central  Ocete Toke West 68 Annex IIE: Number of FGDs Conducted by category of participants District Number of FGDs by category of FGD participants Adult male community members Adult female community members Pregnant women + women with a child <1 year Youths Community leaders VHTs Total Budaka 1 1 1 1 2 6 Serere 1 1 1 1 1 2 7 Tororo 1 1 1 1 1 2 7 Lira 1 1 1 1 1 5 Amolatar 2 1 1 4 Apac 1 1 1 1 1 5 Oyam 1 1 1 1 1 5 Gulu 2 1 1 1 5 Amuru 1 1 2 4 Kitgum 1 1 1 2 5 Total 8 7 9 5 9 15 53 Annex IIF: List of Key District and Sub-county Informants Interviewed District Name of Key Informant Position Budaka Baya Martin Mpindi Feliste Wajege Sam Mako Richard Kukyana Sam Kabise Shaban Kamwadda K. Sepriano Oswam Max Goola Silver Hadija Butunda Mysa Fatiya, Gonsa James Kigaya Isaac Moses Pilingo Edmond Sub County Chief Ochola Robert Capt. Okiroy Chris. M Mr. Munghoni Franc Assistant Chief Administrative officer District Community Development Officer District Health Officer District Health Educator District Information Officer District Planner Environmental Officer Health Assistant Kamonkoli HMS Focal Person Budaka HCIV In Charge Nyanza HCIII CDO, Kameruka Chairman LCIII Sub county Chief LCIII, Chair Kamonkoli Were Rog Malaria / IRS Focal Person Resident District Commissioner Vice Chairman LCV 69 District Name of Key Informant Position Serere Jim Ojara Moses Agum Joseph Opet Sam Amecu Samuel Okalebo Jimmy Otimo Thadeo Oonyu Francis Odeke Dennis Eyoku Odeke Emmanuel Ochom Julius Mohammed Tukei Emuron Isaac Ayau Augustine Onyabuk Joseph Patrick Micheal Anguria James Okia Emmanueil Omeke Onya G. Edward Akonopesea Hon Grace Aujo Philip Keith Magezi Larun Jacqueline Accounts Assistant; Planning Office Assistant Chief Administrative Officer Chairperson LC5 Community development officer District Community Development Officer District Health Educator / Communicator District Health Inspector District Health Officer Environment Compliance Officer Assistant in charge Serere HVIV Nursing officer Serere HVIV Health Inspector; Serere town Council Clinical officer, In charge Kadungulu HCIII IRS sub-county supervisor Kadungulu IRS Supervisor Serere Town Council Kadungulu Sub county chief LC3 Chairperson – Serere Town Council Vector control officer IRS focal Resident District Commissioner (RDC) Secretary for Health Stores manager, Abt Associates Field Coordinator, Abt Associates Tororo Patrick Owino Omita Kirya Aquzas Mr. Jeckins Ongaria Mr. Sam Vitalisi Mr. Methusehah Oloka Mr. Nathan Oboth M/s Evelyn Aol Mr. John Willy Mungoma Mr. Williaum Mulabya M/s Anna Asinde Aluka Jacob Mr. Tom Ofumbi M/s Sarah Achen Acia James Othieno Susan Dr Bwayo Isaac Omella Isaac Aseu Omojong Nicodemus Mr. Okuga /s Anna Asinde Isa Tegabulana Mr. Micheal Opera Assistant District Officer in charge of Environmental Health CDO Mulanda Sub County Chairman LC III Chief Administrative Officer CDOO, Western Divisional Community Liaison officer- Bison Police Post District Environmental Officer District Health Educator District Planner Entomologist Health Assistant Mulanda Sub County Health Assistant Western Divisional Office Health Assistant In Charge, Mulanda Health Center IV Acting LCIII chair person Bison Health Center III Sub- county chief Lab Assistant/ IRS Store Keeper Malaria Focal Person Senior Vector Control Officer Town Clerk, Western Divisional Office Lira Muhammed Were Dalili R.K.Moses Agalo Evelyn Tonny Ocen Mary Okello Chairman LCIII-Railway Division in Lira Chief Administrative Officer Community Development Officer Community Development Officer-Railway in Councilor, Agweng Sub county 70 District Name of Key Informant Position Okello Francis Grace Atim Dr Ocen Patrick Omoo Henry Rashid Mwesige Tino Deborah Geoffrey Ogwal Opio Bonny Munu Nelson Benson Abor Anthony Ojuka Harriet Aboke Ameny Linus District Community Development officer District Health Educator District Health Officers, Lira District Planner, Lira District District Vector Control Officer Health Assistant Health Assistant-Agweng sub-County-Lira Health Facility In-charge Abala HCIII Health Facility In-charge Ayago HCIII IRS Sub-County Supervisor-Railway Division Secretary for Environment Sub-county Chief- Agweng VHT Coordinator, Agweng Sub County Amolatar Peter N.Ruhweeza Anach Jerome Ongu Rembo Denis Orech Michael Egwange John Bosco Dr. Kwinto Okello Oyet Vincent Orech Sam Debong David Dr. Aliga Simon James Odyek Richard Kizza Emmy Awongo Catherine Awor Epel Godfrey Obote James Acting CAO-Amolatar Biostatiscian LC III Chairperson Community Development Officer Community Development Officer District Health Officer Project Assistant, CDFU Health Assistant, Namasale Sub-county Health-in-Charge/Senior Clinical Officer In charge Amolatar Health Centre IRS supervisor for Amolatar Town Council Malaria Focal Person Mayor Amolatar Town council Senior Development Officer Town Clerk Amolatar Town Council VHT Coordinator Apac Martin Oleny Nelson Apel Tom Otim Okello Jimmy Emmanuel Geoffrey Ococ Otimoi Jasper Dr Oyer Mathew Atim Tom Richard Gerald Kalimo Doris Enderu Ongom John Robert Ocan Joseph Omuut John Robert Ongu Patrick Elvis Martin Ogwang Odong David Samuel Acting In-charge Chegere HCII CDO –Apac Town council Chairman LCIII- Chegere sub-county, Apac Councilor/Secretary for Works Deputy Mayor –Apac Town council District Environment Officer District Health Officer APAC DISTRICT District Planner Health Assistant/ IRS coordinator Health Assistant/ IRS Supervisor Chegere Health-in-Charge, Biashara HCII Human Resource Officer, Apac Town Council Resident District Commissioner, Apac Sub County Chief, Chigere Vector Control Officer /Malaria Focal Person Branch Focal Person, Apac Oyam Joseph Owani Opio Emmanuel Aguti Joyce Orech John Bosco Acting-In-charge Anyeke HC IV Community Development Officer, Ngai Sub Community Development Officer, Oyam TC District Health Educator (Acting DHO) 71 District Name of Key Informant Position David Ogwal Opio Moses Emor Denis Alex Okirira William Okori Trasis Odongo Jimmy Anyuru Jillian Akullu D’Ngai Ogwang Ogwal Geoffrey Kandole Samuel Peter District Malaria Focal person / VCO-Oyam Environment Officer Health Assistant, Ngoi Sub County Health Assistant/ IRS Supervisor Health Facility In-charge Ngai HCIII IRS coordinator/ supervisor Mayor – Oyam TC RDC-Oyam Secretary for Health Ngai Sub-county Senior Community Development Officer Chief Administrative Officer Gulu Obang Micheal Owol Constantine Atto Franaska Okello Davis Dr. Ongom Robert Akwir Florence Denis Olobo Ochauca James Lakot Caroline Isaac Newton Ojok Principal health officer District Entomologist Environmental officer District Health Inspector District Health Officer Enrolled nurse Vector Control Officer Municipal Environmental Officer Nursing Officer, Aywee HCIII Vice Chairperson LCV Amuru Donato Dila John Bosco Olum Ajok Doreen Lanyeko Dr. Odongo Patrict Unicee Piloya Odongo Patrick. B Labwola Godfrey Apio Grace Komaketch Micheal. C Chief Administrative Officer Community Development Officer District Environmental Officer District Health Officer Heath Assistant, Amuru TC Laboratory technician, Pabo HCIII Malaria Focal Person Nursing Assistant Sub County Chief Paboo Kitgum Ochengel Ismael Ocamker Gerald Atwong Morris Anyongo George. W Dr. Olwrado Alex Komakech Jolly Bosco Omony Denish Nodert Lanyero Caudida Oloya Robert Dr. Akena Geofrey Chief Administrative Officer Chief Omya- Anyima Sub County Chief CDO District Health Officer Vector Control Officer Health Assistant Enrolled nurse Laboratory assistant Medical Superintendent Otuke Mr. Sebandeke Richard Dr. Oryem Robert Benard Abwang Health Assistant Mr. Acal Denis Okao John Bosco Opio Denis Mark Mr. Opio Patrick Akolla Norbert CAO Otuke District District Health Officer Field Coordinator, Abt Associates Olilim Sub County In charge Atangwata H/C Iii In charge Okwang Health Centre III In charge Orum Health Centre Iv District Health Educator District Supplies Officer 72 District Name of Key Informant Position Akello Vick Ongwang Kenneth. Abooke John Bosco Ocen Duke Adupa Odong Moses Okwir Okello James Nursing assistant Principal Health Inspector In charge /nursing officer Sub county chief CDO/ Sub county Supervisor IRS Vector Control Officer Annex IIG: List of National Level Key Informants Interviewed Name of Key Informant Position Organization Dr. Jimmy Opigo Program Manager National Malaria Control Programme, MoH Mr. Anatol Maranda Byaruhanga Entomologist Vector Control Division, MoH Mr. Denis Mwesigwa Senior Inspector of Drugs National Drug Authority (NDA) Mr. H. Nabaasa National Environment Management Authority (NEMA) Mr. George Mugarura Estate Manager Ankole Tea Estate Mr. Ronald Byarugaba Estate Manager Bugambe Tea Estate Mr. Ndemire Uganda Prisons Mr. Chris Kaweesa Managing Director Uganda Pest Controllers Association Mr. Joel Kisubi Programme Officer PMI Ms Jessica Okui Mission Environment Officer USAID Dr. Kassahun Belay Technical Advisor PMI 73 Annex IIH: List of Documents Reviewed 1. Abt Associates Inc. (2013): Uganda Indoor Residual Spraying (IRS) Project Phase II Detailed Workplan: October 2012 to September 2013. Uganda Indoor Residual Spraying Project Phase II. 2. Abt Associates Inc. (2014): Uganda Indoor Residual Spraying (IRS) Project Phase II Detailed Workplan: October 1, 2013 to September 30, 2014. Uganda Indoor Residual Spraying Project Phase II. 3. Abt Associates Inc. (2015): Uganda Indoor Residual Spraying (IRS) Project Phase II Detailed Workplan: October 2014 to September 2015. Uganda Indoor Residual Spraying Project Phase II. 4. Abt Associates Inc. (2013): Uganda Indoor Residual Spraying (IRS) Project Phase II Annual Performance Report: June 26, 2012 through September 30, 2013. Uganda Indoor Residual Spraying Project Phase II. 5. Abt Associates Inc. (2014): Uganda Indoor Residual Spraying (IRS) Project Phase II Annual Performance Report: October 2013 through September 30, 2014. Uganda Indoor Residual Spraying Project Phase II. 6. Abt Associates Inc. (2015): Uganda Indoor Residual Spraying (IRS) Project Phase II Annual Performance Report: October 1, 2014 – September 30, 2015. Uganda Indoor Residual Spraying Project Phase II. 7. Abt Associates Inc. (2015): Uganda Indoor Residual Spraying (IRS) Project Phase II Performance Management Plan for October 2015 to September 2016, Updated August 28, 2015. 8. PMI (2015): Uganda Indoor Residual Spraying (IRS) Project Phase II Revised Statement of Work. 9. PMI (2011): Uganda Malaria Operational Plan for FY 2012. President’s Malaria Initiative. 10. PMI (2012): Uganda Malaria Operational Plan 2013. President’s Malaria Initiative. 11. PMI (2013): Uganda Malaria Operational Plan 2014. President’s Malaria Initiative. 12. USAID-CDC Inter-agency Group (2005): President’s Malaria Initiative Strategic Plan. 13. USAID/CDC (2015): President’s Malaria Initiative Strategy 2015 – 2020. 14. WHO (2015): Indoor Residual Spraying. An operational manual for indoor residual spraying (IRS) for malaria transmission control and elimination. World Health Organization. Geneva. 15. MoH/NMCP (2015): Rapid capacity needs assessment of the national malaria control programme in Uganda. Report by Health and Education Advice and Resource Team (HEART). Kampala, Ministry of Health, National Malaria Control Programme. 16. MoH (2015): Capacity Development Plan for the National Malaria Control Programme. Kampala, Ministry of Health. 17. MoH DHIS-2 (District Health Information System II) 18. Uganda Bureau of Statistics (UBOS) and ICF International (2015): Uganda Malaria Indicator Survey 2014-2015. Key Indicators. Kampala, Uganda and Rockville, Maryland, USA. 74 19. Uganda Bureau of Statistics (UBOS) and ICF International. (2012). Uganda Demographic and Health Survey Kampala: Uganda Burea of Statistics and Macro International Inc. 20. United States Agency for International Development Title 22, Code of Federal Regulation, Part 216 Agency Environmental Procedures 75 ANNEX III: ANNEXES TO SECTION 4.1 (MEETING HIGH QUALITY PROGRAM DESIGN STANDARDS) Annex IIIA: Sample of a map produced by the IRS project using GIS data 76 Annex IIIB: Improvement in accuracy of household enumeration following introduction of geo-coding technology Source: IRS Project Annual Report for 2014 77 Annex IIIC: Copy of supplementary environmental assessment approval from NEMA 78 79 Annex IIID: Planned and actual spray dates during Year 1 and 2 Year 1 (2012-2013) Districts Spray round one Spray round two Planned Dates Actual Dates Remarks Planned Dates Actual Dates Remarks Group A: Gulu, Apac, Nwoya, Kitgum & Pader October 1 - December 7, 2012 October 1 – Oct 30, 2012 Started on time, completed within planned time March 4 - June 7, 2013 April 2 – May 8, 2013 Start date delayed by about a month but completed within planned time Group B: Amuru, Lamwo, Agago, Kole & Oyam October 1 - December 7, 2012 Oct 22 – November 20, 2012 Start delayed by 3 weeks, but completed within planned time March 4 - June 7, 2013 April 22 – May 28, 2013 Start date delayed by about 6 weeks but completed within planned time Year 2 (2013 – 2014 Districts Spray round one Spray round two Planned Dates Actual Dates Remarks Planned Dates Actual Dates Remarks Group A: Gulu, Apac, Nwoya, Kitgum & Pader October 21 - December 7, 2013 October 14 – November 21 Started a week earlier, completed within time March 31 - May 31 , 2014 March 31 – April 30, 2014 Started on time and completed within time Group B: Amuru, Lamwo, Agago, Kole & Oyam November 4 - December 7, 2013 November 4 – December 7, 2013 Started and ended on schedule March 31 - May 31, 2014 April 22 – May 23, 2014 Started 3 weeks after scheduled time, but completed within time Source of data for planned dates: IRS Phase II project Workplans 2012-2013 and 2013-2014 Source of data for actual dates: IRS Phase II project Annual reports 2012-2013 and 2013-2014 80 Annex IIIE: Soak-pits and bath shelter assessments and repairs, January-March, 2014 Source: IRS Phase II project Annual Report 2013-2014 ANNEX IV: ANNEXES TO SECTION 4.2 (BCC INFLUENCE ON MALARIA PREVENTION PRACTICES) Annex IVA: Proportion of pregnant women that received the second dose of IPTp (IPTp2) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Jan to Mar 2012 Apr to Jun 2012 Jul to Sep 2012 Oct to Dec 2012 Jan to Mar 2013 Apr to Jun 2013 Jul to Sep 2013 Oct to Dec 2013 Jan to Mar 2014 Apr to Jun 2014 Jul to Sep 2014 Oct to Dec 2014 Jan to Mar 2015 Apr to Jun 2015 Jul to Sep 2015 Oct to Dec 2015 Jan to Mar 2016 New IRS districts Old IRS districts None IRS districts District Total Soak-pits Bath shelters repaired Good Minor Repairs Major Repairs Relocated Live Fencing No live fence Relocated Gulu 34 0 8 21 5 34 0 10 44 Nwoya 14 0 6 8 0 12 2 0 14 Apac 30 0 12 16 2 30 0 4 56 Pader 31 0 20 9 2 29 2 4 58 Agago 34 0 26 3 5 32 2 10 58 Kitgum 31 0 20 11 0 31 0 0 62 Kole 16 0 9 6 1 16 0 1 32 Oyam 25 0 19 6 0 20 5 0 50 Amuru 27 0 16 11 0 27 0 0 54 Lamwo 31 1 20 8 2 25 6 4 58 Total 273 1 156 99 17 256 17 33 486 81 Annex V: ANNEXES TO SECTION 4.3 (strengthening of structures and partnerships) Annex VA: IRS project II capacity dashboard process and indicators Measuring and monitoring district IRS capacity improvements: The capacity dashboard tool To assess the capacity transferred to the district personnel who are counterparts in IRS implementation, the project developed a capacity building dashboard tool. The tool serves as a ‘temperature gauge’ for ongoing capacity building transfer to relevant district staff in all the project districts. The project uses the capacity dashboard to measure the capacity being transferred to the district authorities to effectively and efficiently manage IRS operations, specifically in the six IRS capacity areas of: entomological competence, environmental compliance, IEC/BCC, logistics, M&E and spray operations. The tool uses indicators for each of the project area which are scored accordingly on a scale of 0 to100 (See Table below). The scores are then filtered through objective analysis of district involvement in each indicator, to produce a visual dashboard that gives a ‘bird’s eye’ view of capacity building progress in each district. The dashboard process begins with a round of baseline data collection, linked to an initial spray round. Baseline data are cleaned and entered into an Excel spreadsheet, and exported into Tableau software which produces visual and tabular analyses. Each subsequent round of dashboard data is added to create a timeline graph measuring the progress of districts as well as program areas. Dashboard data is collected during the pre-spray training period, during spray operations, and two weeks after the completion of each IRS round using several tools and data sources to populate the dashboard. This process currently occurs twice a year to coincide with the project’s semi-annual spraying cycles. The various types of data collected are used to generate capacity scores across six key project areas. Before spraying begins, the project trains a large volunteer work force which carries out various spray operations functions. Trainees include data clerks, sub-county supervisors, parish storekeepers, mobilization workers, and spray operators. Training courses use pre￾and post-tests to establish the initial and end-of-training capacity of the trainees. These test scores are one of the sources of the data entered into dashboard-linked Excel files. Annex VB: Project Technical areas for capacity building and indicators PROJECT AREA INDICATOR Entomological Compliance Conduct appropriate types of bioassays Explain step-by-step pyrethrum spray collections (PSCs) Explain step-by-step wall bioassays Interpret bioassay report information Numbers and frequency of PSCs Environmental Compliance Assess and observe stores and soak pits District conducting trainings - environmental compliance IEC/BCC Community radio talk feedback Conduct trainings - IEC/BCC - community mobilizers Distribute hard copy materials Effective community mobilization Effective community mobilization - interviews 82 PROJECT AREA INDICATOR Logistics Effective timely delivery of supplies Interview - quality of consumables procured Material delivery at parish stores Observation - district inventory management M&E District conducting trainings - M&E Effective progress report generation Effective reconciliation data report Spray Operations Accurately forecast IRS supplies Carry out efficient spraying Conduct trainings - spray operators Conduct trainings – supervisors Forecast human resources Recruit spray team members Source: Capacity-building dashboard tool – December 2015, Uganda IRS Project Phase II, Abt Associates. Annex VC: IRS capacity improvements in the old project districts between April 2013 and Oct 2014 0% 50% 100% 150% 200% 250% E/cpt E/cpl BC Logs M&E SO April 2013 score 42% 22% 28% 23% 34% 48% Oct 2014 Score 62% 76% 42% 23% 48% 51% % change 48% 244% 52% 0% 41% 7% 83 ANNEX VI: ANNEXES TO SECTION 4.4 (MEL AND CLA) Annex VIA: Sample success story documented by the project 84 ANNEX VII: DATA COLLECTION INSTRUMENTS EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 1: HOUSEHOLD QUESTIONNAIRE Introduction and Consent: Hello. My name is _______________________________________. I am working with a research organization named Socio-Economic Data Centre. We are working on behalf of the USAID Monitoring and Evaluation Learning Contract. We are conducting an evaluation of an indoor residual spraying program implemented in Northern and Eastern Uganda to control malaria. The information we collect will help the Ministry of Health, National Malaria Control Program and partners to plan better delivery of the IRS program. Your household was selected for the evaluation and I would like to ask you some questions about your household. The questions usually take about 20 to 30 minutes. We are an independent team and all of the answers you give will be confidential and will not be shared with anyone other than members of our survey team. Participation in the survey is voluntary, and you will not be affected in any way if you choose not to participate. But we hope you will agree to answer the questions since your views are important. If I ask you any question you don't want to answer, just let me know and I will go on to the next question or you can stop the interview at any time. In case you need more information about the survey, you may contact the person listed on this sheet (handover information sheet). Do you have any questions? (Interviewer: Please respond only to the questions where you have information, and then refer the respondent to other authorities) Do you agree to participate in the interview? 1. Yes (proceed with interview) 2. No (Thank the respondent and end the interview) Were you present during the last round of IRS spraying in your house? 1. Yes (Proceed to interview the respondent) 2. No (Ask to see if you can speak to the person who was present when IRS was last done in the house) If respondent agrees to interview, interviewer please sign here _________________________ SECTION A: HOUSEHOLD IDENTIFICATION DETAILS DATE DD_____MM_______ 2016 DISTRICT SUB-COUNTY PARISH VILLAGE TYPE OF RESIDENCE 1. Rural 2. Urban/Peri-urban INTERVIEWER NAME CHECKED BY: 85 SECTION B: SOCIO-DEMOGRAPHIC CHARACTERISTICS OF THE RESPONDENT AND HEAD OF HOUSEHOLD No. Questions Coding Categories Go To 101 (RECORD THE SEX OF RESPONDENT) 1. Male 2. Female 102 How old are you (in completed years)? 103 What is your highest level of Education? (Probe for the highest level of formal education attained) 1. No Education 2. Functional Adult Literacy 3. Primary incomplete 4. Completed Primary 5. Seconday (O level) 6. Secondary (A Level) 7. Tertiary/University 8. Other Specify_________________ 104 Can you read and write in your local language? 1. Yes 2. No 105 Can you read and write in English? 1. Yes 2. No 106 What is your religion? 1. Catholic 2. Protestant 3. Muslim 4. Seventh Day Adventist 5. Orthodox 6. Born Again/Pentecostal 7. Traditionalist 8. Other (specify)_________________ 107 What is your current marital status? 1. Single/Never married 2. Married/Cohabiting 3. Widowed 4. Divorced/Separated 5. Other (specify)________________ 108 What is your relationship to the household head? 1. Self (Head) If 1 Skip to 110 86 No. Questions Coding Categories Go To 2. Spouse 3. Son/daughter 4. Other relative 5. Not related (specify) ____________ 109 Sex of household head 1. Male 2. Female 110 Main occupation of the household head (Only one response needed: probe for the one in which the HH head is engaged most of the time) 1. Farmer 2. Businessman/woman 3. Blick laying/artisan 4. Civil servant 5. Other salaried worker 6. Casual labourer 7. Others (specify) _______________ 111 Is the main house owned by this household or by another person(s)? 1. Owned 2. Rented 3. Borrowed 4. Just caretaking 5. Other owner (specify)___________ 112 In total, how many rooms are in this compound/household? (count rooms in all housing structures in the compound including kitchens, animal houses, etc that belong to this household). _______________ rooms 113 How many of these rooms are used for (read out): a) Sleeping____________________ b) Sitting/entertainment/dining ____ c) Animals___________________ d) Cooking __________________ e) Storage of food ____________ f) Storage of other things _______ 87 No. Questions Coding Categories Go To g) Other (Specify) _______________ Number ___________________ SECTION C: INDOOR RESIDUAL SPRAYING No. QUESTION RESPONSE CATEGORIES SKIP INSTRU CTIONS 201 Has your household ever received Indoor Residual house Spraying (IRS) against malaria transmitting mosquitoes? (IRS is limited to spraying conducted by a government, private, or NGO) 1. Yes 2. No 3. Don’t know If 2 or 3 skip to 301 202 When was the last time this house was sprayed? (If don’t' remember –Probe using month when it was done or whether done alongside other households.) Month ______ Year 20______ 99. Don’t know / can’t remember 203 The last time all houses in this village were sprayed, was this house also sprayed? 1. Yes 2. No If 2, skip to 301 The following questions are experience of the last time your household was sprayed. 204 The last time this house was sprayed (IRS), who did it? 1. Government program officials 2. NGO 3. Private company/person 4. Don’t know 5. Others (Specify)________________ 205 Which sections of this house (houses) were sprayed with the insecticide the last time your house was sprayed? (Multiple responses possible, circle all mentioned and probe for more) 1. All rooms used for sleeping 2. Sitting room/dining room 3. Store rooms for none food items 4. Food storage rooms 5. Windows/doors 6. Animal houses/rooms 7. Ceiling / inside roof 8. Kitchen 9. Others (Specify)__________________ 10. Don’t know 206 Other than the walls, are there household items that were sprayed with 1. Yes If 2 or 3 Skip to 208 88 insecticide the last time this house was sprayed? 2. No 3. Don’t know 207 If yes, which other items were sprayed with the insecticide the last time this house was sprayed? (Multiple responses possible, circle all mentioned and probe for more) 1. Clothes/beddings 2. Utensils e.g. Plates/cups/pans 3. Water containers 4. Wall hangings/pictures/posters 5. Food items/drinks/fruits 6. Toys 7. Furniture 8. Others (Specify)__________________ 208 Are there any rooms that were not sprayed the last time this house was sprayed? 1. Yes 2. No 3. Don’t know If 2 or 3, skip to 210 209 Which rooms were not sprayed the last time this house was sprayed? (Multiple responses possible, circle all mentioned and probe for more) 1. Sitting room / dining room 2. Store rooms for non-food items 3. Store rooms for food items 4. Bed room(s) 5. Animal houses 6. Others (specify)__________________ 210 The last time this house was sprayed, did you receive any information about the spraying before the day of the exercise? 1. Yes 2. No 3. Don’t know If 2 or 3, Skip to 212 211 What information did you receive about the spraying exercise? (Multiple responses possible, circle all mentioned and probe for more) 1. Dangers of malaria 2. Benefits of having the house sprayed 3. The insecticide used not harmful to human life 4. When the spraying would take place 5. Allowing the spray operator into the house 89 6. Avail clean water to spray operator 7. Waiting for 2 hours after spraying before entering the house 8. How to dispose the dead insects 9. Continued use of LLIN after spraying 10. Not plastering/washing the walls for six months 11. Others (Specify)__________________ 212 On the day of spraying, did the spray operator give you any information about the spraying before and/or after the actual spraying? 1. Yes 2. No 3. Don’t know If 2 or 3 Skip to 214 213 What kind of information were you given by the spray operator (Multiple responses possible, circle all mentioned and probe for more) 1. Things/items to remove from house 2. How to manage the items that could not be removed from the house 3. The likely side effects of the insecticide 4. What to do to manage the side effects 5. How to keep the sprayed walls for next 6 months 6. Time to stay outside the house after spraying 7. Malaria control and management 8. How to manage dead rodents / insects 9. Others (specify)__________________ 214 The last time this house was sprayed, did the spray operator tell you the name of the insecticide to be used? 1. Yes 2. No 3. Don’t remember 215 Before the spraying, were you informed that you had to remove items from your house? 1. Yes 2. No 3. Don’t know If 2 or 3, skip to 217 90 216 What items were you advised to remove from the house before the spraying? (Multiple responses possible, circle all mentioned and probe for more) 1. None 2. Beddings 3. Clothes 4. Cooked Food 5. Uncooked food items 6. Utensils 7. Wall hangings and posters 8. Sick people and children 9. Domestic animals 10. Others (specify)__________________ 217 From the time the spray operator finished spraying this house, how long (in hours) did it take you (or any other member of the household) to enter the sprayed house? 1. Less than ½ hour (30 minutes) 2. Less than 1 hour 3. Less than 2 hour 4. After two hours 5. Don’t know 6. Others specify __________________ If 4 or 5 Skip to 219 218 If entered the sprayed house in less than two hours after IRS spraying, what are the reasons? 1. I was told it’s okay 2. I thought it was okay 3. It was threatening to rain/it rained 4. It was getting dark / nighttime 5. There was a sick person 6. Others (Specify)_________________ 219 Did any of domestic animals enter this house before the lapse of 1 hour after spraying? 1. Yes 2. No 3. Don’t know 220 The last time this house was sprayed, were there some insects/rodents that were killed after spraying the insecticide? 1. Yes 2. No 3. Don’t know/don’t remember If 2 or 3, skip to 222 221 If yes, how did you dispose off the dead insects/rodents gathered from this house? 1. Thrown in pit latrine 2. Buried in soil 3. Thrown in garden/bush 91 4. Given to domestic animals 5. Others (specify) _________________ 6. Don’t know 222 The last time this house was sprayed, were you told of any likely side effect of the insecticide to be used? 1. Yes 2. No 3. Don’t know If 2 or 3, skip to 224 223 If yes, what were some of the likely side effects you were told? (Multiple responses possible, circle all mentioned and probe for more) 1. Skin irritation 2. Eye irritation 3. Difficulty in breathing/coughing/running nose 4. General weakness/fatigue 5. Burning mouth/throat 6. Dizziness 7. Headaches 8. Fever 9. Others specify __________________ 224 Were you told what to do in case these side effects are experienced by any member of your household? 1. Yes 2. No 3. Don’t know/don’t remember If 2 or 3 skip to 226 225 If yes, what were you told? 1. Washing body/eyes with clean water 2. Staying out of house for more time 3. Seeking medical advice from health facility/worker 4. Others (specify)___________________ 5. Don’t know 226 Did any member of your household experience any side effect the last time this house was sprayed? 1. Yes 2. No 3. Don’t know If 2 or 3, skip to 228 227 What side effects did they experience? (Multiple responses possible, circle all mentioned and probe for more) 1. Skin irritation 2. Eye irritation 92 3. Difficulty in breathing/coughing/running nose 4. General weakness/fatigue 5. Burning mouth/throat 6. Dizziness 7. Headaches 8. Fever 9. Others (specify) _________________ 228 (Only ask this question if answer in Q224 is Yes) How helpful did you find the advice given on handling the likely side effects? 1. Very helpful 2. Helpful 3. Somehow helpful 4. Unhelpful 5. Very unhelpful 229 May you give an example to explain your opinion on the helpfulness/unhelpfulness of the advice? 230 Are you satisfied with the insecticide used in spraying inside this house the last time your house was sprayed? 1. Highly satisfied 2. Satisfied 3. Indifferent 4. Unsatisfied 5. Highly unsatisfied 231 What are the reasons for your satisfaction / dissatisfaction? (Multiple responses possible, circle all mentioned and probe for more) Reasons for satisfaction 1. Experienced reduced mosquitoes for more than 6 months 2. For some time (< 6 months) there were reduced mosquitoes in the house 3. Reduced side effects were experienced 4. Did not experience malaria for more than 6 months 5. Other (Specify)__________________ 93 Reasons for dissatisfaction 6. Still experienced mosquitoes in house after spraying 7. Still experienced malaria before elapse of 6 months 8. Side effects were experienced after spraying 9. Other (specify) _________________ 232 How satisfied were you with the character and general behavior of the spray operator who sprayed your house? 1. Very satisfied 2. Satisfied 3. Indifferent 4. Dissatisfied 5. Very dissatisfied 233 What are the reasons for your satisfaction/ dissatisfaction with the spray operator? (Multiple responses possible, circle all mentioned and probe for more) Reasons for satisfaction 1. Appropriate time of spraying 2. Spray operator was friendly 3. Gave all required information 4. Answered all our questions/ explained in detail 5. Took good care of household property in the house 6. Others (specify)___________________ Reasons for dissatisfaction 7. Seemed not have time to explain to us the exercise 8. Came at inappropriate time of day to spray 9. Could not listen to or answer our concerns 10. Did not care about household property in the house 11. Others (specify)__________________ 94 234 Did the spray operator/IRS official give you a card to sign after spraying? 1. Yes 2. No 3. Don’t know 235 What did you provide as a household to the spray operator when he/she came to spray this house? (Multiple responses possible, circle all mentioned and probe for more) 1. All the water required 2. Soap 3. Food 4. Nothing 5. Others specify __________________ 236 Would you be willing to have this house sprayed in future? 1. Yes 2. No 3. Don’t know If 1, skip to 238 237 What are the reasons you would not be willing to have this house sprayed in future? (Multiple responses possible, circle all mentioned and probe for more) 1. Fear the side effects of the insecticide 2. Religious/traditional beliefs don’t agree with exercise 3. Have members allergic to chemicals 4. Not wanting chemicals to contaminate my organic crops 5. I do not see the necessity 6. House not owned by me/us 7. Information given was is convincing enough for me to allow spraying of my house 8. Others (specify)__________________ 238 In the last three months, has any member of this household suffered from malaria? 1. Yes 2. No 3. Don’t know 239 Since the last time this house was sprayed with insecticide during IRS program, how often do you or your household members suffer from malaria? 1. Frequently 2. Sometimes 3. Rarely 4. Never at all 5. Other (specify)___________________ 95 240 Considering the time before and after this house was last sprayed, would you say the cases of malaria in your household have_____? (Read the options): 1. Increased 2. Reduced 3. Remained the same 4. Can’t tell the difference Go to 401 Section D: HOUSEHOLDS THAT HAVE NEVER AND THOSE THAT DID NOT RECEIVE INDOOR SPRAYING DURING THE LAST ROUND Interviewer: (Ask only if the answer in Qn. 201 and 203 was No i.e. if the house was never sprayed or if it was not sprayed during the last round) 301 Why was this house not sprayed the last time IRS was done? 1. Spray team did not show up 2. Spray team came when household members were away 3. Was never informed 4. Did not get sufficient information / did not understand details of the spraying 5. Household refused the spray 6. Recently moved / house is new 7. Other (specify)___________________ If 1,2,3,5 skip to 303 302 If the household refused, what were the reasons? (Multiple responses possible, circle all mentioned and probe for more) 1. Feared the side effects of the insecticide 2. Neighbor/friend/relative advised against spraying 3. Religious/traditional beliefs don’t agree with exercise 4. Had sick person in the house 5. Had young baby in house 6. Have members allergic to chemicals 7. Not wanting chemicals to contaminate my organic crops 8. No wanting spraying to contaminate my other crops 9. I did not see the necessity 10. House not owned by me/us 11. Information given was not convincing enough for me to allow spraying of my house 96 12. Others (specify)__________________ 303 Had you received any information about the spraying exercise before it was done in the rest of this village? 1. Yes 2. No 3. Don’t remember 304 What information had you received about the spraying exercise? (Multiple responses possible, circle all mentioned and probe for more) 1. Dangers of malaria 2. Benefits of having the house sprayed 3. The insecticide used not harmful to human life 4. When the spraying would take place 5. Allowing the spray operator into the house 6. Avail clean water to spray operator 7. Waiting for 2 hours after spraying before entering the house 8. How to dispose the dead insects 9. Continued use of LLIN after spraying 10. Not plastering/washing the walls for six months 11. Others (Specify)______________________ ___ 305 Had you received any information about malaria or its prevention/control before spraying was done in the rest of the village? 1. Yes 2. No 3. Don’t remember 306 What information had you received about malaria or its prevention/control? 1. Malaria kills 97 (Multiple responses possible, circle all mentioned and probe for more) 2. Malaria is dangerous to young children and pregnant mothers 3. Malaria is caused by mosquitoes 4. Malaria can be prevented through IRS 5. We should seek early treatment if we think we have malaria 6. We should sleep in a LLIN to avoid malaria 7. Pregnant women should take fansidar anti-malarials even when not sick 8. We should close windows early to avoid malaria 9. We should clear stagnant water, bushes and other breeding places of mosquitoes 10. Other (specify) __________________ 307 Would you be willing to have your house be sprayed in future in case all your concerns are answered by the implementers? 1. Yes 2. No 3. Don’t know Continue to 401 SECTION E: COMMUNITY PERCEPTIONS ON INDOOR RESIDUAL SPRAYING Am going to read for you some statements. We would like to get your opinion on the following statements by stating whether you Agree, Disagree or Indifferent with them Agree Neither agree nor disagree Disagree 401 Even after spraying the house with the insecticide, it is still necessary to sleep in a mosquito net 1 2 3 402 Enough precautions have been taken to ensure that the insecticide does not pose any harm to humans 1 2 3 403 Enough precautions have been taken to ensure that the insecticide does not pose any harm to domestic animals 1 2 3 404 Enough precautions have been taken to ensure that the insecticide does not pose any harm to soils and water bodies 1 2 3 405 Enough precautions have been taken to ensure that the insecticide does not pose any harm to crops 1 2 3 406 The insecticide used to spray against malaria carrying mosquitoes can make people sick/die (even if they don’t drink it in huge amounts) 1 2 3 407 The insecticide used to spray against mosquitoes can make men impotent or lose their sexual drive 1 2 3 408 The insecticide used to spray against mosquitoes can make women barren / infertile 1 2 3 409 The insecticide used to spray against mosquitoes can negatively affect crops 1 2 3 98 410 The insecticide used to spray against mosquitoes can negatively affect soils 1 2 3 411 The insecticide used to spray against mosquitoes can harm domestic animals 1 2 3 412 Spraying against mosquitoes is against my religion/religious beliefs 1 2 3 413 There is no need for spraying to kill mosquitoes since malaria can be prevented through other means 1 2 3 414 There is no need for indoor residual spraying since malaria can be treated if one gets it 1 2 3 415 Spraying against mosquitoes is useless because mosquitoes do not die 1 2 3 99 SECTION F: EXPOSURE TO IEC/BCC MESSAGES Interviewer, please say: Am now going to tell you some of the information messages that could have been disseminated before, during or after the spraying exercise last time. I will then ask you a few questions. I hope that is okay with you. 500. Messages regarding malaria did you see/hear? Read to respondent) 501 Did you hear/see this message? 1. Yes 2. No (If No, move to the next message) 502 Where did you see/hear the message 1. Radio 2. TV 3. Poster 4. Brochure 5. Worship center 6. Community meeting 7. VHT/Health worker 8. Others (Specify) 503 Did you have challenges in accessing the message through this channel? 1. No challenge 2. Time of day not appropriate 3. Costly to access channel 4. No full time access 5. Little content 6. Understandable Language 7 Others (specify) 504 Radio 1: Voice of Lango 2: Unity FM 3: Dokolo FM 4: Delta FM 5. Etop FM 6 Aisa FM 7. Bugwere FM 8. Open Gate FM 9. Step FM 10 NBS FM 11. Eastern Voice 12. Rock Mambo 13. Big FM 14. Others 505 Language Radio 1.English 2.Langi 3.Acholi 4.Ateso 5.Lugwere 6.Lusoga 7.Luganda 8.Lunyore 9.Japhodola 10. Kuma 11. Others 506 Time Radio 1) Before 6 Am 2).6Am -8 Am 3)9Am-Noon 4)1Pm -5Pm 5)6Pm-8Pm 6)9Pm-Midnight A Malaria Prevention messages A1 Use of indoor spraying to kill/repel mosquitoes A2 Need for pregnant women and children < 5 years to sleep under LLIN A3 All people should continue using LLINs even after spraying A4 Pregnant women should swallow anti-malarial medicine at the health centre even if not sick B Malaria Treatment 100 B1 Seek for malaria treatment for pregnant women and < 5 years children within the first 24 hours after noticing symptoms B2 All people should seek malaria treatment promptly within 24 hours after noticing symptoms 101 507 Has the message you have heard/received on malaria prevention and/or treatment influenced in any way your household level practices in relation to malaria? 1. Yes 2. No 3. Don’t know If 2 or 3 skip to 509 508 If yes, what kind of practices have you adopted? (Multiple responses possible, circle all mentioned and probe for more) 1. Pregnant woman/women sleep under LLIN 2. All children under 5 years sleep under LLIN 3. All household members sleep under LLIN 4. Seek early treatment for malaria 5. Clear bushes and mosquito breeding areas around the house 6. Allowed IRS team to spray my house 7. Close windows before dark 8. Adherence to treatment for those under malaria treatment 9. Other (Specify) ___________________ 509 How often did you hear/receive message about IRS? 1. Never received any message on IRS 2. Many times a day 3. Once a day 4. Twice or more times a week 5. Once a week 6. Few times a month 7. Few times in last 6 months 8. Others (Specify) ________________ If 1, skip to 511 510 How satisfied were you with the content of the message delivered to you? 1. Very satisfied 2. Satisfied 3. Just enough 4. Un satisfied 5. Very unsatisfied 511 Through what other channels would you like to receive messages on indoor spraying and malaria? 102 512 (If radio mentioned), what time do you normally listen / tune in to the radio? 1) Before 6 Am 2) 6Am -8 Am 3) 9Am-Noon 4) 1Pm -5Pm 5) 6Pm-8Pm 6) 9Pm-Midnight 7) Others (Specify)______________________ Comments about the respondent Comments about specific questions END THANK YOU 103 EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 2A: INTERVIEW GUIDE FOR IRS II PROJECT MANAGEMENT, CAPACITY BUILDING AND M&E STAFF Record name and position of informant; date of interview 1. What structures were put in place to ensure that the IRS II project is successfully implemented? (Probe for Office space, human resource, organization structure, storage facilities logistics procurement, supply and management plan, commodities from outside Uganda always get into the country in good time, waste disposal) at both the national and district level. a. What problems/limitations were encountered while setting up the above? 2. What was done to ensure that the different structures provide the services as required? Probe if training was provided, who organized the training, how long was the training, was there a curriculum for each category of staff) 3. What kind of logistics was provided to ensure adequate implementation of the project? (Probe for the gaps in the resources) 4. Who were the different stakeholders involved in the planning and implementation of the IRS project at (a) national level (b) district level? Probe for the role of each of the different stakeholders mentioned 5. What are the different guidelines and regulations followed for the selection of the insecticides for the IRS? Were these guidelines followed? a. To what extent has the project implementation adhered to WHO standards, GoU and USAID guidelines for environmental safety? (Probe for any aspects that were not complied with and the reasons for the non-compliance. Structures, Partnerships, Capacity building and Sustainability 6. What innovative approaches in terms of capacity building, strategic partnerships were implemented by the IRS program (probe for the types of capacity building approaches conducted i.e. training, mentorship etc., for the district and national staff, the types of partnerships established at the district and national level, availability of skilled staff, equipment and other resources to implement IRS, Extent of use of existing structures). 7. What results have been realized from the capacity building and partnership development interventions during the implementation of the IRS project? (Probe for any reported 104 improvements in knowledge, skills, relationships, other types of value additions to the implementers, quality/strength of coordination mechanisms) a. Before the start of the project, how would you rate the NMCP capacity to implement its IRS strategy in Uganda? b. What were the specific challenges NMCP faced in relation to its capacity to implement the IRS strategy in Uganda? c. How has the project improved the NMCP capacity to plan and implement IRS? d. What specific steps/activities has the project taken to improve NMCP human resource capacity in logistic planning; in environmental compliance, in entomological monitoring;? Have these led to an improvement? e. Has the project contributed to training materials, to national policy documents, to research, to development of BCC materials? 8. What collaborations did the IRS make while implementing the project? (Probe for the collaborations made with the civil society organizations (CSOs) international universities, research institutes and private sector agencies (PSOs) in implementing IRS) What was the relevance of each of the collaborations made in the implementation of the IRS project? 9. Is there a phase out plan for the IRS project? Who was involved in the design of the phase out plan and what was the role for each of the stakeholders? Probe for the existence of a documented exit plan? a. What is the content of the exit plan? b. Do the phase-out procedures include skills transfer? In your opinion, what do you think about GoU preparedness to roll out and continue funding and implementing IRS activities? 10. What future activities can the IRS Project contribute to, in order to help sustain malaria program achievements to date? Monitoring, Evaluation, Learning and Adaptation 11. How has CLA been applied in this program? Probe for a. how CLA was defined, and used in the IRS project, b. Any evidence of the results of CLA application, c. Available evidence of the application of CLA in the IRS project? d. Availability of a section of the CLA included in the Activity Monitoring and Evaluation Learning Plan? e. A learning agenda for the IRS project? f. How was the CLA applied and operationalized by the IRS project? g. 12. What are the key lessons learned by the IRS Project? h. How have these lessons been documented? Have these lessons been shared with the stakeholders? Give details i. How have such lessons been used to benefit the IRS Project and its partners? j. What specific decisions and adaptations have been made as a result of learning from your work? 105 k. Please tell me what you feel are the key achievements of the Uganda IRS project to date and to what extent do you think the objectives have been achieved to date and will be achieved by the end of the project. [Probe on this question, to elicit in-depth response) 13. Describe the MEL system established to implement the IRS activity? (Probe for i. Organizational governance/leadership ii. MEL Structures and Functions iii. Link with National M&E systems and other sources of data used by the project iv. Existence of an approved Activity Monitoring, Evaluation and Learning Plan (AMELP), (progress against individual indicators - milestones, process indicators. Why was the PMP updated in August 2015? v. CLA Approach, Learning Plan, Learning Activities vi. Data Management Systems vii. Data Collection and Reporting (process whereby progress against targets and indicators is measured) viii. Data Quality Control, ix. Data Storage and Management x. Data Analysis, Dissemination and Use (the degree to which sex-disaggregated data are collected (for which program indicators, whether these data are program-specific and/or routine HMIS, survey, etc) and used to inform planning) xi. Documentation of Standard Operating Procedures (SOP) for data handling (data collection, data storage, data quality control, data aggregation, analysis, confidentiality, reporting & dissemination) xii. Access and Use of SOPs xiii. How evidence-based is the Uganda IRS project 11? Stakeholder perceptions 14. In your opinion, what has been people’s response to the IRS programme? Probe for; a. Measures that were put in place to ensure that the IRS project is accepted in the communities b. How has the project dealt with resistance to IRS in the different districts of operation? 15. What was the role of gender dynamics in the success of the IRS campaign? 106 16. In your opinion, what could have caused the malaria that we are noticing in the northern region of Uganda? Probe; a. What lessons do we learn at (household, district, community and national levels) from the post-IRS experiences in Northern Uganda? What should be done differently? b. Do you think the community was well prepared for the IRS project exit in the different districts of IRS implementation? 17. If a follow on IRS project was to be designed, in your opinion, what do you think can be done differently in the design and implementation of the project? 107 EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT ANNEX 3: INTERVIEW GUIDE FOR BCC STAFF Record name and position of Informant; date of interview 1. What BCC activities were implemented? (Probe for when the activities started, availability of human resources, staff, training for staff, length of the training, curriculum used and any other required logistics) a. Discuss the role of CDFU and the districts in the implementation of the BCC IRS project. Activities (Probe for the relationship in the implementation of the BCC activities. How did CDFU address the issue of resistance to IRS? 2. What BCC strategy was used for the implementation of the IRS project? (Probe for who developed, when developed, was the strategy adapted and implemented as proposed? What contextual shifts happened? 3. Was the BCC campaign designed and implemented as guided by the BCC strategy? Probe for the reasons why/ why not the campaign was not implemented as planned/ designed, and how/whether the BCC campaign was adapted, contextual shifts that happened), existence of a BCC plan, the content of the implementation plan, the campaign adhered to original plan 4. What were the key messages for the campaign? (probe for the key messages and support points adopted with respect to i. IRS ii. Other malaria prevention and control measures (probe for the role of MOH, districts in the development of the messages and materials, the target group for the messages) a) Probe for whether the campaign messages and materials were pre-tested with the intended audience before they were used in the campaign, the language used to pretest the messages and materials, the target group with whom the materials and messages were pretested. 5. What communication channels were used to disseminate the BCC messages? Probe for the basis for the selection of the BCC communication channels used during the implementation of the IRS phase II project, the timing and frequency for disseminating the BCC messages) 6. What other BCC materials were used for the implementation of the BCC during the IRS phase II project? 108 7. In your opinion, how have the different BCC interventions contributed to the desired malaria prevention behavior? (Probe for the different behavior changes that you have observed in the implementation of the IRS phase II project) 8. What challenges have you experienced during the implementation of the BCC IRS activities? 9. What more needs to be done to improve the uptake of IRS? EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 4: KEY INFORMANT GUIDE FOR DISTRICT AND S/COUNTY TECHNICAL & POLITICAL LEADERS AND CSOS/PSO STAFF [DHO, DHE, Malaria Focal Person, Environment Officer, Information Officer, CDO, District Planner, CAO, C/Man LCV, RDC, S/County Chiefs, C/Man LCIII, Health Assistants] Record Name and Position of Informant 1. What was/is your role in the implementation of the IRS project? a. Did you receive any training/orientation regarding your role in the project? Who organized the training? How long was the training? What did the training cover? b. Apart from training, what other forms of capacity building did you receive (as a district, department, sub-county, individual) c. In what ways have you utilized the capacity acquired? How useful was these capacity building processes? 2. Overall what is you view about the IRS program, was it necessary, useful? a. What has been the people’s response to the IRS program? b. What are some of the reasons you think some people did not want their houses to be sprayed? c. As leaders/technical people, how did you address this challenge? d. What have you done to increase the acceptability of the program? 3. What BCC /IEC activities have been implemented regarding IRS and malaria control in general? a. How useful has been the BCC activities? b. Were there particular areas/concerns that you think were not addressed in the BCC campaign? 4. Tell me your view of the relationship between the district of and the Uganda IRS project. (Probe in depth on issues such as: 109 a. how does the project work with the district, b. whether and how each supports the other, c. if there is effective joint activity planning and delivery, d. if there is effective responsiveness from each side, e. whether there are any challenges and what you feel are the main achievements of any joint planning, working and support). 5. To what extent were/have you been involved in the planning activities for the project? a. To what extent were you involved in the selection of the insecticide, equipment and accessories to be used in the various geographic areas targeted for spraying 6. To what extent did the implementers (Abt Associates) deploy adequate human resources, office space, storage facilities, waste disposal logistics, skills/ training etc to implement the program? a. What were the gaps in these resources and logistics? 7. How was the program implemented? a. Did the different stakeholders adequately play their roles? b. How have you worked with CSOs and PSOs in implementing IRS? Which CSOs/PSOs? c. How useful have been these partnerships? d. What are the gaps in these partnerships, and what needs to be done to strengthen them? e. What relationships are still useful? 8. Were you involved in the monitoring and evaluation of the different IRS program activities performed? a. What are the different monitoring activities you were involved in (Probe for monitoring of program activities, environmental compliance and vector surveillance) b. To what extent did the IRS program adhere to existing Government and WHO guidelines and standards? c. Please explain where the project did not adhere to plans and guidelines d. In case of non-adherence, what were the reasons? 9. In general, would you say the program was successful? a. Would you say the IRS campaign helped to reduce the malaria problem in this community? Please explain why you think so. b. What was malaria situation like in your community before the indoor residual spraying took place? What was the situation like after the spraying? c. What are the reasons for success/ no success? d. Apart from spraying what are people in your community doing to prevent malaria? e. To what extent are you satisfied with the performance of the program (rank on a scale of 0- 5). 110 f. What, in your opinion, could have been the role of gender dynamics in the success of the IRS campaign? 10. What challenges did you face during your involvement in the program? 11. What lessons have been learnt from the implementation of the IRS program? a. Have these lessons been discussed in your organization? b. What do you think could be done differently to help communities understand IRS better? 12. Overall, how prepared is your local government to continue implementing IRS? a. As a district/sub-county, what capacity have you acquired to continue implementing IRS? b. How can capacity of local governments be enhanced to advocate and implement further for IRS after the project closes? 13. There has been an upsurge in malaria cases in Northern Uganda following the end of the IRS project. What, in your opinion, could have caused this? What lessons can we learn from the post￾IRS experiences in Northern Uganda? What should be done about the current epidemic? Ask CSOs/PSOs Only 1. How has your organization been involved in the IRS project? a. When did your organization start being involved? b. What was the motivation for your organization to be involved in IRS activities? c. How have you personally been involved? d. What resources did your organization contribute? 2. Did you/ your organization receive any training related to IRS? a. What did the training entail? b. What knowledge/skills did you acquire from this training? c. Apart from training, what other forms of capacity building did you receive? d. How have you used the skills and knowledge acquired? 3. How was the program implemented? a. What kind of structures and partnerships existed? b. What were the opportunities for dialogue with and feedback from community members about progress and barriers to communication about IRS? c. How effective were these opportunities? 111 4. How useful has been your partnerships with NMCP, Abt Associates, districts and other actors in IRS work? a. What gaps still exist in the existing partnerships? b. What partnerships are still useful? c. What needs to be strengthened? What should be done? 5. Did the BCC activities reach the intended audiences? If not, why? a. What was the impact of BCC activities on knowledge, attitudes, and practices of the community? b. What was the community’s perception of the messages? 6. How interested and prepared is your organization to continue implementing IRS in future? a. What would it take for you to continue implementing IRS/participating in IRS work? 7. What recommendations do you want to make to improve future IRS activities? EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 5: KEY INFORMANT GUIDE FOR HEALTH FACILITY IN-CHARGES AND STAFF Name Health Facility Cadre 1. What is the current status of malaria as a cause of out-patient and in-patient cases at your facility? a. Has your facility noted change in the number of cases of malaria morbidity or mortality after IRS rounds? Is it improving, deteriorating or unchanged? Are more or fewer men, women and children presenting? b. If no improvement what are the main obstacles to achieving a good level of control? c. How has this situation changed over the last two to three years? d. What do you attribute the changes observed in the malaria patterns noted above? Probe for the role of BCC, IRS, ITN use to the changes in the malaria cases noted above 112 2. What is the content of the health talks provided to the patients at this health facility? Probe for any other malaria prevention supplies that are provided to the patients. 3. What do you know about indoor residual spraying (IRS)? Has there been any spaying in this communities this health facility serves in the last 1-3 years? Probe: Were you trained in IRS? What was the role of the health facility, was there any training? 4. Is IPTp (SP fansidar) provided to pregnant women at this health facility? a. Is SP fansidar always available or are there incidents of stock outs? b. Comment on the uptake/acceptance of IPTp at this health facility c. How has this changed over the last three years? d. What could be the reasons for these changes? 5. Overall do you think indoor residual spraying has helped to reduce malaria in this community? 6. What suggestions do you make to improve the prevention and control of malaria? 113 EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 6: FGD GUIDE FOR COMMUNITY LEVEL STAKEHOLDERS (COMMUNITY MEMBERS, COMMUNITY LEADERS, VHTS, PREGNANT WOMEN) Record sex and position of participants; date and venue of FGD 1. What is the extent of the problem of malaria in this community? Probe for: a) Is malaria prevalence currently high or low? b) Have there been any change in malaria prevalence as compared to about 4 years ago? (For Old districts [Gulu, Amuru, Kitgum, Apac], also ask about Malaria in the last months of 2014 i.e. when the project had just phased out) c) Reasons for the observed/perceived changes in malaria prevalence 2. Have you heard about and or participated in the Indoor Residual Spraying program in this community? Probe for: a) What do you know about the IRS (Probe: benefits and problems of IRS, precautions, role of community etc) b) What was the general reception of the community towards indoor spraying? Are there particular groups that resisted the spraying? c) What major reasons have been given by households that refused the spraying? d) What do you think about those reasons/concerns? 3. How were the community members and leaders involved with IRS in this community? Probe: a) Role played by community members in IRS b) Role played by cultural/political/religious leaders in IRS c) What preparations (e.g. training, sensitization etc.) was given to those involved? Was this preparation adequate? 114 4. What have been the benefits of spraying houses? 5. Have there been any problems arising from the spraying? Please explain. 6. What other challenges were faced during the spraying exercise? Probe: a. The insecticide used, conduct of the spray operators, making the house ready for spraying for some people; availing water and any other requirement; b. Any unmet expectations (e.g. need to use LLIN, Not killing all mosquitoes, continued malaria cases etc.) 7. Apart from spraying, what else are the community members doing to prevent and manage malaria? Probe: a. Do people still use nets after spraying? b. Do pregnant women swallow Fansidar when they go for antenatal care? Is it available at health centres? Do they accept to take it? c. Are there any changes in methods of prevention and management of malaria as a result of IRS IRS SBCC campaign – Ask community members only 8. Did you receive any message or information about IRS and other malaria control measures before or during the spraying? Probe: a) The kind of information received on IRS and malaria control b) Channels through which message was received and their appropriateness c) Comprehensiveness of information in answering all issues about IRS d) How information has helped in acceptance of IRS e) How the information has helped in changing practices on malaria prevention and management of malaria Ask Leaders and VHTs only 9. What BCC /IEC activities have been implemented regarding IRS and malaria control in general? a. Do you think the BCC activities reached the intended audiences? If not, why? 115 b. How useful were the BCC activities during the implementation of the IRS project? c. What was the community’s perception/response to the messages? d. Were there particular issues during the IRS BCC campaign that you think were not addressed? 10. In your opinion what do you think is the impact of BCC activities in terms of knowledge, attitudes, and practices of the community? Achievements and Suggestions – Ask all 11. In general, would you say the IRS campaign helped to reduce the malaria problem in this community? Please explain why you think so. 12. Would you be willing to have your houses sprayed again? Why/ Why not? a. Considering your experiences with the spraying, would you be willing to pay/contribute some money to have your house sprayed? 13. If the IRS campaign were to be redone, what would you advise the project staff to do to improve the success of the interventions? EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT ANNEX 7A: INTERVIEW GUIDE FOR SPRAY OPERATORS A Sex of respondent E District B Age of respondent F Sub￾county C Date H Parish D Name of Interviewer I Village 1. Can you please tell me the number of times Abt Associates IRS Project has recruited you as a spray operator? 116 2. Can you tell me the process Abt Associates IRS Project used to recruit you as a spray operator? a. Did you under-go a medical test/examination before being hired as a spray operator? 1. Yes 2. No 3. What are your roles as a spray operator? 4. Did you receive training for this role? 1. Yes 2. No a. For how many days did Abt Associates Project train you to be able to spray? ________ days b. What are the knowledge and skills you gained from the training? 5. Can you name personal protective equipment which you received to use during the implementation of the IRS project? (Please read out the list and circle as appropriate) Yes No Comments if any 1 Gumboots 1 2 2 Overalls 1 2 3 Helmet and Face shield 1 2 4 Mouth/nose masks 1 2 5 Heavy duty short gloves 1 2 6 Other (specify) 1 2 6. Are there other protective equipment which you needed but never received? Please mention them. 117 7. What other personal measures were you told to do and not to do to prevent insecticide contamination before, during and after? Mentioned unprompted Mentioned after Prompting Not mentioned 1 Wear protective equipment and clothing 1 2 3 2 Wash hands, face, neck with soap and water after spraying and before eating, smoking or drinking 1 2 3 3 If spill has occurred, restrict access and cover spill with earth or sand 1 2 3 4 If clothes are contaminated, remove them and wash with water and soap 1 2 3 5 Change personal protective equipment (PPE) daily 6 Wash soiled overalls daily 1 2 3 7 Wash all spray equipment at the soak pit 1 2 3 8 Dispose of water used to wash contaminated clothes and equipment in the soak pit 1 2 3 9 Ensure all people move out of the house before spraying 1 2 3 10 Ensure food and other movable property and removed from the house and the immovable are covered before spraying. 1 2 3 11 Inform people of safety procedures e.g. not to move into the house that has been sprayed until after 2 hours 1 2 3 12 Never use streams, rivers or public tap stands for bathing, washing clothing worn during spraying or washing IRS equipment 1 2 3 8. Were you always able to practice these measures? 1. Yes 2. No a. If not which ones were you not able to practice? b. What were the reasons? 9. What equipment and materials are you required to go with to the field when going to spray? 118 Mentioned unprompted Mentioned after Prompting Not mentioned A Insecticide 1 2 3 B Spray pump 1 2 3 C Hava sack 1 2 3 D Masks 1 2 3 E Polythene sheet 1 2 3 F Masks 1 2 3 G Cloth filter 1 2 3 H Spare parts 1 2 3 I Pliers and spanners 1 2 3 J Spray card 1 2 3 10. Did you always have these equipment and materials whenever you went spraying? 1. Yes 2.No a. If not what were the reasons? 11. Can you tell me the information you gave to household members before spraying? a. What information do you provide to the household after spraying? 12. Were the community members happy to have their houses sprayed? 1.Yes 2. No a. Why were they happy or not happy? b. For those who could have refused the spraying, what were their reasons? c. What are the beneficiaries saying about the Abt Associates IRS Project after spraying? 13. Can you describe the roles and responsibilities the community members play in spraying activities? 119 a. Did people actually play those roles? b. How could this participation be improved to support the spraying activities? 14. Do you think spraying houses is useful and should be continued this area? 15. Did you fill any forms about the houses sprayed? a. What information did you record on the spray operator’s form? b. How did you account for the insecticide sachets you received? 16. What challenges do you face in your work as a spray operator? a. In your opinion how should these challenges be addressed? 17. Overall to what extent would you say you were able to follow/adhere to all the required procedures and guidelines governing your work? Would you say, you followed them fully, partially or only to a small extent? 1. Fullly 2. Partially 3. To a small extent 4. Not at all a. If not fully, please explain what was not fully adhered to b. If not fully what were the reasons? End, Thank You. 120 EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 7B: INTERVIEW GUIDE FOR STORES STAFF AND OBSERVATION AT IRS STORES AND SITES Village____________________________ Parish _______________________ S/county __________________________ District _______________________ Date ____________________ Store Code _____________________ Age of respondent _______________ Sex 1. Male 2. Female _________ 1. What are your specific responsibilities as a store keeper? 2. Were you trained on the different roles and responsibilities that you provide to the project? 1. Yes 2. No d. How long was the training? __________________ days 3. Did you undergo a medical test/examination before being hired as a store keeper? 1.Yes 2.No 4. Why was this place selected as a site for the store? 5. What commodities do you keep in your store? Probe: Is the store used only for IRS related materials or other things as well? 121 6. What rules/guidelines guide the operation of your store? (probe for how the records are stored, use of a bin card, store ledger, vouchers or computer in their different storage facilities.) 7. How do you keep your stores records? 8. Do you use a bin card, store ledger, vouchers or computer in your storage facility? Yes No Comment Bin card 1 2 Store Ledger 1 2 Vouchers 1 2 Insecticide tracking form 1 2 Computer 1 2 Phone transmission of data to Abt. Associates 1 2 9. What are the purposes of stores records? 10. Describe the mechanism used to deliver the commodities to your store? a. Is it efficient/ delivery is done in time? b. If not, why were the deliveries delayed? c. Were adequate materials delivered to you? Please explain what was not adequate? d. Were they in good condition? 11. Describe the method and process you use to distribute commodities from your store to the users of these commodities. 122 12. How do you monitor the usage of insecticides as a store keeper? a. Describe stock reconciliation (balancing) and its purpose. b. How often do you do stock reconciliation? 13. Describe the management of insecticides at the site store? 14. How do you ensure that the pumps are in a good working condition? 15. What personal measures were you told to do and not to do to prevent insecticide contamination? (Please circle the appropriate response) Mentioned unprompted Prompted Not mentioned 1 Wear protective equipment and clothing 1 2 3 2 Wash hands with soap and water after handling insecticides before eating, smoking or drinking 1 2 3 3 Handling spillage 1 2 3 4 If clothes are contaminated, remove them and wash with water and soap 1 2 3 5 Wash soiled overalls daily 1 2 3 6 Dispose of water used to wash contaminated clothes and equipment in the soak pit 1 2 3 7 Never use streams, rivers or public tap stands for bathing, washing protective wear or spraying equipment 1 2 3 e. What are the different personal protective equipment you received to use during the implementation of the IRS project? (Please read out the list and circle as appropriate) 123 Yes No Gumboots 1 2 Overalls 1 2 Helmet and Face shield 1 2 Mouth/nose masks 1 2 Heavy duty short gloves 1 2 Others (specify) 1 2 16. How do you account for the insecticide sachets or other materials and supplies for which you are responsible? (Probe for any forms filled, how often this is done? 17. Are you supervised? 1.Yes 2.No a. Who supervises you? b. How often are you supervised? c. Check visitors book and record accordingly. 18. Are you satisfied with your work? 1.very satisfied 2.satisfied 3.Not satisfied 19. What are the different challenges that you face while performing your services? ----------------------------------------------------------------------------------------------------------------------------------- a. How can these challenges be addressed? b. Any other suggestions to improve your work and the success of IRS? 124 OBSERVATION CHECKLIST AT IRS SITES AND STORES (PLEASE CIRCLE ACCORDINGLY) Items to observe Available/ appropriate Not available/ appropriate Comments 1 Storage facility: a. Double dook 1 2 b. 24-hour guard 1 2 c. Windows barred 1 2 d. Secured doors 1 2 e. Danger warning signs 1 2 f. Fire extinguisher/Sand bucket 1 2 g. Position of fire extinguisher/sand bucket 1 2 h. Pallets 1 2 i. Location of facility 1 2 j. Labeled containers for used sachets 1 2 k. Liquid stored above dry insecticides 1 2 l. Emergency spill kit 1 2 m. Fully stocked 1 st aid kit 1 2 n. Available PPE 1 2 o. Well maintained concrete floor 1 2 p. Roof well maintained 1 2 q. Well ventilation system 1 2 r. Soap and water 1 2 s. Storekeeper wearing appropriate PPE 1 2 t. No food stuffs in store 1 2 2 Stock management: a. System of recording stocks 1 2 b. Up to date cards 1 2 c. Adequate filling system 1 2 d. Orderly shelved stocks 1 2 e. Properly labeled insecticides 1 2 f. Insecticide ordered in a 1st in, 1st out system 1 2 g. Expired insecticides 1 2 h. Labeled containers for waste 1 2 125 3 Transport a. Vehicle spill kit 1 2 b. Emergency 1st aid kit 1 2 c. Certified vehicle 1 2 d. Trained drivers in safety 1 2 e. Sign of pesticide leakage in trucks 1 2 f. Driver wearing appropriate PPE 1 2 g. Spray operators wearing full PPE 1 2 4 Liquid waste disposal a. Wash site for spray operators 1 2 b. Soak pits for progressive rinse method 1 2 c. Wash site for washing PPE 1 2 d. Ablution blocks 1 2 e. Wash sites fenced 1 2 f. Location of soak pit – steep slope or uphill 1 2 g. Location of soak pit at least 10m from residences and gardens 1 2 h. Warning signs at soak pit 1 2 5 Solid waste disposal a. Container for contaminated waste 1 2 b. Container for uncontaminated waste 1 2 c. Disposal guidelines 1 2 d. System of disposal solid waste 1 2 e. Incinerator 1 2 EVALUATION OF THE UGANDA INDOOR RESIDUAL SPRAYING PHASE II PROJECT TOOL 8: KEY INFORMANT GUIDE FOR PMI/USAID STAFF Record name and position of informant; duration in this position, date and place of interview, Note taker/recorder, interviewer Project design and implementation 1. What specific issues/factors informed the design of the Uganda IRS phase II project? Have there been any changes in the design since 2012? 126 2. How did PMI/USAID staff support the IRS project? Probe for preliminary assessments, selection of the insecticide, equipment and accessories, planning, training and operations to be used in the various geographic areas targeted for spraying? Overall assessment of IRS 3. What have been the notable achievements of the IRS project? Also probe for the shortcomings a. What would you say has been the impact of the IRS project on malaria prevention and control in Uganda? b. Please tell me what you feel are the key achievements of the IRS project to date and to what extent do you think the objectives have been achieved to date and will be achieved by the end of the project. [Probe on this question, to elicit in-depth response c. What have been the strengths of IRS (organization of the project, management of the project, strategies adopted by the project, external factors) that contributed to these successes? d. What have been the shortcomings of IRS project that constrained it from being more effective? e. On a scale of 0-5 to what extent are you satisfied with the performance of the project? Capacity of the NMCP 4. What structure have been there for IRS work at national level? a. How effectively were the TWGs serving the purpose of IRS? b. Should there have been a coordination committee? c. What was the conceptualization of the partnerships? 5. How has the capacity of NMCP been strengthened to manage and implement IRS? a. Do you feel the project has contributed to the sustainability of malaria prevention and control interventions in terms of national, district and community capacity? (Probe on NMCP capacity development, planning focus, capacity to deliver IRS, gender-sensitive approaches to malaria prevention, BCC capacity, M&E including entomological and environmental monitoring). b. How can MoH/NMCP be engaged more? Sustainability and Follow-on to IRS 6. What is your opinion about capacity of the GoU/MOH/NMCP and district to continue implementing IRS? a. Tech capacity – can the MoH/NMCP do IRS on their own? b. Financial capacity – government commitment - has it been discussed? c. What future activities can PMI contribute to, in order to help sustain IRS achievements to date? d. Capacity of PSOs/NGOs to do IRS 7. Please tell us something about plans and possible designs for any projects that will follow on IRS? This will help us to focus our recommendations. For how many more years do you expect total PMI funding for Uganda to be maintained at roughly the present levels? 127 Challenges and constraints 8. There has been an upsurge in malaria cases in Northern Uganda. What lessons can we learn from the post-IRS experiences in Northern Uganda? How can the same be avoided for the current project districts? Recommendations/suggestions 9. Any recommendations/suggestions 128 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Denis Muhangi Title Dr Organization Socio-Economic Data Centre Ltd Evaluation Position? Team Leader Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date October 8, 2016 x x x 129 U.S. Agency for International Development Ggaba Road, P.O.Box 7856 Kampala, Uganda