APRIL 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the evaluation team from Tropical Health LLP in conjunction with the USAID/Uganda Monitoring, Evaluation and Learning Program implemented by the QED Group LLC under contract no. AID–617-C-13-00007. EVALUATION (PHASE II) OF THE UNIVERSAL COVERAGE CAMPAIGN FOR LONG-LASTING INSECTICIDAL NETS IN UGANDA 2 Evaluation (Phase II) of the Universal Coverage Campaign for Long-Lasting Insecticidal Nets in Uganda ASSESSING EFFECTIVENESS, EFFICIENCY AND IMPACT OF THE MASS LLIN DISTRIBUTION DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 ACKNOWLEDGEMENTS This evaluation report was supported with funding from UK Department for International Development and US Government’s Presidential Malaria Initiative. The evaluation team is grateful for support in compiling critical data elements that were essential for this assessment. In particular, the team would like to thank staff from the USAID Monitoring, Evaluation and Learning Program who served as the primary liaison with USAID and provided logistical and technical support as well as valuable comments. The evaluation team would especially like to thank Ministry of Health staff at the National Malaria Control Program and leadership from the Universal Coverage Campaign’s main implementing partner, Malaria Consortium, both of whom were instrumental in ensuring access to necessary information. Other implementing partners included staff from World Vision International/Uganda. Community beneficiaries and key informants at the district level added valuable insights to preliminary quantitative data from the evaluation. Cover photo: Caroline Vanderick 4 CONTENTS Acronyms ...................................................................................................................................................................... 7 Executive Summary....................................................................................................................................................... 8 Background..................................................................................................................................................................13 Malaria Control with Insecticide Treated Nets in Uganda ......................................................................................13 The Universal LLIN Coverage Campaign 2012-2014 ............................................................................................13 The Theory of Change Underlying the UCC ..........................................................................................................14 Evaluation Purpose and QuEstions..............................................................................................................................15 Evaluation Design and Methods..................................................................................................................................17 Quantitative Methods..............................................................................................................................................17 Data Preparation and Analysis................................................................................................................................18 Qualitative Methods................................................................................................................................................20 Limitations of Data and Mitigation .........................................................................................................................21 Findings, Discussion & Conclusions...........................................................................................................................22 I. Quality of the UCC Registration and Distribution Data Findings .......................................................................22 Findings ..............................................................................................................................................................22 Discussion...........................................................................................................................................................26 Conclusions.........................................................................................................................................................27 II. Quality and Targeting of Behavioral Change Communication and its Effects on Net Use ................................28 Findings ..............................................................................................................................................................28 Discussion...........................................................................................................................................................32 Conclusions.........................................................................................................................................................33 III. Health Outcomes: Malaria Morbidity, Parasite Prevalence & Anemia.............................................................33 Findings ..............................................................................................................................................................33 Discussion...........................................................................................................................................................38 Conclusions.........................................................................................................................................................39 IV. Effects, Costs and Cost-effectiveness ...............................................................................................................39 Findings ..............................................................................................................................................................39 Discussion...........................................................................................................................................................48 Conclusions.........................................................................................................................................................48 Recommendations .......................................................................................................................................................50 References ...................................................................................................................................................................51 Annexes.......................................................................................................................................................................53 Annex I: Evaluation Statement of Work .................................................................................................................53 Annex II: Getting to Answers matrix ......................................................................................................................61 Annex III: Evaluation Methods and Limitations.....................................................................................................63 Annex IV: Data Collection Instruments and Summary of Field Data.....................................................................73 5 Annex V: Sources of Information ...........................................................................................................................78 Annex VI: Evaluation Team Composition ..............................................................................................................80 Annex VII: Detailed Findings................................................................................................................................. 82 Annex VIII: Sample of Radio Script.....................................................................................................................146 6 List of Figures Figure 1: The Theory of Change for the UCC Linked to Evaluation Questions and Methods....................................14 Figure 2: Comparison of Demographic Variables .......................................................................................................25 Figure 3: ITN use by Age, Gender and Household Supply with ITN..........................................................................31 Figure 4: Exposure to BCC Channels by Education and Wealth.................................................................................32 Figure 5: Trend in Monthly and Quarterly OPD-adjusted Malaria Incidence by Age.................................................35 Figure 6: Malaria Incidence by Age Comparing 12 Months Before and After UCC .................................................. 36 Figure 7: Trend in Quarterly Malaria Incidence in IRS and Non-IRS Areas in Mid-North Region............................37 Figure 8: Changes in Malaria Parasite Prevalence by MIS Region and IRS ...............................................................38 Figure 9: Trend in ITN Ownership Indicators 2006 to 2015 .......................................................................................41 Figure 10: Trend of Population Access to ITN by MIS Regions.................................................................................42 Figure 11: Intra-household Supply Situation with ITN and UCC Nets by Household Size ........................................43 Figure 12: Projected Population Access 2006-2018 Compared to Survey Results .....................................................45 List of Tables Table 1: Registered Population from UCC Records, Database Extract and Census 2014...........................................24 Table 2: Distributed and Allocated LLIN in UCC Records and UCC Database .........................................................26 Table 3: ITN ‘Use If Access’ to ITN by Year and ITN Use by Target Groups...........................................................30 Table 4: Changes in Malaria Parasitemia and Anemia by Age of Child .....................................................................37 Table 5: Expected Net Crop by Year Assuming a 3 Year Survival and Estimates from MIS 2014/14.......................45 Table 6: Projected Need for ITN Comparing UCC and Continuous Distribution .......................................................46 Table 7: Economic and Financial Cost of UCC...........................................................................................................46 Table 8: Economic and Financial Cost per Outcome ..................................................................................................47 7 ACRONYMS AMP Alliance for Malaria Prevention ANC Ante-Natal Care BCC Behavior Change Communication CAO Chief Administrative Officer CD Continuous Distribution CI Confidence Interval DFID (UK) Department for International Development DHE District Health Educator DHIS2 District Health Information System 2 DHO District Health Officer DHS Demographic and Health Survey EPI Expanded Program for Immunization FGD Focus Group Discussion GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GOU Government of Uganda HMIS Health Management Information System INGO International Non-Government Organization IRS Indoor Residual Spraying ITN Insecticide Treated Nets KII Key Informant Interview LC Local Council LLIN Long-Lasting Insecticidal Nets LQAS Lot Quality Assurance Sampling MC Malaria Consortium MIS Malaria Indicator Survey MOH Ministry of Health NMCP National Malaria Control Program OPD Out-Patient Department PCA Principal Component Analysis PMI (US) President’s Malaria Initiative RDT Rapid Diagnostic Test SMP Stop Malaria Project SOW Scope of Work TOC Theory of Change UC Universal Coverage UCC Universal Coverage Campaign UMRS Uganda Malaria Reduction Strategy USAID United States Agency for International Development VHT Village Health Team 8 EXECUTIVE SUMMARY BACKGROUND In May 2013, the Government of Uganda (GOU) supported by the Global Fund, UK Department for International Development (DFID), the US Government Presidential Malaria Initiative (PMI), and World Vision launched its first nationwide Universal Coverage Campaign (UCC) to provide each household in Uganda with at least one long-lasting insecticidal net (LLIN) per two members of the household. Over a fifteen-month period, this undertaking provided more than 22 million LLINs to residents throughout Uganda. In December 2014, Phase I of a two-pronged evaluation was completed and provided details about the process and performance of the UCC. After the close of the Phase I process evaluation, PMI, DFID and Government of Uganda’s National Malaria Control Program (NMCP) commissioned Phase II of the evaluation. Phase II of the UCC evaluation took place between September 2015 and January 2016. EVALUATION PURPOSE AND EVALUATION QUESTIONS The major purpose was to provide key development partners and Government of Uganda with an analysis of the overall performance and impact of the campaign in order to improve future distributions. Specific questions of the evaluation included: 1. What is the quality of household registration data that was collected during the UC campaign and then used to inform procurement and/or distribution of the LLINs? 2. How have the different Behavior Change Communication (BCC) channels affected use of nets? Was the target audience selected correctly? Were the materials and channels used for the target audience of high quality? 3. What measureable health outcomes in terms of morbidity, parasite prevalence and anemia can be attributed to the universal coverage LLIN campaign? How do changes compare to other countries where similar evaluations have been undertaken? 4. Was the UC campaign implemented in the most cost-effective manner? If not, what alternative means could have been pursued by the program to accomplish the same outcomes at a lower cost? What is the comparative cost of the different distribution waves? DESIGN, METHODS AND LIMITATIONS The principle design of the evaluation was a before-after comparison of outcome and impact measures as well as comparison of outputs and costs against expected values. The evaluation used a mix of quantitative and qualitative approaches. Data sources included the population census of 2014, UCC registration and distribution records, malaria cases from HMIS and DHIS2 databases 2006 to 2015, results from population representative household surveys including the 2014/15 Malaria Indicator Survey (MIS), and costing data from implementers of the UCC. Quantitative analysis was based on secondary data only and no primary data was collected. Analytical methods included descriptive statistics, univariable and multivariable comparative statistics and inferences as well as modeling. This was complemented by qualitative data from expert review of the quality of BCC and Key Informant Interviews (KII) and Focus Group Discussions (FGDs) during field visits to four purposively sampled districts: Jinja, Soroti, Apac and Hoima. The major limitation was the limited availability of BCC and cost data to undertake the in-depth analyses of these aspects as originally anticipated. FINDINGS AND CONCLUSIONS 9 I. Quality of UCC Registration and Distribution Data The analysis of data quality of the UCC found that the electronic database was not timely as it was only available in two districts. Reliability around the metric “nets allocated and delivered to sub-county” was found to be insufficient as it was inconsistently defined, making it impossible to use as a management tool to improve distribution efficiency. However, the main problem was found in the accuracy of the data which came from three major error sources: i) an exaggeration of the registered population which was predominantly an increase in number of households and to a lesser extent an increase in the number of people per household; ii) inaccuracies in data aggregation and/or transcription leading to significant differences between the UCC records and database extract; iii) accuracy issues around the nets distributed in comparison to the population, which, at least in part, are suspected to be an “over-correction” of data in the UCC database to match the expected “need”. In Conclusion: • The registration process for UCC had several flaws, the most important being inaccuracy from exaggeration of households size, double counting of households and poor data management. However, it was not possible to determine retrospectively and from the available data exactly at which points in the data management processes the errors occurred. • The registered population of 41 million in 2013/14 (UCC records) is most likely 10% above the true population meaning that the UCC overestimated by about 5 million people based on data triangulation and modeling, thus significantly contributing to inefficiency of the UCC distribution. • The quality of the UCC registration and distribution data was compromised in three areas: o Timeliness: Data from the electronic database was not available to the UCC management team except for the two districts of UCC wave 1 and, therefore, the potential advantages of the electronic database could not be realized. o Reliability: The indicator “LLIN allocated and delivered” was not consistently used across time or between the two data sets (records and electronic database) and could not support optimization of the distribution logistics. o Accuracy: Both electronic data and records had issues with inconsistent data and there was evidence that data in the database had been over-corrected based on the expected rather than the observed results. II. Quality and Targeting of BCC and its Effects on Net Use A BCC strategy for the UCC existed, but was not followed during a highly centralized implementation, circumventing existing district structures and relying heavily on media channels (radio) rather than interpersonal communication. The media campaign in turn was delayed, only covering from UCC wave 6 onwards. The available radio message scripts were found in part to be of questionable quality using scare tactics rather than a positive or encouraging attitude. Nonetheless, ITN use among those who could have used an ITN was high in 2014 at 84%, an increase of 10%-points from before the UCC, and at a very high level in international comparison. Comparison of survey data from 2006 to 2014 showed that children under five and women in reproductive age (and especially those pregnant) were prioritized when ITN in the household are scarce as early as 2006. The UCC enabled groups such as older children and men, who had shown lower ITN use rates before, to catch up. Gender and age differences were found to be minimal post-UCC if a household had ITN for all members. A secondary effect of this increased access and use was that in 2014 there was an increased proportion of nets used by a single person, which lowered the average number of people sharing a net to less than two. About two thirds of women interviewed during the MIS 2014 had been exposed to BCC messages in the previous six months, mainly radio. However, data suggested that media channels reached the better educated and wealthier households, and that interpersonal communication was more suitable to evenly reach the population. While knowledge on malaria transmission and the preventive potential of ITN was very high, neither this knowledge nor recent exposure to BCC messages was associated with increased ITN use, supporting the notion that over the 10 past 10 years a strong net use culture has developed in Uganda that does not depend on a single message but is rather a result of positive experiences with net use, reduced morbidity, and a gradual change of social norm. In Conclusion: • The target audience was the general population and as such adequate for a UCC. • The BCC activities focused on media channels rather than interpersonal communication which could have better reached the rural poor and thus were poorly implemented. • BCC activities were highly centralized and would have benefitted from decentralization and involvement of district level actors. • As far as can be stated from the limited examples of materials the evaluation team had access to, the quality of some of the materials for radio communication was poor as it used a scare tactics approach rather than encouraging correct behavior. • In spite of the limitations in BCC implementation, ITN use was very good and favored the most vulnerable population groups of young children (boys and girls alike) and women in reproductive age. As an effect of improved ITN access older children and men are now using ITN, representing a significant progress achieved by the UCC. • The evaluation established a sufficiently strong net use culture that does not depend on a single message or BCC exposure, but rather on long-term experience and reinforcement through interpersonal communication. III. Health Outcomes of Malaria Morbidity, Parasite Prevalence and Anemia Data from the routine health services indicated that after correction for under-reporting and adjustments for changes over time in OPD attendance there was a clear downward trend between 2006 and 2014 in reported malaria incidence, malaria test positivity and reported anemia for children, older persons, males and females. The reduction in malaria incidence was overall around 35% but three distinct phases of change could be identified: a modest decrease up to the targeted LLIN mass campaigns in 2010/11 of approximately 5%; a 10% reduction following the targeted campaign up to the UCC; and a 25% reduction after the UCC. This pattern was seen in all regions except the North where the IRS created significant morbidity reductions between 2010 and 2014. There also was evidence that an accelerated decline was associated with the UCC with a six-fold increase in malaria incidence decline in the 12 months after the distribution. Both malaria parasite prevalence and moderate/severe anemia in children 6-59 months reduced by about 50% between 2009 and 2014 with strong impact in the IRS districts in the North and a reduction of parasite prevalence below 1% in Kampala. The least reductions were observed in the East Central region where the lowest ITN ownership levels were also observed. In Conclusion: • The Uganda UCC significantly contributed to an impressive decline of malaria incidence of 33-35% since 2007, a 55% reduction in malaria parasite prevalence, and a 52% reduction of moderate/severe anemia. • Although an exact proportionate contribution of the UCC to these declines cannot be made, the evidence is strong that the UCC was a significant part of the success. • Parasite prevalence in the Kampala area was below 1% in children, evidence that in this urban environment malaria is now marginal. IV. Effects, Cost and Cost-effectiveness The analysis of ITN ownership from survey data found that the UCC in Uganda achieved full universal coverage with population access to ITN at over 80%, as high as any other country has ever reported following a mass campaign. The resulting ITN ownership was highly equitable and achieved a high community-level coverage that should ensure optimal effect on malaria transmission in the absence of 11 insecticide resistance. However, there was evidence that the UCC left some geographical gaps around Kampala (wave 8b) and in the North East and East Central (pilot). In addition, small households were shown to have been more likely to be missed while large households were systematically under-supplied. There also was some evidence of oversupply in some waves (1, 5, 8a) and generally among those smaller households that were reached. The data suggests that households tried to fill gaps left by the UCC by obtaining nets from the markets including LLIN while routine distribution of nets through health facilities was very low and far below its potential. Modeling supports the finding that the “true” population in 2014 was more likely to be closer to the census results or slightly above, but below the numbers obtained in the UCC registration process. Modeling also suggests that by the time of the next planned UCC there will be an ITN crop of around at least 6-10 million and that a hypothetical continuous distribution instead of UCC could save around 9 million ITN over a six year period. Overall financial cost of the UCC without the indirect contributions of government and based on the data available was USD 98 million and the economic cost USD 100 million. The vast majority of these costs (83% and 91% respectively) were caused by the net procurement even though the average cost per LLIN was significantly lower than in previous years. Economic cost per net distributed was USD 4.49 and the average annual cost per net year USD 1.49. Based on the estimated 369,233 child deaths averted by the UCC this resulted in a very favorable estimate of USD 271 per death averted and USD 8.2 per DALY averted. However, sensitivity analysis showed that significant savings could have been achieved with more accurate registration data and a more efficient distribution. In Conclusion: • By the end of 2014 Uganda had achieved universal coverage with ITN for the prevention of malaria with over 80% of the population having access to an ITN within the household. This was largely achieved through the UCC as without it coverage would have been merely 11%. But the UCC was not as efficient in achieving ITN ownership as it could have been with some areas having been oversupplied while at the same time also leaving some gaps. Small households tended to be missed more frequently than larger ones and large households were often under-supplied. • Households filled some of the gaps by obtaining nets, and these were mainly LLIN, from the commercial sector. This was particularly the case in those areas where the campaign was not designed to achieve universal coverage (Kampala-Wakiso) and in the pilot districts where in addition to under￾supplying, the distribution was more than two years ago. • Routine distribution of ITN through health facilities was much lower than expected and this distribution is as yet not sufficiently utilized in Uganda. • The average annual cost per ITN year was very low compared to historical data at USD 1.49, largely due to a low commodity price. This made the UCC highly cost-effective with respect to the estimated cost per DALY averted. • The up to 5 million potentially excess nets based on inaccurate registration data were conservatively estimated to have added ~18 Million USD to the cost of the campaign. It appears possible, therefore, that for relatively minor investments in improved registration it might be possible to achieve significant cost savings through better alignment of net procurement and distribution with actual need. • The overestimation of the population and the existence of a considerable net crop of several million additional LLIN still in place by the time the next UCC is planned in 2016/17 shows that efficiency and cost-effectiveness can be further improved and that the UCC approach may not be the best strategy to sustain the achieved UC in the long run. RECOMMENDATIONS Based on the findings and conclusions presented above and in more detail in Annex VII, the evaluation team makes the following key recommendations for future ITN distributions in Uganda: • In order to improve efficiency, the National Malaria Control Program and partners should redesign 12 the registration process and the way data is captured and used for quality assurance. This should include: o Involvement of the district level health staff in the registration process as these usually have a sense of the communities they serve and can enhance the feed-back loop for data quality. o Investment in an improved UCC registration database that should consider real-time data entry and availability via SMS or other phone-based systems to allow immediate plausibility checks on the reported number and can serve as a logistics management tool (nets allocated, received, returned etc.). o Independent data quality audits should be implemented post registration. • The BCC strategy around UCC should be reconsidered and scaled down focusing on the following elements: o Decentralization of the planning and implementation to the districts with stronger involvement of the District Education Officer and the LC and VHT staff. o Focus on interpersonal communication that positively enhances the existing net use culture and presentation of messages on net care and repair. o Clear communication of the allocation rules of nets during the UCC to avoid misunderstanding that each person is to be given their own net. • Future evaluations of LLIN distributions in Uganda should be planned well in advance and start at the planning stage of a UCC or other type of distribution so that necessary information will be available in adequate detail and quality for the post-distribution evaluation. o This applies particularly to the cost data for which cost categories must be defined in advance and consistent data collection tools be used throughout. o BCC is another area where better planning of future evaluations is critical and standardized records of channels and messages (with target groups) should be prepared during implementation stating location, time and intensity of activities. • Given the low malaria parasite prevalence in Kampala in conjunction with high presence of commercial LLIN and obvious willingness of large parts of the population to purchase them, NMCP and partners should consider excluding Kampala and possibly Wakiso from future UCC and instead strengthen routine net distribution channels and possibly community distributions in socially and economically disadvantaged areas. • The medium to long-term strategy to maintain universal coverage should be revised to move away from an effective but comparatively inefficient repeat campaign system. Instead, a comprehensive continuous distribution system should be designed and implemented that allows more initiative of the household to “manage” their ITN supply based on actual need and also enables the engagement of the private sector with markets contributing to achievement of the universal coverage target. 13 BACKGROUND Malaria Control with Insecticide Treated Nets in Uganda In Uganda, malaria transmission, primarily with Plasmodium falciparum, is intense and perennial in more than 95% of the country. In highland areas in the South West, West and East, malaria transmission is low and more epidemic prone. Although Uganda’s malaria control policy embraced the use of Insecticide Treatment Nets (ITN) as early as 1998, ownership and use remained low in the initial years [1]. Efforts to improve this began with social marketing of ITN through international non-governmental organizations (INGO) strengthening the retail markets for nets and ITN. There were some local distributions of ITN through various partners in selected areas including the conflict areas in the North but not at national level. Because most of the nets at the time were conventionally treated ITN that needed retreatment every 6-12 months, the National Malaria Control Program (NMCP), with support from partners, carried out several mass dipping campaigns from 2003 and 2007 treating around half a million nets in each round. At the same time, the new ITN technology of long-lasting insecticidal nets (LLIN) was introduced as the standard product1 . In line with the Uganda Malaria Control Strategic Plan 2005-2010 continuous distribution of LLIN through Ante Natal Care (ANC) and Expanded Program for Immunization (EPI) services was launched in 2006 with support from various partners including the US President’s Malaria Initiative (PMI) and the Global Fund to Fight AIDS, TB and Malaria (GFATM). While ITN ownership and use increased, the Malaria Indicator Survey (MIS) of 2009 showed that this increase was not yet sufficient with only 47% of households owning any ITN and 33% of children sleeping under an ITN the previous night. NMCP and partners therefore implemented the first national LLIN distribution campaign in Uganda in 2010-2011. This campaign distributed slightly more than seven million LLIN to beneficiaries, targeting children under five and pregnant women. The Universal LLIN Coverage Campaign 2012-2014 The paradigm for the use of ITN or LLIN for malaria prevention changed in 2008 when the World Health Organization (WHO) and Roll Back Malaria Partnership (RBM) moved from the overall goal of malaria control to elimination and ultimate eradication. This meant that the strategy for ITN moved from individual protection to being a major tool for vector control and transmission reduction. It implied that rather than targeting only the biologically vulnerable (children under five and pregnant women) the entire population at risk for malaria was now the target group. In order to achieve maximum impact universal coverage was now the objective meaning that all communities at risk should be reached and at least 80% of the population at risk should be able to use an ITN. The Uganda Malaria Reduction Strategy (UMRS) 2014-2020 reflects these changes and outlines a scale-up of interventions to achieve universal coverage of key malaria prevention and treatment activities as a means to achieve sustained malaria control in Uganda. The Universal Coverage Campaign (UCC) for LLIN was first piloted in September 2012 and then implemented in eight waves between late 2013 and August 2014. It was led by the Ministry of Health (MOH), supported by other parts of government and development partners, and financed by GFATM, PMI, the UK Department of International Development (DFID), and World Vision. 1 Since conventionally treated ITN have become more or less obsolete and LLIN are a sub-category of ITN, today the two terms ITN and LLIN are used interchangeably. 14 The Theory of Change Underlying the UCC The UCC is not a project or program in itself and, therefore, has neither an explicit Theory of Change (TOC) nor a log-frame of clearly defined goals, purpose and outcomes, nor indicators to measure success. However, the TOC is implicit in the Malaria Reduction Strategy and was summarized by the evaluation team as shown in Figure 1. The flow from the output of LLIN through UCC to achieving the outcomes of universal ITN ownership and use, and the ultimate impact on malaria transmission and disease burden forms the basis of this evaluation. Figure 1 demonstrates how the evaluation questions and methods are linked to the various parts of the UCC TOC. FIGURE 1: THE THEORY OF CHANGE FOR THE UCC LINKED TO EVALUATION QUESTIONS AND METHODS* *The circled numbers represent the evaluation questions 15 EVALUATION PURPOSE AND QUESTIONS After the successful completion of the process evaluation of the UCC in October 2014 (Phase I), USAID, PMI, DFID and NMCP commissioned Phase II of the evaluation. The major purpose of Phase II was to provide Government of Uganda and development partners with answers to critical questions about the overall performance and impact of the UCC in Uganda in order to improve future distributions. The key evaluation questions as presented in the Scope of Work (see Annex I) were discussed with MOH, the team of the USAID, Monitoring and Evaluation Learning Contract, key stakeholders and USAID/PMI Uganda during the inception phase of the evaluation, and slight modifications were agreed upon and presented in the inception report. There are four major evaluation questions which directly relate to the TOC as shown in Figure 1 and each have a set of sub-questions that were defined in the “getting to answers” matrix as part of the SOW: 1. What is the quality of household registration data that was collected during the UC campaign and used to inform procurement and/or distribution of the LLINs? a. What is the accuracy and completeness of the registration data? b. How do the registration data compare against previous (2002) and recent (2014) census data? c. What is the overall quality of the UCC registration data based on the five (USAID) quality criteria? 2. How have the different behavior change communication channels affected use of nets? Was the target audience selected correctly? Were the materials and channels used of high quality for the target audience? a. Which communication channels were used where in which intensity used during the UCC? b. What was the quality of messages delivered or materials used? c. What was the use pattern of nets by age, gender and other factors? Who used the nets from the campaign? d. Which BCC channels reach which type of households? e. Is there a link between exposure to BCC and net use after the UCC? 3. What measureable health outcomes in terms of morbidity, parasite prevalence and anemia can be attributed to the universal coverage LLIN campaign? How do changes compare to other countries where similar evaluations have been undertaken? a. What have been the trends in malaria morbidity before and after the UCC by region, age, gender? b. What have been trends in parasite prevalence and anemia in children under five before and after the UCC? c. To which extent can changes in incidence, prevalence and anemia be attributed to the UCC? d. How do trends in malaria morbidity, parasite prevalence and anemia in Uganda compare to other countries in Africa South of the Sahara? 4. Was the UC campaign implemented in the most cost effective manner? If not, what alternative means could have been pursued by the program to accomplish the same outcomes at a lower cost? What is the comparative cost of the different distribution waves? a. What were the outcomes of the UCC regarding ITN ownership? b. What was the economic cost for the UC? Total and per LLIN delivered? 16 c. What were the cost drivers and how did cost/LLIN delivered differ between waves? d. How does the cost of implementing UC in Uganda compare with UC costs in other countries? e. Could there have been significant savings without loss of outcome? f. What is the ratio of benefits to costs? During the evaluation work the team added some more sub-questions regarding ITN ownership in line with the overall evaluation objectives (question 4, sub-question a): • What was the ITN ownership coverage achieved in late 2014 at household and community level? How do these outcomes compare to other countries that have done UCC? • What was the contribution of the UCC to these outcomes? What would have been the coverage without the UCC nets and what if only the UCC nets would have been there? • Was there any evidence of over- or under-supply with ITN? And if so where? • Was ownership of UCC and other ITN equitable across wealth quintiles? • What can we learn from ITN ownership with respect to the planning of future distributions, especially regarding the most likely “true” population size and the expected coverage and “net crop” before the next UCC? The report is structured to present findings, discussion and conclusions around four headings, which directly relate to the four evaluation questions and their sub-questions. Additional detail on methods and findings is presented in Annexes III and VII respectively. 17 EVALUATION DESIGN AND METHODS The principle design of the evaluation was a before-after comparison of outcome and impact measures as well as comparison of outputs and costs against expected values. The Phase II evaluation methodology involved a mix of quantitative and qualitative methods in order to fully address the questions of interest. However, in contrast to Phase I, which was based predominantly on qualitative data, this analysis is primarily quantitative. Furthermore, quantitative analysis was based exclusively on secondary analysis of existing data and primary data collection by the evaluation team was limited to the qualitative data. Quantitative Methods Data Sources: The Phase II of the Uganda UCC evaluation provided a unique opportunity to address the key evaluation questions due to the almost simultaneous availability of data from two independent data collections that directly relate to the campaign and hence can be used to assess the outcomes and data quality of the UCC. These are the National Census, which took place in August 2014 at the end of the UCC implementation, and the Malaria Indicators Survey (MIS) undertaken between December 2014 and January 2015, 5-15 months after the core of the UCC waves. There were five major secondary data sources. How these relate to the evaluation questions is shown in Figure 1 and also described in the “getting to answers matrix” in Annex II. National Household Census 2014: At the time of the evaluation only the provisional census data were available [5]. Data on number of households and population by district were extracted from the report. No census data at sub-county level were included. Routine data on malaria morbidity from HMIS: Malaria cases, malaria diagnostic tests and positivity, as well as anemia cases and OPD attendance by age and gender were extracted by month and district from the HMIS/DHIS2 databases for the period January 2006 to June 2015. Household Surveys: Representative household interview surveys used were either. Demographic and Health Surveys (DHS 2006 and 2001) or MIS (2009 and 2014/15) and all data sets were obtained with permission from the DHS Program website (https://dhsprogram.com/data/). The sampling for these surveys was two-stage cluster sampling with probability of cluster selection proportionate to size and 10 regions as domains. All surveys included the malaria module on net and ITN ownership and use. The MIS also included malaria parasite and hemoglobin data for children and malaria, as well as exposure to messages and knowledge for women in reproductive age. Sample sizes were 8870 (DHS 2006), 4421 (MIS 2009), 9033 (DHS 2011) and 5345 households (MIS 2014/15). UCC Registration and Distribution Data: There were two distinct data sets available, both originating from the registers at community level. • UCC records for registration of population, allocation and distribution (LLIN) figures per district were used as reported in the final version of the Phase I report dated September 30, 2015 (Annex 8, Table 8). This data was collated before the electronic database had been finalized and is based on the aggregated forms from the field as well as waybills of delivery to sub-counties from the logistics group. In addition, a version of these data by sub-county from December 2014 was also available and served as comparison of sub-county level data for the UCC database extract (see 18 below). • UCC database as maintained by the Ministry of Health Resource Center. The evaluation team was not able to access to full data set from the database and only an extract by sub-county was made available for a subset of 19 districts purposively selected by the evaluation team. Originally the team had requested 21 districts of three equal groups of seven based on variance of population figures from the census 2014 results [5]: high (>10%), slightly high (0-9.9%) and low (<0%) representing all UCC waves and all regions except Kampala. However, as the districts from the UCC pilot in September 2012 were not included in the UCC database, only 19 districts could be included in the analysis. The variables available by sub-county were: population registered, number of children under five, pregnant women and households, LLIN allocated and LLIN distributed. Costing data: Costs were collected retrospectively from financial and operational records kept by Malaria Consortium, Population Services International, John Snow/DELIVER and World Vision, and through interviews with program managers and implementers involved in the deployment of the UCC campaign in Uganda. Data Preparation and Analysis In general, quantitative data analysis was conducted using Microsoft Excel (Office 2010) and the Stata 14.0 software package. Analytical methods started with univariate exploratory and descriptive analysis of the data using graphs, tabulations and correlations. For multivariable analysis linear and logistic regressions were used. For testing of statistical significance parametric and non-parametric tests were used as applicable, and a significance level of p<0.05 was applied. Mapping was done using the Mapviewer 8.3 software (Golden Software). Since the specific analytical methods differed for each evaluation question, these are presented separately: I. Quality of UCC Registration and Distribution Data The criteria used to assess quality of data vary with the type of data, e.g. they are different for administrative and survey data. In addition, the terms used are often times defined in slightly different ways by the users or donor agencies. Therefore, the evaluation used “fitness for use” as a general guidance to assess data quality [2]. We applied – to the extent possible – the quality criteria as proposed by USAID [3] with two changes: i) as the USAID data quality checklist is designed for survey data and the criteria of precision2 does not apply to administrative data, we used instead the criteria of accuracy as defined in the GFATM routine data quality assessment tool [4]; ii) we added the criterion of completeness which is important for administrative data. Therefore, the following criteria were included with definitions presented in Annex III: timeliness, completeness, validity, reliability, integrity and accuracy. The quantitative analysis of the UCC data for accuracy was done in two steps: first, the population registered was considered and compared with projections from the 2002 census (which had served as benchmark for the procurement of LLIN), the 2014 census results, and between the two UCC data sources (records vs. database) for the 19 districts of the UCC database extract. The findings were triangulated with the three demographic measures of “% of children under five”, “% of currently pregnant women” 2 Defined as: Data have a sufficient level of detail to permit management decision-making; e.g. the margin of error is less than the anticipated change. 19 and “mean household size” from MIS 2014 and census 2014 in order to assist the interpretation of findings; second, the number of nets allocated and distributed were explored for agreement and/or variation between the two UCC data sources and in relation to the registered population (people per net distributed). II. FGD and its Effects on Net Use Quantitative assessment of net use was based on the DHS and MIS data sets 2006 to 2014 comparing “population access to an ITN” with the “ITN use the previous night” by gender and various age groups. Exposure and effects of BCC were assessed using the MIS 2014/15 women’s module, which contained the questions on BCC, malaria transmission knowledge and suggested ways to prevent malaria. Variables were re-grouped where necessary. Reported knowledge on malaria transmission was grouped into three categories: i) “correct” knowledge which included mention of mosquitoes, parasites or stagnant water; ii) “incorrect3 ” knowledge which included exposure to rain, cold or poor hygiene; iii) “wrong” knowledge which included eating certain food, witchcraft or being infected by a malaria patient. III. Health Outcomes of Malaria Morbidity, Parasite Prevalence, Anemia For the raw HMIS/DHIS2 data a lengthy cleaning and preparatory process was applied which is described in detail in Annex III. The main outcome variables used for the trend and before-after analysis were the proportionate malaria rate, i.e. malaria cases per 100 OPD visits and the reported malaria incidence rate (cases/1000 population/year) adjusted for under-reporting and variations in OPD attendance. Trend analysis was done using moving averages of the monthly or quarterly data and the difference in trends before and after UCC were explored using linear regression analysis. For the analysis of trends of malaria parasite prevalence and anemia two data points were used: the MIS 2009 and MIS 2014/15. Malaria parasite prevalence analysis was based on the microscopy results and classification of anemia was done based on WHO standard hemoglobin cut-offs of 11.0g/dL for any anemia, below 10.0 and above 7.0 for moderate and below 7.0 for severe anemia. In addition, the RBM recommended cut-off of 8.0g/dL was used for “moderate/severe” anemia. Analysis of parasitemia and anemia was limited to children aged 6-59 months. IV. Effects, Costs and Cost-effectiveness To assess the effects and efficiency of the UCC distribution, ITN ownership across the available DHS and MIS data sets was used for the before-after analysis with some complementary insights from the FGDs during the field visits (see qualitative methods). The core net and ITN ownership indicators were used as recommended by WHO [9].The calculation of the access indicator was modified slightly in order to ensure that any person using a net was categorized as having access and details are provided in Annex III. Additional indicators were calculated which are not commonly used in DHS/MIS surveys to assess intra￾household ITN density (over- and under- supply) and these are also described in detail in Annex III. Equity of the UCC outcomes was assessed based on the standard wealth index and quintiles and the analytical methods used were the equity ratio, concentration index and concentration or Lorenz curves [11]. Two models in NetCALC4 were used to help interpret findings. First, a situation was simulated where the UCC had achieved 95% coverage with any ITN at the time of the distribution and then the expected 3 Incorrect in the sense that these criteria are associated with fevers in the broadest sense, but not with malaria fever. 4 Details can be found at: http://www.vector-works.org/resources/netcalc-planning-tool/ 20 decline of coverage over time was estimated based on a 2.5, 3.0 and 3.5 year median survival of the nets. This range is based on findings from two areas in Uganda, the Mid-West where LLIN effectiveness studies have been done for over 10 years using polyester nets [12] and Mid-Eastern region with a recent two-year LLIN durability study involving both polyester and polyethylene LLIN [13]. The outcome of the model was compared to observed ITN coverage by UCC waves, i.e. taking into account time since distribution. Second, a model was developed using the historical distribution of nets from various sources from 2003 to 2015. This included untreated nets as well as conventionally treated and LLIN distributions through campaigns, and facility-based by the public sector (NMCP, INGO, UN-organizations) and the private sector (retail, social marketing and institutional sales). This data had been compiled over time from NMCP records and confidential reports from net distributors. Where no exact numbers were available estimations were made (e.g. for import of untreated nets through “informal” channels). The provider perspective was used for the cost analysis. Travel or time costs to users, other household￾level costs or cost savings have not been measured or included. Details of the procedures to calculate economic cost and cost-effectiveness are presented in Annex III. Qualitative Methods Qualitative assessment of BCC planning and implementation was done by expert review comparing BCC strategy, implementation guidelines and reports and data from Key Informant Interviews (KII) during the field visits (see below). Expert review was also used to evaluate quality of BCC materials based on the limited samples collected during this evaluation (see list in Annex III). Field Visits for Qualitative Data Collection During the second week of December 2015, the evaluation team visited four districts in Uganda. The aim of the field visits was to gather qualitative information from interviews with key informants and community beneficiaries to corroborate the evaluation findings. The qualitative focus was to review respondents’ perceptions of the UCC registration and distribution process, to verify which BCC channels were used during the UCC, and to understand if the intended messages were deemed effective. Districts were selected purposely guided by pre-determined criteria to include districts that i) had IRS experience - Apac ii) had participated in the 2009-2010 UCC in mid-western Uganda when over 7 million LLINs were distributed to vulnerable children and pregnant women - Hoima iii) were in the initial wave of the 2013-2014 UCC and therefore had allocation and distribution activities guided by real-time data entry - Soroti, and iv) included a district with rural and urban characteristics - Jinja. Thematic areas were identified as targets for inquiry and served as the basis for the open-ended interview tool that was used (see Annex IV). District administrative and health officials served as key informants and were selected for interviews based on their engagement with planning or implementation of the UCC. They included the Regional District Commissioner, the Chief Administrative Officer, and some members of the District Health Team, including the malaria focal person. Purposive sampling was used for focus group discussions (FGDs) and incorporated community leaders (Village Health Teams - VHTs) in addition to community beneficiaries from small (<5) and large (>7) member households. Local leaders (VHTs and Health Assistants) determined membership for the FGDs. Seventeen FGDs with a target of eight respondents each were led by research assistants who conducted interviews in the local language. 21 Limitations of Data and Mitigation Access to the MIS 2014/15 data set was only obtained in early November 2015 due to the delayed official dissemination of the report and release of the data sets. There were difficulties in accessing the full national data from the UCC registration and distribution database maintained at the MOH Resource Center due to issues of confidentiality. In response the team held intensive discussions, organized a half-day workshop and provided detailed analysis plans as well as a written agreement on data use and confidentiality. This then allowed access to a small extract of the data for a sample of districts selected by the evaluation team, which was obtained in mid-December 2015. It proved difficult to obtain detailed costing data that would have allowed – as originally planned – to analyze cost for the different UCC waves as a measure of efficiency. This was not due to lack of cooperation but due to the fact that such analysis had not been anticipated and the accounts did not allow a retrospective extraction of the necessary information. Also, cost categories were not comparable between different donors limiting the detail that could be applied in the cost analysis. Since one of the key players for BCC implementation, the Stop Malaria Project, had ended by the time of Phase II of the evaluation, it proved difficult to obtain BCC materials in as much detail as had been hoped for. This was mitigated by the evaluation team by including data from a previous visit to Uganda regarding the UCC by the BCC expert of the evaluation team during an earlier work (August 2014) not related to this evaluation which included the “Stop Malaria Project” then still active. 22 FINDINGS, DISCUSSION & CONCLUSIONS I. Quality of the UCC Registration and Distribution Data Findings The flow of data and origin of the two data sets used (UCC records vs. UCC database extract) and the potential sources, errors or problems are shown in Annex VII. Findings Timeliness Definition: Data should be available in sufficient time to influence management decision-making. The electronic database was essentially unavailable for the actual allocation and distribution except for the two districts of wave 1 (Soroti and Busia). The remaining data was only entered into the database in December 2014 due to delays in fund acquisition. For the pilot and waves 2-8b, the UCC was managed by the manual aggregation of data from the registration forms to allocate nets to the sub-counties. While these were generally available in time, the span between registration and distribution for waves 5-8b was prolonged due to transportation delays (p. 16, Phase I report). Completeness Definition: Results are appropriately inclusive: it represents the complete list of eligible units, not just a fraction, and all data fields have valid entries. Since the UCC records as well as the UCC database extract were only available as aggregate data at sub￾county and district level, a full assessment of data completeness at lower levels of data collection during the UCC was not possible. In the UCC records by sub-county that included all waves but 8b (Kampala￾Wakiso) only one issue regarding completeness was found: in Rukungiri District the figures for “households registered” were not included while the other variables (population, nets allocated and distributed) were complete. In the UCC database extract of 19 districts, in one instance, the name of a sub-county was missing and in two instances the figures for LLIN distributed was missing. This represents 1.2% of data rows (sub-counties) and 0.4% of all data fields (5 per sub-county). Validity Definition: Data should clearly and adequately represent the intended result. The key metrics of the UCC data were: • Number of persons registered • Number of LLIN allocated and delivered • Number of LLIN distributed to beneficiaries These metrics are all adequate. The same is true for the additional metrics of number of households, children under five, and currently pregnant women that were used to “verify” the registration data. Data validity is very good with clear measures of intended results. Reliability Definition: Data should reflect stable and consistent data collection processes and analysis methods over time. The UCC was a one-time activity and, therefore, the consistency in the data collection process can only 23 be checked across the geographic coverage (UCC waves) and not over time. Given the limitations of an evaluation of processes over one year ago, one major issue with reliability of the UCC data was found based on the quantitative analysis of the UCC records and UCC database extract. With reference to the “number of LLIN allocated and delivered” to district or sub-county, the UCC records reported a balance between allocated and distributed LLIN for most districts between roughly minus 10,000 and plus 24,000 LLINs. Therefore, between 81% and 104% of distributed LLIN had actually been allocated. In most cases, the number of nets shifted to other districts or received in addition was noted. However, in 15% of districts the number of LLIN allocated and distributed was exactly the same. This suggests that a different approach was taken by either entering the allocated as distributed or vice versa. In the UCC database extract not a single district had a negative balance, so the three districts with between 102% and 104% of allocated LLIN distributed were corrected downwards. While some differences would be expected through correction of obvious summation or recording errors (see Annex III flowchart Figure 13), the observations of the data strongly suggest that this metric was not reliably captured across all districts and was interpreted differently between the UCC records and UCC database. This was not an issue of reliability of the tools (forms) for data processing, but rather of data management. Integrity Definition: Data collected should have safeguards to minimize the risk of transcription error or data manipulation. For UCC records, the Phase I report (page 15) describes crosschecks of registration information by sub￾county and cluster supervisors. At the central level, records of allocated LLIN were checked and approved by the logistics sub-committee. Reports on LLIN distributed were included in the weekly reports of the implementing partner (Malaria Consortium) and could be checked by the National Coordination Committee. This element was captured in Phase I of the UCC process evaluation. For entry of the UCC data into the database at the MOH Resource Center, a hierarchy was in place to check and verify entered data (system administrator, production officer, verification officer, supervisors) and the databases recorded log in and out, and quantity of records entered for each data entry clerk (details see Annex VII), but no double-entry and validation was used. As for the UCC records, the evaluation team could not verify to which extent these measures were implemented. However, scrutiny of the UCC records and UCC database extract do not suggest any major unauthorized data manipulation. Potential data manipulation below the level of detection in this evaluation cannot be completely excluded. Accuracy Definition: Accurate data are considered correct when the data measures what they are intended to measure. Accurate data minimizes error (e.g., recording or respondent bias, transcription error) to a point of being negligible.5 Registered population figures are captured in Table 1. Including the four pilot districts, the population registered according to the UCC records was 41.0 million and 6.2 million more than the 34.8 million counted in the census. Despite the population census being completed immediately after the end of the UCC in August 2014, this significant discrepancy with a variance of plus 15.2% was noted. When analyzed by census region, the difference was largest in the North at 19% and similar in the other three 5 In general, accuracy is often also described as validity in the statistical and evaluation literature, but because in the USAID data quality guidance validity is defined otherwise, we avoid this use here. 24 regions ~14%. The 2014 population projected from the 2002 census was 35.6 million, which was close to the actual 2014 census results. However, mapping the variance between the 2014 and 2002 census data shows that the fit by district was very poor, especially in the North where most districts were significantly lower in the 2014 census than projected, suggesting that the growth rates decreased compared to the previous census period, most likely due to out-migration. This over-estimation by projections from the 2002 census in the North was also true when comparing to the UCC registration results, while for most other districts the UCC registration was higher. This confirms that using over 10 year old census data for calculating the procurement need for a UCC always has a significant risk of error. Mapping of the variance between the UCC registration and the 2014 census population shows that there were only seven districts (6%) where the UCC registration was below the census. For 24 districts (21%) it was less than 10% above the census and in 81 (72%) the variance was above 10% and up to 57% (Kaabong District). The largest proportion of high positive variance was in the North (83% of districts) followed by the East (78%), West (65%) and Central (58%). Table 1 presents the population results from the UCC database extract in total and by the three variance groups in comparison to UCC records and 2014 census. The total population was lower by 169,000 in the UCC records by district. Most of the difference came from those districts that had low populations compared to the census and these were 11% higher in the UCC database than in the UCC records. Comparison of the UCC records by sub-county with the UCC database provides two important observations: first, among the 255 sub-counties there was not a single one where the data in the UCC records and UCC database was exactly the same and this was true for all variables (population, households, nets allocated and nets distributed). While some differences are to be expected from corrections of entry error or omissions (see also Figure 13, in Annex III), this 100% variation appears too high. This high variation implies that in each sub-county there was at least one parish or LC1 that had incorrect data; second, the UCC records by sub-county also did not align with the UCC records by district which was especially true for the low variance group and shows that there was considerable inconsistency or confusion in managing the aggregation of the various forms. This could have been avoided had the UCC database been available during the distribution process. TABLE 1: REGISTERED POPULATION FROM UCC RECORDS, DATABASE EXTRACT AND CENSUS 2014 Category Sub category Population Relative variance UCC records Census 2014 UCC database Records vs census in % of records Database vs records in % of database Database vs census in % of database UCC total 41,034,354 34,813,459 n.a. 15.2% n.a. n.a. Regions Central 11,122,119 9,579,119 n.a. 13.9% n.a. n.a. Eastern 10,521,225 9,029,408 n.a. 14.2% n.a. n.a. Northern 8,938,752 7,230,661 n.a. 19.1% n.a. n.a. Western 10,452,258 8,974,271 n.a. 14.1% n.a. n.a. UCC database extract 7,764,316 7,186,478 7,933,165 7.4% 2.1% 9.4% Variance group High 2,846,208 2,221,131 2,842,500 22.0% -0.1% 21.9% Slightly high 3,233,838 3,045,037 3,194,171 5.8% -1.2% 4.7% Low 1,684,270 1,920,310 1,896,494 -14.0% 11.2% -1.3% 25 In order to see at which level registration data differed from the expected, demographic variables derived from the registration data of the UCC database extract were compared with results from the 2014 MIS and the 2014 census. Results are shown in Figure 2 below. For all three variance groups the proportion of children under five and pregnant women among the total population was higher in the registration data by 3-4%-points for children under five and 0.5-0.8%-points for pregnant women. In contrast, the mean number of people per household was very similar between registration and MIS 2014 and slightly lower in the census. But this difference was not sufficient to explain the variance in overall population figures. This suggests that most of the variance came from a higher number of households rather than increases in household members, despite exaggerated figures for specific risk groups. FIGURE 2: COMPARISON OF DEMOGRAPHIC VARIABLES Comparisons made between UCC database extract, MIS 2014 and Census 2014 by variance group of population register vs. census An additional approach highlighting data differences is described in detail in Annex VII. Here, variances of different variables expressed as a percentage are plotted against each other. The greatest inconsistencies were noted between the UCC implementation records and the registration database. In contrast, the population variances in i) the UCC database compared to the census and ii) the UCC implementation records compared to the census had reasonable correlation. In ~80% of UCC records and database, the differences varied similarly away from the census data with greater variance in the database compared with the UCC implementation records. Finally, the household variance (database vs. census) was compared to the population variance (database vs. census) and variance for households was lower than that for the population, indicating that the difference in population was mainly related to a higher number of households rather than a higher number of household members. Information from the field visits suggested that in some cases registration data was inflated by the mechanisms previously suggested. In all four districts KII as well as FGD respondents mentioned that the numbers of households were exaggerated by households themselves as well as by village leaders (LC1). Some households registered multiple times or “ghost households” were created. Interviews in all districts visited mentioned that district teams were not involved, thereby perpetuating inaccuracies of the registration data, as district teams were not involved in the processing of the data since forms went directly from the sub-county level to the center. 26 TABLE 2: DISTRIBUTED AND ALLOCATED LLIN IN UCC RECORDS AND UCC DATABASE Category Sub category LLIN distributed during UCC Relative measures UCC records UCC database % of allocated distributed UCC records % of allocated distributed UCC database Persons per LLIN distributed UCC records Persons per LLIN distributed UCC database UCC total 22,267,777 n.a. 99.3% n.a. 1.84 n.a. Regions Central 5,673,744 n.a. 99.5% n.a. 1.96 n.a. Eastern 5,746,076 n.a. 98.7% n.a. 1.83 n.a. Northern 5,021,127 n.a. 98.7% n.a. 1.78 n.a. Western 5,826,830 n.a. 100.0% n.a. 1.79 n.a. UCC database extract 4,491,081 4,349,273 99.5% 98.9% 1.73 1.82 Variance group High 1,599,265 1,538,489 99.5% 98.1% 1.78 1.85 Slightly high 1,839,988 1,768,413 100.3% 99.4% 1.76 1.81 Low 1,051,828 1,042,371 99.5% 98.1% 1.78 1.85 The second part of the data quality assessment focused on the nets allocated and distributed in relation to the population registered between the UCC database extract and UCC records. Data for nets distributed is shown in Table 2 and indicates that at national level 99.3% of the allocated nets were distributed with 1.84 persons per net distributed, close to the 1.78 estimated for UC [6]. While the database population was higher than the records, the opposite was true for the nets where 141,808 less nets were captured in the database versus UCC records. This was true for all three variance groups. As a result the population per net distributed was lower in the UCC records than in the database. Even though the lower variance sounds suspicious, it is possible that the “corrections” represented a true correction in the records during data entry. Soroti, together with Busia districts, comprised wave 1 of the UCC –the only occasion where the electronic database was used for net allocation. Even in this case, the records and database disagree with 1.66 population per net in the records (suggesting some oversupply) but +4.5% population, -4.3% nets allocated and -11.6% nets distributed in the final database establishing a people per net ratio of 1.95. Data from the MIS 2014 (see also next section) as well as the qualitative data from that district suggest that, indeed, there was some oversupply: 18 months after the UCC 80% of people still had access to a campaign net, 4.6% of households reported having some "extra" nets, and 26% had some of their nets not hanging even though 92% of those with access to a net used it that night before. So there is good evidence that there was some oversupply and the original record was more accurate than the final database. Discussion How does the Uganda experience of a significantly higher population in the registration than in a census done at the same time compare with previous campaigns in Uganda or in other countries? Even though in most of Uganda the 2010 mass distribution of LLIN was targeted to children and pregnant 27 women, in some districts in the West a UCC was undertaken as part of a project funded by the British INGO Comic Relief and implemented by Malaria Consortium. The Registration records for Hoima, Kiboga (including Kyanwanzi at the time) and Buliisa were used to compare to census and 2014 UCC registration records6 . While for two districts the results were closer to the 2014 census with a variance of - 2% (Hoima) and +4% (Kiboga) and considerably far from the UCC records (+19% and +10% respectively) suggesting that here the census is probably the more correct result, the third district was closer to the UCC records with +3% (Buliisa) and far above the census with +29%. Experience from many other countries exists, but is poorly documented. The NetWorks project in Senegal attempted to explore the discrepancies between the projected procurement need based on the 2002 census and the 2010 campaign registrations in 18 districts (unpublished data from Eric-Marie Dupuy). Overall the variance of registration to census projection was +7% but varied significantly between districts from - 60% to +26%. Another example comes from the 2010 UCC in Tanzania where sleeping places rather than population were registered [8]. At national level the variance of registered sleeping places to projection from the 2002 census was +5% but varied between the seven zones from -13% to +30%. In Senegal and Tanzania, the census was 8 years before the UCC and differences are to be expected. In the Uganda case with the census done immediately following the UCC one would expect less variances to be noted. A critical question is what the likely “true” population of Uganda was at the time of the UCC and hence how many LLIN would have been needed. It is, of course, impossible to provide a definite answer in the context of this evaluation, but comparing all five census results since 1969 with the result of the UCC registration (see Annex VII) shows that the 41.0 million clearly appears too high as it would imply a growth rate of 4.5% since 2002 which is unlikely as the 2011 DHS [7] reports a reduction in the fertility rate from 6.9 in 2000 to 6.7 in 2006 and 6.2 in 2011. Such a reduction is also consistent with a reduction of the population growth rate reported in the census [5] from 3.2% in 1991 to 2002 to 3.0% from 2002 to 2014. However, it is also possible and even likely that each census under-estimates the true population to some extent. So if we assume that 5%-points of the 15% variance between UCC registration and 2014 census was due to census errors and 10%-points due to registration errors, one can estimate the “true” population to be 36.9 million. This would imply a need of 20.7 million LLIN for the UCC meaning that in this scenario 1.5 million LLIN more than necessary had been distributed. Conclusions From these findings the following conclusions are drawn: • The registration process for UCC had several flaws, the most important being inaccuracy from exaggeration of household size, double counting of households and poor data management. However, it was not possible to determine retrospectively and from the available data exactly at which points in the management process the errors occurred. • The registered population of 40 million in 2013/14 is most likely 10% above the true population meaning that the UCC overestimated by about 5 million people. This significantly contributed to inefficiency of the UCC distribution. • Based on data available for review, the completeness of registration data was high. • The quality of the UCC registration and distribution data was compromised in three areas: 6 In order to adjust for population growth 2010 to 2014 the registered population was projected forward using a 3.0% growth rate as indicated by the 2014 census. 28 o Timeliness: the electronic database was not available to the UCC management team except for the two districts of wave 1 of UCC and, therefore, the potential advantages of the electric database could not be realized. o Reliability: the indicator “LLIN allocated and delivered” was not consistently used across time or between the two data sets (records and electronic database) and could not support optimization of the logistics of the distribution. o Accuracy: both electronic data and records had serious issues with inconsistent data and there was evidence that data in the database had been over-corrected. II. Quality and Targeting of Behavioral Change Communication and its Effects on Net Use Findings BCC Strategy The BCC strategy was taken from the National Communications Strategy for Malaria 2012-15 and further informed by the UCC pilot conducted in 4 districts in August/September 2012. The pilot implementation featured a number of BCC activities including local radio, banners, film-vans and T-shirts for VHT who undertook interpersonal communication (IPC) using job aids. In addition, post-campaign BCC activities led by VHT were undertaken. However, attempts to replicate this activity at scale for the main UCC were not met. The UCC Implementation Guidelines outlined numerous national and sub-national level BCC strategies outlined to co-opt many entities to support the campaign including faith-based organizations, telecoms companies, oil and gas companies, etc. but this did not materialize. Other strategic aims, including at sub￾national level, were not fulfilled. These included mass media campaigns with local language translations on radio to have been led by the District Health Educators with other district level support, but in effect district engagement was severely limited. A further strategy planned, but with patchy implementation, was the demonstration of LLIN use at distribution points with IEC sessions. IPC led by VHTs was planned and implemented. However, the use of IEC materials including posters, leaflets, laminated sheets, and fact sheets, was not fully executed and relied on variable supplies from previous campaigns. A final strategy articulated in the guidelines was the inclusion of primary schools to be sensitized and co￾opted within the campaign. This strategy was not executed systematically during the campaign but had some traction in select districts e.g. Soroti where some anecdotal reports suggest an impact on net use by children. While the training content for the UCC was thorough, it is unclear to what extent it was thoroughly administered by VHT given varying backgrounds in levels of education and a lack of completely available job aids. The management of UCC BCC activities was highly centralized, largely managed by implementing organizations and not tightly coordinated by government. This led to a reliance on mass media, which was secured by one agency from Kampala. From stakeholder interviews, at both national and subnational levels, it was felt that the BCC strategy would have benefitted from greater decentralization to limit, for example, lack of alignment with net delivery and radio messages as occurred with some waves of the campaign (wave 6). The primary target audience was the entire population in the UCC, reached primarily by VHTs who served as an essential key to the success of an exercise of this scale. An additional key feature of success 29 was the mobilization of sub-national leaders. While a follow-on post-campaign strategy was mentioned by stakeholders as essential, it was not implemented. Implementation of BCC Activities Due to the pooled nature of the BCC funding (PMI and Global Fund), two implementing agencies, Stop Malaria Uganda and Malaria Consortium were responsible for BCC. Each agency was initially responsible for different regions of the country as specified in their operating agreements; this was then changed during the execution to focus activity on each wave of distribution. Teams of supervisors and coordinators were assigned to the districts to lead training for social mobilization efforts among district leaders and trainers to be further led at the village level. Implementation was not carried out as fully as planned in the strategy and implementation plans. Mass media, below-the-line materials and IPC were all anticipated. However, during implementation a key constraint in implementation was an insufficient budget, which created compromises, e.g. the decision to centralize all mass media purchase. An advertising agency was commissioned to create and air radio advertisements and talk show formats. A media plan was made available to the evaluation team, but it is not clear if indeed this plan was implemented, nor is it comprehensive for the whole campaign. From interviews with the advertising agency, it is likely that the weight of the radio campaign was lighter than planned, since it was not possible to fund the planned heavyweight blitzing of advertising at the key times of registration and distribution. There were occasional problems in misalignment of advertising and on￾the-ground registration or distribution. Given donor fund restrictions, a further constraint on the radio campaign was its limitation to larger radio stations, monitored by Ipsos, in lieu of smaller local stations. The expenditure on BCC was heavily weighted toward mass media rather than IPC, which has less power to effect the population at the lowest levels. While TV was not used for the UCC specifically, previous national malaria television advertising may have been running as public service messages. Film vans were also not used systematically – there was no NMCP coordination of this resource for the campaign. Below-the-line materials created were limited to UCC T-shirts for VHTs. There was no public relations and PR management – any free press came from the media, rather than campaign initiated. Faith groups and schools were not co-opted systematically, as these were not encouraged from the center as planned. The UCC did however have top-level advocacy directly from the office of the President. Both at launch and at campaign completion, President Museveni attended the events organized and spoke passionately and on-message about malaria and net use. At local level also, implementation of key activities was coordinated by the Regional District Commissioner, supported by monitors and encouraged by the presence of the Medicines and Health Service Delivery Monitoring Unit, a specialist unit, under the offices of the President to monitor all health expenditure in the country. This advocacy was able to enhance the awareness about the UCC and its BCC messages. Quality of BCC Materials A script was made available for one radio advertisement and two talk show guides (see Annex VIII). The evaluation team was not given access to any pre-testing results for development of these advertisements, to better understand how the target audience received them. However, from the review a number of concerns emerged from the scripts including scare tactics and a number of negative assertions, rather than more positive behavior change messages, which are generally thought to be more effective for a general audience. The advertisements and scripts state inaccurate statistics about the number of people dying from malaria. Otherwise, the scripts for the talk shows appear comprehensive. However, given the quantity of information, IPC would be more effective in ensuring appropriate behavior. The job aid and home visit script were developed for the targeted campaign prior to the UCC. The sample given to the evaluation team is written in English, it is not known how many languages it was translated into, or how widely this 30 was distributed to, and used by, VHTs during the UCC. Net Use Outcomes When evaluating the use of nets and ITN it is essential to differentiate between the non-use of nets due to behavioral reasons and non-use due to a lack of access to a net in the household. Table 8 presents the use of ITN from the four representative household surveys (2006 to 2014) and shows a steady increase of both the ITN use last night and access indicators (access to an ITN within the household). Accordingly ITN use, provided there was access, was high at 75% in the first three surveys and then increased to 84% after the UCC. Although the surveys were done at different times of the year, there was no indication that ITN use if access varied significantly with the rains (see Annex VII). TABLE 3: ITN USE IF ACCESS TO ITN BY YEAR AND ITN USE BY TARGET GROUPS Indicator Population group DHS 2006 MIS 2009 DHS 2011 MIS 2014/15 Used ITN last night 6.9% 24.2% 34.1% 68.6% Had access to ITN in household 9.9% 33.8% 47.0% 82.1% Use of ITN if access 74.7% 75.7% 74.6% 83.6% Children <5 yrs. 9.6% 32.8% 42.8% 74.3% Children 5-14 yrs. 4.4% 17.6% 29.0% 62.4% Women 15-49 pregnant, not head 9.3% 46.1% 46.6% 74.7% Women 15-49 not pregnant, not head 9.9% 30.6% 39.5% 72.7% Males 15-49 not head 4.9% 13.6% 21.4% 51.5% Head of household 9.6% 30.2% 29.6% 76.5% Age 50+ not head 5.0% 24.6% 26.8% 69.8% Table 3 also presents ITN use by specific population sub-groups including the target groups of children under five and pregnant women. Even when population access to an ITN was still low, young children and pregnant women always had higher use rates with women in reproductive age but not pregnant and heads of household close behind. However, as access increased from 47% to 82% after the UCC the previous “non-users”, namely older children, men in reproductive age and persons 50 years or older, showed the steepest increases in use, suggesting that the main reason for non-use had been lack of access and not a principle unwillingness to use. This is more clearly seen in Figure 15 where ITN use by age group, gender and household supply with ITN is compared between the four surveys. When there are not enough ITN in the household there is i) a significantly lower ITN use for age groups 5-19 years and 50+ years, and ii) a clear gender imbalance favoring women in the age bracket 10-49 years. When there are enough ITN as defined by the one ITN per two people indicator, the gender difference is much lower although still favoring women between 10-49 years and differences in use by age groups have largely disappeared. More importantly, in the group of households with at least one ITN per two people the age pattern equalizes. Over the last eight years a strong net use culture has developed in Uganda. 31 FIGURE 3: ITN USE BY AGE, GENDER AND HOUSEHOLD SUPPLY WITH ITN Specific ITN Use Patterns The intra-household supply of nets determines net use. In Uganda, the mean users of nets in the household have remained stable in between representative household surveys. However, through statistical modeling, when there is greater availability of nets in the household, more single-users of nets exist, the assumption of two persons sharing a net is no longer true and some potential users are no longer covered likely related to cultural reasons, e.g. adolescents not sharing a net. This is described fully in Annex VII. Heads of household and people from the poorest wealth quintile also had consistently higher ITN use and people from households with a female head had a lower likelihood of using an ITN (all other things being equal) with an OR of 0.62. Compared to the East Central region as reference the North East and Mid￾West had significantly higher ITN use and Central 1, Central 2 and especially the South-West had lower ITN use. The latter was mainly driven by the high altitude areas in the South-West: in areas below 1,350 meters ITN use if access was 78%, between 1,350 and 1,550 meters 74% and above only 61%. Other variables such as urban residence, household size or education of the women interviewed for the BCC section of the MIS questionnaire were not significant in the models. There was also no indication of a lower use of ITN in the households that received IRS in the Mid-North region. Although in the DHS 2011 there had been a marginal difference of about 7%-points (73% vs. 81%) in ITN use if access, this difference was not visible in the MIS 2014/15. Exposure to BCC and Attributable Effects During the 2014/15 MIS, 93% of women in reproductive health were interviewed regarding exposure to 32 any net related BCC in the last six months regarding their knowledge about malaria transmission and ways to prevent malaria (N=5,322). While exposure to IPC was very similar across education or wealth background, exclusive exposure to IPC only was more common among women with poor education and from the lower wealth quintiles (Figure 15). In contrast, exposure to media channels significantly increased with increasing education and wealth. This pattern was confirmed in multi-variable regression models, which showed that media exposure also increased with increasing age, but IPC exposure did not. Variables that did not influence exposure to either channel were urban residence, gender of head of household, family size, ownership of any ITN or UCC nets and the different UCC waves7 . Exposure to media was particularly high in Kampala, Central regions and South-West and particularly low in North East and Mid-East. IPC was strongest in North East and West Nile and lowest in Central regions. FIGURE 4: EXPOSURE TO BCC CHANNELS BY EDUCATION AND WEALTH Additional details are available in Annex VII. Multi-variable logistic regression showed that neither recent exposure to net related BCC messages nor good knowledge on malaria transmission or prevention was directly associated with ITN use in the current situation in Uganda. Discussion The evaluation team encountered significant limitations in adequately assessing the implementation of BCC activities as one of the key partners of the BCC efforts, the Stop Malaria Project, was no longer active, and it proved impossible to obtain comprehensive records or find people with institutional memory. However, what could be found is in keeping with the findings of the process evaluation of Phase I and, therefore, the findings can be considered valid. 7 Only waves 7, 8a, 8b, and part of 6 fell into the 6 months bracket for the BCC question in the MIS 2014/15. 33 The results from the UCC evaluation showing a very high ITN use are consistent with what has been reported in other countries when the “use gap” was considered [19]. A process of growing net use culture has been seen in similar magnitude following the UCC in Tanzania (A. Kilian, unpublished) and has also been described for Kamuli district in Uganda as part of a cluster-randomized trial on the effects of repeated home visits on net hanging and use. While the visits had no measurable effect, the general exposure to a socio-cultural environment that accepted and promoted ITN use created a positive effect on ITN use in all study arms, including the control [20]. Evidence from the FGD confirms the existence of a strong net use culture, namely that “people use the nets” found in all four sites visited. With respect to potential misuse of nets respondents mentioned that other uses (such as making ropes to tether goats, reinforce roofs or fences, constructing bath shelters, protecting chicken or in some cases fishing) are predominantly for old nets which is in keeping with other quantitative [21] and qualitative studies [22]. There also was repeated mention of a certain dislike of net users that obviously referred to Olyset nets (rough texture, too large mesh, difficult to wash, shrink when washed). However, this had no correlation in the MIS data where there was no difference in use of any of the LLIN brands. The analysis showed no impact of recent BCC exposure including that from the UCC activities on actual net use because it was already high. This is in keeping with the steady increase in net use since 2006 (see Figure 3) and suggests that the build-up of a net use culture is a gradual change of social norms following repeated exposure through multiple channels rather than the effect of one single message. The finding from the Uganda UCC seems to somewhat contradict findings from other countries [23, 24] that did find a significant impact of BCC on ITN use, but in these cases the assessments were done earlier in the development of a net use culture. In addition, the capture of BCC exposure in the 2014/15 MIS was limited in the time scale (six months) and scope of BCC channels, and cannot be considered a comprehensive assessment of BCC activities. Conclusions • The BCC activities were poorly implemented, focused on media channels rather than interpersonal communication which could have better reached the rural poor. • BCC activities were highly centralized and would have benefitted from decentralization and involvement of district level actors. • As far as can be stated from the limited examples of materials the evaluation team had access to, the quality of some of the materials for radio communication was poor, using a scare tactics approach rather than encouraging correct behavior. • The target audience was the general population and as such adequate for a UCC. • In spite of the limitations in BCC implementation, ITN use was very good and favored the most vulnerable population groups of young children (boys and girls alike) and women in reproductive age. As an effect of improved ITN access older children and men are using ITN, which represents a significant progress achieved by the UCC. • There is a sufficiently strong net use culture that does not depend on a single message or BCC exposure but rather on long-term experience and reinforcement through interpersonal communication. III. Health Outcomes: Malaria Morbidity, Parasite Prevalence & Anemia Findings HMIS Reporting Completeness and OPD Attendance The reporting completeness increased from 76% in 2006 to 88% in 2010 and 95% in 2015. However, there was considerable fluctuation in the reporting completeness and a significant drop to as low as 55% 34 in November 2011 during the transition from HMIS to DHIS2. In general the transition to the electronic, district-based DHIS2 system improved data quality and completeness significantly. In total the database reported 253 million new OPD visits in the public sector between 2006 and 2015, which, after adjustment for under-reporting, increased to 298 million. The OPD attendance rate was essentially identical for boys and girls less than five years of age, but differed between males and females in the age-group above five years with male attendance being only about two-thirds of the female attendance. For children under five the utilization rate was between 1.5 and 1.75, which is at or above the WHO recommended level of 1.5 visits per child and year. Similarly, the utilization of public facilities was around the recommended 1.0 or above for women, but only 0.6 to 0.8 for men. There was a marked increase of all utilization rates during 2011 which is an artefact caused by inclusion of more reporting non-government facilities in the DHIS2. Interestingly, there was a slight decline in the utilization rates 2012 to 2015 for all age groups and genders. The proportion of reported malaria cases out of all new OPD attendances was initially around 60% for children under five and between 40% and 50% for patients above five years. The ratio between OPD attendance and malaria cases was very similar for males and females indicating that the lower number of malaria cases in the database was not a result of a lower malaria incidence in men but due to their lower attendance. The OPD-adjusted malaria incidence, which assumes a constant utilization rate over time, was almost identical for men and women. This suggests that the exposure to malaria is the same for men and women but that men either do not attend health facilities as often,or obtain their treatment more frequently from the private sector not captured in the database. Trends in Malaria Incidence Between 2006 and mid-2010 the malaria incidence for children was around 900 reported cases per 1,000 children per year, or close to one episode per child and year. Considering that this comprises only those cases attending public facilities and that DHS surveys suggest that these are only 30-40% of all fever cases in children, the overall clinical malaria incidence in this period can be estimated to be between 2.0 and 2.5 episodes per child per year. Based on the slide positivity rate of around 50% during this period, the confirmed malaria incidence in children under five is estimated to be 1.0 to 1.25 per child and year. Incidence for adults in the HMIS database is about half that of the children but projection to national level is not possible as the proportion attending the public sector is not known for this age group. The trend in OPD-adjusted malaria incidence over time is shown in Figure 5 in connection with the two national LLIN distribution campaigns: the targeted campaign of 2010/11 and the UCC. Although there is significant seasonal variation in the monthly and quarterly data, the trend in the moving averages show a declining trend more clearly visible in the quarterly data. Starting in mid-2010 there was a modest decline in incidence to about 750 cases per 1,000 for children and 400 for persons above five until about mid￾2013. Thereafter, the decline was steeper for both age groups reaching about 600 per 1,000 for children and 300 for older persons by early 2015. 35 FIGURE 5: TREND IN MONTHLY AND QUARTERLY OPD-ADJUSTED MALARIA INCIDENCE BY AGE PALLID LINE MONTHLY/QUARTERLY DATA, BOLD LINE 6-MONTHLY OR 3-QUARTERLY MOVING AVERAGE, U5&PW TARGETED CAMPAIGN Analyzing incidence data by census shows that incidence is highest and incidence decline the lowest in the East, followed by the North, while Central region and West are significantly lower and show a more consistent long-term decline in malaria incidence. These relationships reflect the situation in malaria transmission intensity and epidemiology in Uganda from historical data [1]. However, in all regions there is an acceleration of decline that starts between 2013 and 2014. At national level there was a 33% and 37% reduction in malaria incidence between 2007-15 for children under five and older persons respectively. The UCC period showed consistently the most pronounced decline of between 24% and 25%, meaning that 67% to 72% of the long-term decline came during the UCC period. In order to obtain a clearer picture of the impact of the UCC malaria incidence data for each wave was grouped into the 12 months preceding the UCC and the 12 months after. Results for the aggregate data are shown in Figure 6 and demonstrate a statistically significant six fold increase of the rate of decline (coefficient from regression analysis). However, the situation varied by wave or region and the declines induced by the UCC were less pronounced in the East and North than in the West and Center. Further details are presented in Annex VII. 36 FIGURE 6: MALARIA INCIDENCE BY AGE COMPARING 12 MONTHS BEFORE AND AFTER UCC* * Numbers show gradient (coefficient) from linear regression analysis with 95% CI in brackets, and p-value refers to the change in gradient before against after UCC Impact of Indoor Residual Spraying in the North Ten districts in the Mid-North region received two annual rounds of Indoor Residual Spraying (IRS) between 2009 and 2014 (see Figure 42, Annex G), initially using the pyrethroid alphacypermethrin and, due to increasing resistance, switching to the carbamate bendiocarb at the end of 2010. As shown in Figure 7, there was little change of incidence in the first year of IRS followed by a rapid, about 50% decline of reported malaria incidence compared to the other districts in the Mid-North MIS region. After the initial sharp drop incidence curves for the IRS and non-IRS districts seem to run in parallel and both show a second very clear decline that coincides approximately with the UCC in this area. In the second quarter of 2015 (more specifically in the months of May and June) a significant increase of malaria cases can be observed that was more pronounced in the IRS districts than the non-IRS districts. This also coincides with the end of the IRS program in the 10 districts and can be interpreted as a “rebound” of malaria incidence to “LLIN only” levels of protection. However, during this time there was a significant “upsurge” of malaria cases observed also in other regions of Uganda, namely West-Nile, Mid-West and Central 1 and 2 (see Annex VII for details) while it was absent in the North East, Mid-East, East Central and Kampala. The increase of cases and test positivity rates is most likely an effect of above normal rainfall in the Western, Northern and Central areas of Uganda in early 2015 due to the El Niño weather phenomenon. Since the increase of cases is only seen in the last two months of the HMIS data time series available for this evaluation, no further assessment on the magnitude or duration can be made. But is it very likely that the increase seen in the IRS districts in Mid-North at the end of the IRS program is a combination of the two described mechanisms. 37 FIGURE 7: TREND IN QUARTERLY MALARIA INCIDENCE IN IRS AND NON-IRS AREAS IN MID￾NORTH REGION PALLID LINE QUARTERLY DATA, BOLD LINE 3-QUARTER MOVING AVERAGE OF OPD-ADJUSTED INCIDENCE Parasite Prevalence and Anemia Malaria parasite prevalence in children 6-59 months reduced at national level by 55% from the 2009 MIS to the 2014 survey (Table 4) with the largest reduction observed in the Central region (63%) followed by Western (56%), Northern (52%) and Eastern (49%). Reductions occurred at all ages within the 6-59 months bracket. When analyzed by MIS region (Figure 8) three observations can be made: i) in keeping with the reduction seen in malaria incidence in the IRS districts in Mid-North there was a dramatic 87% reduction in parasite prevalence in this area from 63% to just 8%; ii) in the Kampala area where in 2009 parasite prevalence was already low at 6%, a further dramatic reduction to only 0.4% was seen, which means that in Kampala malaria is at negligible levels; iii) the least reductions were seen in East Central region where the lowest community and household level ITN ownership coverage had also been observed. TABLE 4: CHANGES IN MALARIA PARASITEMIA AND ANEMIA BY AGE OF CHILD 38 Age group (in months) Malaria parasitemia Any anemia Moderate/severe anemia 2009 2014 2009 2014 2009 2014 6-59 (95% CI) change 44.7% (39.9 – 49.7) 20.1% (17.3 – 23.2) -55% 62.2% (58.9 – 65.4) 53.0% (50.5 – 55.4) -15% 9.6% (8.0 – 11.6) 4.6% (3.8 – 5.7) -52% 6-11 31.8% 11.2% 80.8% 80.0% 20.5% 10.3% 12-23 37.1% 14.7% 74.2% 68.3% 15.5% 6.6% 24-35 45.1% 22.2% 64.8% 53.3% 9.9% 4.2% 36-47 49.4% 21.5% 52.3% 40.8% 6.2% 3.2% 48-59 53.1% 26.0% 49.1% 38.2% 2.1% 2.2% For anemia the overall reduction of any anemia was only moderate, from 62% to 53%, but when hemoglobin levels of less than 8g/dL were considered, the reduction from 2009 to 2014 was in a similar range as parasite prevalence at 52% (Table 5) with the biggest gains made in the IRS districts in the Mid￾North and in the Central Regions, and the least again in East Central. Analysis by age group shows that the overall reduction of any anemia was mainly in older children and due to a reduction in moderate anemia, while infants showed a significant reduction in severe anemia but not so much in moderate anemia. FIGURE 8: CHANGES IN MALARIA PARASITE PREVALENCE BY MIS REGION AND IRS Discussion Data from routine health services used in this evaluation tend to be difficult to interpret because of noisy data due to recording inaccuracies, data entry errors and trends in reporting completeness, OPD attendance and population growth that tend to obscure changes in malaria incidence. However, the 39 evaluation team went through a very rigid protocol of data cleaning and preparation and adjusted for all potential influences so that the resulting trends can be considered a true reflection of trends in the data. But since only public sector data was available, which does not cover the entire population, it cannot be excluded that comprehensive data for the private sector (which does not exist) would show a slightly different picture. However, the evaluation team believes that this is not very likely. The decline of malaria incidence by 33-35% observed in Uganda is very similar to the decline of 37% reported in the most recent World Malaria Report [29]. Declines of parasite prevalence of 40% for Africa have been estimated for the period 2000-2015 [30]. Country specific data from Malawi showed a 30% incidence reduction that is attributable to ITN [32] and from Zambia a 53% decline in parasite rates was reported [31], very similar to Uganda. Decreases of 70-80% in malaria mortality and morbidity have been reported from selected health facilities in Ethiopia [33] and Rwanda [33] though these are countries with much lower transmission potential than Uganda. Considering historical data from Uganda, such as those from Kamwenge District (Annex VII, A. Kilian, unpublished) show that tremendous reduction in parasite prevalence has occurred in Uganda since the 1990’s that are not even captured in the MIS data. Conclusions • The Uganda UCC significantly contributed to an impressive decline of malaria incidence of 33-35% since 2007, a 55% reduction in malaria parasite prevalence and 52% reduction of moderate/severe anemia. • Although an exact proportionate contribution of the UCC to these declines cannot be made, the evidence is strong that the UCC was a significant part of the success. • Parasite prevalence in the Kampala area was below 1% in 2014 among children under five giving evidence that in this urban environment malaria is now marginal. IV. Effects, Costs and Cost-effectiveness Findings UCC Outcome of ITN Ownership As the implementation of the UCC was done in waves and stretched over a considerable time period, it is important to keep in mind the geographical spread and time since distribution at the time of the MIS 2014/15. Waves rolled essentially from the East to the Central area, to the West and South West and then to the North, ending in the Kampala and Wakiso area. The time since distribution was longest for the pilot with 26 months, followed by waves 1 and 2 with 15-18 months. Waves 3-5 had a 10-12 months period between UCC and MIS, waves 6-8a around 5 months and wave 8b less than 4 months. Since the type of LLIN distributed might impact on user acceptability and use and/or the durability of the product, the team attempted to map out the various brands used in the UCC. Overall, two polyester-based brands (Permanet 2.0 and 3.0, Dawa Plus 2.0) and two polyethylene-based LLIN brands (Olyset, Royal Sentry) were distributed. However the bulk of the nets were Permanet 2.0 (46%) and Olyset8 (32%) and only smaller contributions from Dawa Plus (17%), Royal Sentry (5%) and Permanet 3 (1%). As far as could be established from distribution records and triangulated by the brand identification in the MIS, the North and East received predominantly Olyset; central region Permanet; a mix of Permanet and Dawa in 8 Most likely the new version with improved knitting pattern to avoid “unraveling” of small holes. 40 the South West, West and West Nile; and a mix of Permanet, Olyset and Royal Sentry in the West and SouthWest in wave 6. Household Ownership The household ownership of any net at national level had increased from 34% in 2006 to 74% in 2011 to 94% in 2014/15 (Figure 9). The proportion of ITN among all nets was only 45% in 2006 and 65% of the ITN were LLIN, 35% conventionally treated. This changed dramatically in favor of LLIN, and in 2011 78% of all nets were ITN and 98% of the ITN were LLIN, meaning that by that time “dipping” of nets had essentially disappeared. In 2014/15 95% of all nets were ITN and 99.8% of ITN were LLIN, showing that now untreated nets play only a very marginal role in Uganda. Untreated nets are mainly found in the central area where they comprise 8-13% of the net crop while in all other regions their contribution is between 0.5% (North East) to 3.7% (South-West). Nationally, only 3.5% of households owned only untreated nets and outside the central region this was the case in only 1.5% of households. While increase of household ownership of any ITN and population access to ITN increased in parallel over time (Figure 9), there was initially a slower growth of households with enough ITN for all members – a reflection of the targeted distribution strategy during that time that favored geographical spread (at least one ITN) over intra-household net saturation (enough ITN). 41 FIGURE 9: TREND IN ITN OWNERSHIP INDICATORS 2006 TO 2015. DASHED LINES REPRESENT COVERAGE WITHOUT THE UCC NETS Population Access The trend over time for the most important of the ITN ownership indicators, population access by the MIS regions, is shown in Figure 10 and reflects the history of ITN distribution in Uganda. Before the targeted campaign of 2010 the North, including Mid-East, had the highest coverage, due in part to the focus on internally displaced people. Kampala was also initially high where availability of nets from the retail market was high – subsidized and at cost. During the 2010/11 campaigns West-Nile and Mid-West and Central 19 made the biggest gains while Mid-East and North East stagnated or even dropped. It is worth noting that throughout the time period East Central had the lowest rates of access and this was true again in 2014/15, when only 72% had access to an ITN, 10%-points lower than the national average. 9 In four districts of Mid-West and Central 1 the campaign actually was UCC instead of targeted due to a project funded by the British INGO Comic Relief. These were Hoima, Kiboga, Kyankwanzi and Buliisa. 42 FIGURE 10: TREND OF POPULATION ACCESS TO ITN BY MIS REGIONS Without the contribution of the UCC only 16% of households would have had any ITN in 2014/15 and only 11% of the population would have had access. Conversely, with only the UCC nets considered (i.e. ignoring any net from other sources which is the underlying assumption for the quantification of the UCC nets when calculating it as population/1.8 [6]) 83% of households had at least one ITN, 54% enough ITN and 75% of the population had access to an ITN within the household. This implies that there were some households that had both UCC nets and non-UCC nets. Nationally these were 8%, with another 8% having only non-UCC nets, leaving 74% that had UCC nets only. By far the largest contribution of non￾UCC nets was in Kampala where 45% of households owned any non-UCC nets and only 41% had exclusively UCC nets. This was followed by the North East (23%), Central 1 (21%) and East Central (20%). The lowest contribution of non-UCC nets was found in Mid-West (85) and Mid-East (9%). When the analysis was done by waves the largest contribution of non-UCC nets was in Kampala/Wakiso (wave 8b, designed to limit one or two LLIN per household) followed by the pilot districts where only 40% owned UCC nets only. Ownership of UCC nets for waves 1 and 2 was still very high at 89% despite this being 15-18 months after distribution, i.e. half-way towards the next scheduled distribution. Community Coverage Coverage at community level is critical for the development of a mass-effect of ITN that provides protection for people without an ITN. At national level 94% of households resided in communities with at least 80% ITN coverage and 71% in communities with 95% coverage. This is very high and much higher than what was found in Nigeria in the MIS 2010 for states that had had a recent campaign, where the rates were 66% and 31% respectively (A. Kilian, unpublished). This suggests that spatial coverage was very high after the UCC (details see Annex VII). Intra-household Supply Indications of LLIN oversupply were noted with a higher population registered than expected, as previously described in the data quality section. The ITN ownership analysis further supports this possibility and suggests the possibility of undersupply as well. The NetCALC model was used to compare expected coverage as a function of time since distribution. Results are shown in detail in Annex VII. The analysis shows that in particular three waves of the UCC (waves 1, 5 and 8a) were oversupplied and that the pilot districts were undersupplied at the lower limit of what was expected. This corroborates the data quality accuracy assessment as well. The most important determinant of the intra-household supply with ITN was household size as, shown in 43 Figure 11, and here two opposite trends can be seen. On the one hand the likelihood of having no ITN at all was highest among small households and decreased with increasing household size. This could be a reflection of the probability of being registered, given that with large households it is more likely that somebody was present at the day of registration than for a household of three or less persons. On the other hand small households had the highest rate of oversupply (26%) while large households were almost never oversupplied, and only 33% had enough nets for all members. In a logistic regression model, region was the only other variable associated with a household having no ITN at all (likelihood in the Central region higher than North and West) while oversupply was – in addition to household size – positively associated with being in the highest wealth quintiles and negatively with having children under five in the household. When household supply with ITN or UCC nets was categorized into five groups to consider under-, sufficient and oversupply, it can be seen that in addition to the 62% of households with at least one ITN for every two people another 19% (81% in total) had almost enough ITN for all members, and 21% actually had more than they would need based on the allocation algorithm. For UCC nets only the figures were lower, with 72% having almost enough and 16% “too many”. FIGURE 11: INTRA-HOUSEHOLD SUPPLY SITUATION WITH ITN AND UCC NETS BY HOUSEHOLD SIZE The Phase I report stated that some informants reported net limitations (maximum of four) for large households. Qualitative assessments in this evaluation confirmed that many of the large households in all four visited districts reportedly received less than the one ITN for every two people. Some households were missed entirely and generally the concept of universal coverage was misunderstood as many expected that each person would receive their own ITN. The intra-household supply by region and UCC wave confirms the picture already seen in the previous analysis that East Central region, the pilot, and wave 8b were the most undersupplied. Other Sources of Nets 84% of households received nets from the UCC. The sources of nets for the remaining 16% of households included nets from the retail market, shops and marketplaces (9%), from a previous campaign, or from a public or INGO health facility (4%), likely ante-natal care services. But overall this distribution channel was far below its potential of reaching up to 30% of households, which shows that to date delivery of ITN through routine services is not yet working in Uganda. Nets from the private sector were also found to 44 some extent in all other regions, especially the North East (8%) and East Central (9%), which includes the pilot districts. When looking at sources of nets by region, the highest contribution from private sector nets was highest in Kampala (42%) and Central 1 (19%) (including Wakiso). This comes as no surprise as here the UCC did not supply sufficient nets and LLIN are available for purchase. When analyzed by wave the pilot area had a 21% contribution of private sector nets, suggesting that households filled the existing gaps by buying additional nets where possible. Equity Measures of equity were taken for all net ownership indicators and compared across the most recent four representative household surveys. Equity was also compared between nets obtained by the public or private sectors with the exception of the DHS 2011 data for which this distinction was not possible. Results are shown in detail in Annex VII. There is a clear trend towards perfect equity over time. As the ownership of nets and ITN increases, the initial slightly pro-rich distribution is increasingly shifted towards equity for both the equity ratio and the concentration index, and in 2014/15 these were 1.00 and - 0.002 for the ownership of any ITN respectively. Outlook for Future Distributions By using the second NetCALC model, projection of ownership indicators under various assumptions of median net survival and underlying population can be compared with results from household surveys. This additional data point may help to assess the most likely “true” population size at the time of the UCC. By estimating this, the likely ownership coverage at the time of the next UCC currently planned for 2016/17 can be inferred. The population size in the model was varied between the census estimates for 2014 of 34.8 million, the result of the UCC registration records of 41.0 million, and the estimate of 36.9 million based on a 5% under-counting in the census. The first observation is that when the 22.3 million LLIN distributed in the UCC are entered on top of the previous distributions, the model indicates an excess of nets in the system that varies between 0.6 and 7.3 million with a median of 4.3 million nets. All excess nets in the model were removed in order to allow a conservative estimate of future coverage and net crop. Based on the three population scenarios and considering the two indicators “Household has any ITN” and “population access to ITN within the household” six projections were made, each with the expected coverage curves for median net survival varying between 2.5 and 3.5 years. These curves were compared to the respective survey estimates and their 95% confidence intervals. The projection with the best fit is shown in Figure 12 while all scenarios are presented in Annex VII. For all estimates it was observed that “any ITN” coverage for 2009 and 2011 was underestimated in NetCALC while the “access” indicator showed a much closer fit. This was caused by the previously described effect (Figure 8) that during the phase of primarily targeted distributions before the UCC there was a disconnect between increase in geographical coverage of “at least one ITN” and intra-household saturation with ITN (enough ITN) that cannot be represented in NetCALC as the model always assumes a balanced and simultaneous increase of these two indicators. Considering not only the coverage results from the four DHS and MIS but also the estimate of coverage in 2014/15 without the UCC nets, and assuming further that the most realistic median survival in Uganda is between 2.5 and 3.0 years at the moment [13], the best fit between model and data is achieved based on the census population in 2014 of 34.8 million while a population estimate of 36.9 million still appears plausible. However, the results suggest that assuming a population of 40.0 million at the time of the UCC would only be possible if median survival of nets in the past 10 years was consistently at 3.5 years or above. 45 FIGURE 12: PROJECTED POPULATION ACCESS 2006-2018 COMPARED TO SURVEY RESULTS RESULTS FROM NETCALC WITH AND WITHOUT UCC AND ASSUMING POPULATION OF 34.8 MILLION The expected ITN coverage with any ITN from the models varies between 42% and 55% for 2016 and 21% and 37% for 2017. This translates to an ITN crop of between 10 and 13 million at the end of 2016 and between 6 and 8 million in 2017 (Table 5). In order to see how this compares to the estimate of numbers of ITN in the population from the MIS 2014/15, the mean number of ITN owned per household was multiplied with the total number of households (assuming 4.9 persons per household as has been reported). As shown in the table, the figures for the 2014 ITN crop compare quite well with the estimates from the survey with the best fit for a population of 34.8 million in 2014. This suggests that the models have a reasonably good fit and that the likely median survival of nets is probably just below 3.0 years on average. TABLE 5: EXPECTED NET CROP BY YEAR ASSUMING A 3 YEAR SURVIVAL AND ESTIMATES FROM MIS 2014/15 2014 2015 2016 2017 NetCALC Population 34.8m 18.4m 14.7m 10.4m 6.4m Population 36.9m 19.6m 15.8m 11.4m 7.1m Population 41.0m 21.8m 17.8m 12.9m 8.1m MIS 2014/15 Population 34.8m 17.7m Population 36.9m 18.7m Population 41.0m 20.8m Given that the concept of the repeated mass campaigns is to distribute the full amount of the need for universal coverage (calculated as population divided by 1.8), i.e. assuming each time that there is no viable net left in the population, this approach, by design, will replace some nets that are still in usable 46 condition and may provide one or two years more of useful life. It has previously been estimated from modelling that if there still is a coverage of 20% or more of nets that are at two years old or less, a campaign is not cost-effective [16]. Alternatively, one can estimate the need for ITN in a hypothetical system of comprehensive continuous distribution (CD) through a mix of channels as suggested by WHO as a medium-term objective [17]. This is presented in Table 6 based on NetCALC projections for two campaign cycles assuming that in 2014 full universal coverage is achieved and maintained at 100% and that median survival is 3.0 years. Because in the first years after a UCC all nets are new and initial loss is slow, the replacement need in the first three-year cycle from NetCALC is lower than the steady state, which is reached in the second cycle. Nonetheless, the model shows that the need for ITN would be 10% lower in the first 6 years under CD compared to UCC, which would translate to ~9 million ITN under the assumption of a population near 35 million in 2014. TABLE 6: PROJECTED NEED FOR ITN COMPARING UCC AND CONTINUOUS DISTRIBUTION Population size Distribution 2015-17 2018-20 Total 34.8m CD 13.1m 21.4m 34.5m (- 9.2m) UCC 20.9m 22.8m 43.7m 36.9m CD 14.0m 22.7m 36.7m (-9.8m) UCC 22.2m 24.3m 46.5m 41.0m CD 15.5m 25.2m 40.7m (-10.8m) UCC 24.6m 26.9m 51.5m Interestingly, several respondents in the qualitative KII and FGD (e.g. the deputy CAO in Hoima and communities in Soroti) mentioned that they thought that a “continuous” distribution would be preferable, particularly to replace nets when needed and to ensure that all households are reached, as in such a system the likelihood that a household might be missed would be much lower. Cost and Cost-effectiveness Base-case cost analysis resulted in an estimate of approximately 95 Million USD in financial costs and 100 Million USD in economic costs for the distribution of ~22 Million LLINs across all waves of the UCC campaign in Uganda. Table 7 shows the total costs of the program by line item category for both the economic and financial analysis as well as the relative contribution of each line item. Not surprisingly, the net procurement was the largest cost driver with 83% of the financial and 91% of the economic cost. However, within the delivery cost alone the line item of procurement and supply management was the largest contributor comprising 30% of the delivery cost. So, in total, 95% of the economic cost of USD 100 million was linked to the procurements of the nets. TABLE 7: ECONOMIC AND FINANCIAL COST OF UCC Line Item Financial Economic Total cost USD % of total % of delivery Total cost USD % of total % of delivery Human Resource 2,054,187 2.1% 15.4% 2,054,187 2.1% 15.4% Training 1,004,292 1.0% 7.5% 1,004,292 1.0% 7.5% Procurement and supply management 4,053,283 4.1% 30.4% 4,053,283 4.1% 30.4% Communication material 1,177,482 1.2% 8.8% 1,177,482 1.2% 8.8% Technical assistance 69,532 0.1% 0.5% 69,532 0.1% 0.5% Monitoring and evaluation 2,264,340 2.3% 17.0% 2,264,340 2.3% 17.0% Planning and Administration 2,447,230 2.5% 18.3% 2,447,230 2.4% 18.3% 47 Overheads 283,747 0.3% 2.1% 283,747 0.3% 2.1% Nets 81,678,867 82.8% 90,499,036 90.5% Grand Total 98,682,965 100.0% 99,981,976 100.0% Distribution Only 13,354,097 13.5% 100.0% 13,354,097 13.4% 100.0% Human Resource 2,054,187 2.1% 15.4% 2,054,187 2.1% 15.4% Table 8 shows the total cost per outcome in base-case analysis for both financial and economic analyses as well as for distribution only, i.e. excluding LLIN commodities. TABLE 8: ECONOMIC AND FINANCIAL COST PER OUTCOME Output or Outcome Cost per outcome in USD Financial Full Cost Economic Full Cost Financial Cost Distribution Economic Cost Distribution Nets Distributed 4.25 4.49 0.60 0.60 Net Year 1.42 1.49 0.20 0.20 Person Years Protected 0.71 0.74 0.10 0.10 Children Years Protected 1.41 1.49 0.20 0.20 Child Deaths Averted 257.38 270.78 36.17 36.17 DALYs Averted 7.80 8.21 1.10 1.10 Sensitivity Analysis In order to test the basic assumptions of the cost and effectiveness models presented above, a one-way scenario analysis and probabilistic sensitivity analysis was implemented. The main assumptions investigated were surrounding the numbers of LLINs distributed and their cost, as well as the translation of numbers of nets distributed to the number of deaths and DALYs averted. Under standard NetCALC assumptions and using census data for population estimates as opposed to 2014 registration data, the standard NetCALC formulation estimates that approximately 5 Million excess nets were distributed during the 2014 UCC. The evaluation team used the NetCALC implementation to determine the excess number of Net Years of Protection provided by distributing those 5 million nets and subsequently the potential cost savings and cost-effectiveness when these 5 million fewer nets were distributed. The results of alternative simulations with 5 million less nets indicates that the coverage under the population scenario using census data would still be expected to have achieved greater than 80% coverage in the majority of simulations and achieve nearly 100% coverage in at least 10% of simulations. These results indicate that significantly fewer LLINs may have been able to be distributed without significant losses of coverage. Consequently, we also examined the cost implications of distributing 5 million fewer LLINs in the UCC. Distributing 5 million fewer LLINs during the UCC would result in a total financial cost estimate of approximately 77 Million USD as opposed to 95 Million in the base scenario, resulting in a cost savings of approximately 18 Million USD. Cost per outcome would increase but only to ~4.64 USD per Net Distributed (economic analysis). Estimates of cost per other outputs or outcomes would similarly increase only slightly. We also varied the discount rate applied in the analysis from 10% to 0%. This had little effect on estimates of the cost per outcome or output, partially because the duration of the program 48 was very short. Reducing the cost per net purchased from ~3.65 USD to ~ 3 USD reduced the cost per outcome and output universally and resulted in cost per net distributed of under 3.78 USD in both economic and financial analysis. In all cases reducing the assumptions around the usage and durability of LLINs used in the program significantly increased the cost per outcome in terms of child years of protection, treated net years and deaths and DALYs averted. However, even in the case where worst case scenario approaches were taken (e.g. LLINs were assumed to last only one year, and be relatively infrequently used, estimates still indicated that cost per DALY averted and death averted fell into a range which would be considered highly cost effective by WHO standards as applied to Uganda (cost per DALY averted ≤ GDP per capita). Discussion The results from the UCC as documented in the MIS 2014/15 compare favorably with results from other countries that have undertaken UCC. The 2011 MIS in Tanzania, undertaken just after the “top-up” campaign, showed 91% of households had any ITN, 55% had at least one for every two people and 80% of the population had access to an ITN within the household. A similarly high access rate as the 82% found in Uganda was also documented in the 2013 MIS in Rwanda (72%) and the 2011 MIS in Madagascar (72%), and in a post-campaign survey in Eastern Ghana (77%). Other post-campaign surveys only reported the coverage with any ITN and these were 88% in Sierra Leone [14] and 97% in Togo [15]. The economic cost per LLIN delivered of 4.49 USD or average annual economic cost of 1.49 USD compares very favorably with reports from other countries. Early reports (2008) from six countries reported annual cost between 2.75 and 4.80 USD [25]. More recent data from Ghana [26] found annual economic cost of 2.90 USD and in Uganda cost of between 2.88 and 3.55 USD were found for previous and local campaigns [27]. The only report with a similar total economic cost is from Togo [28] with USD 4.41 per LLIN delivered, but this was calculated by applied shared cost in an integrated campaign. There is no doubt that economic costs in recent years have come down significantly due to lower prices of the LLIN, only USD 3.65 (c.i.f.) on average for the UCC, which is 1.31 USD less than paid in Ghana in 2012 [26]. But one must also keep in mind that some costs were possibly not included in this evaluation due to the retrospective approach of collecting data, as the relative cost of net procurement is somewhat higher (91%) than was found in Ghana (60%) or Togo (61%). Conclusions • By the end of 2014 Uganda had achieved universal coverage with ITN for the prevention of malaria with over 80% of the population having access to an ITN within the household. This was largely achieved through the UCC, as without it coverage would have been merely 11%. However, the UCC was not as efficient in achieving ITN ownership as it could have been with some areas oversupplied while at the same time also leaving some gaps in supply. Small households tended to be missed more frequently than larger ones and large households were often undersupplied. • Households filled some of the gaps in supply by obtaining nets, mainly LLIN, from the private sector. This was particularly the case in those areas where the campaign was not designed to achieve universal coverage (Kampala-Wakiso) and in the pilot districts where, in addition to undersupplying, the distribution had been more than two years ago. • Routine distribution of ITN through health facilities was much lower than expected and this distribution is as yet not sufficiently utilized in Uganda. • The average annual cost per ITN year was very low compared to historical data at 1.49 USD, largely due to a low commodity price. This made the UCC highly cost-effective with respect to the estimated cost per DALY averted. • The up to 5 million potentially excess nets based on inaccurate registration data were conservatively estimated to have added ~18 Million USD to the cost of the campaign. It appears possible, therefore, 49 that for relatively minor investments in improved registration it might be possible to achieve significant cost savings through better alignment of net procurement and distribution with actual need. • The overestimation of the population and the existence of a considerable net crop of several million additional LLIN still in place by the time the next UCC is planned in 2016/17, show that efficiency and cost-effectiveness can be further improved and that the UCC approach may not be the best strategy to sustain the achieved UC in the long run. 50 RECOMMENDATIONS Based on the findings and conclusions presented above and in more detail in Annex VII, the evaluation team makes the following key recommendations for future ITN distributions in Uganda: • In order to improve efficiency, the National Malaria Control Program and partners should redesign the registration process and the way data is captured and used for quality assurance. This should include: o Involvement of the district level health staff in the registration process as these usually have a sense of the communities they serve and can enhance the feed-back loop for data quality. o Investment in an improved UCC registration database that should consider real-time data entry and availability via SMS or other phone-based systems that will allow immediate plausibility checks on the reported number and can serve as a logistics management tool (nets allocated, received, returned etc.). o Independent data quality audits should be implemented post registration. • The BCC strategy around UCC should be reconsidered and scaled down focusing on the following elements: o Decentralization of the planning and implementation to the districts with stronger involvement of the District Education Officer and the LC and VHT staff. o Focus on interpersonal communication that positively enhances the existing net use culture and presentation of messages on net care and repair. o Clear communication of the allocation rules of nets during the UCC to avoid misunderstanding that each person is to be given their own net. • Future evaluations of LLIN distributions in Uganda should be planned well in advance and start at the planning stage of a UCC or other type of distribution so that necessary information will be available in adequate detail and quality for the post-distribution evaluation. o This applies particularly to the cost data for which cost categories must be defined in advance and consistent data collection tools be used throughout. o Another area where better planning of future evaluations is critical is BCC and standardized records of channels and messages (with target groups) should be prepared during implementation stating location, time, and intensity of activities. • Given the low malaria parasite prevalence in Kampala, in conjunction with high presence of commercial LLIN and obvious willingness of large parts of the population to purchase them, NMCP and partners should consider excluding Kampala and possibly Wakiso from future UCC and instead strengthen routine net distribution channels and possibly community distributions in socially and economically disadvantaged areas. • In the medium to long-term the strategy to maintain universal coverage should be revised to move away from an effective but comparatively inefficient repeat campaign system. Instead, a comprehensive continuous distribution system should be designed and implemented that allows more initiative of the household to “manage” their ITN supply based on actual need and also enables the engagement of the private sector, with markets contributing to achievement of the universal coverage target. 51 REFERENCES 1. Talisuna AO, Noor AM, Okui AP, Snow RW: The past, present and future of use of epidemiological intelligence to plan malaria vector control and parasite prevention in Uganda. Malaria Journa 2015, 14:158 2. Irwig W, Berning M, Marck P, Prell M. Data quality assessment tool for administrative data. US National Agricultural Services, US Census Bureau, US Economic Research Service, Feb 2013, http://www.bls.gov/osmr/datatool.pdf 3. USAID: Data quality checklist – USAID Learning Lab. http://usaidlearninglab.org/sites/default/files/resource/files/Data%20Quality%20Assessment%20Chec klist.pdf 4. GFATM: Routine data quality assessment tool. The Global Fund to Fight AIDS, Tuberculosis and Malaria, Draft July 2008, http://www.cpc.unc.edu/measure/resources/tools/monitoring-evaluation￾systems/data-quality-assurance-tools/RDQA%20Guidelines-Draft%207.30.08.pdf 5. UBOS: National population and housing census 2014, provisional results, revised edition, November 2014, http://www.ubos.org/onlinefiles/uploads/ubos/NPHC/NPHC%202014%20PROVISIONAL%20RES ULTS%20REPORT.pdf 6. Kilian A, Boulay M, Koenker H, Lynch M: How many mosquito nets are needed to achieve universal coverage? Recommendations for the quantification and allocation of long-lasting insecticidal nets for mass campaigns Malaria Journal 2010, 9:330 7. Uganda Bureau of Statistics (UBOS) and ICF International Inc. Uganda Demographic and Health Survey 2011. Kampala, Uganda 2012 8. Renggli S, Mandike R, Kramer K et al.: Design, implementation and evaluation of a national campaign to deliver 18 million free long-lasting insecticidal nets to uncovered sleeping spaces in Tanzania. Malaria Journal 2013, 12:85 9. WHO RBM: Household survey indicators for malaria control, Measure Evaluation, June 2013 http://archiverbm.rollbackmalaria.org/toolbox/docs/rbmtoolbox/tool_HouseholdSurveyIndicatorsFor MalariaControl.pdf 10. Biedron C, Pagano M, Hedt BL, Kilian A, Ratcliffe A, Mabunda S, Valadez JJ: An assessment of lot quality assurance sampling to evaluate malaria outcome indicators: extending malaria indicator surveys. International Journal of Epidemiology 2010, 39: 72-79 11. Kakwani NC, Wagstaff A, van Doorslaer E: Socioeconomic inequalities in health: measurement, computation, and statistical inference. J Econometrics 1997, 77:87-103 12. Kilian A, Byamukama W, Pigeon O, Gimnig J, Atieli F, Koekemoer L, Protopopoff N: Evidence for a useful life of more than three years for a polyester-based long-lasting insecticidal mosquito net in Western Uganda. Malaria Journal 2011, 10:299 13. Helinski MH, Namara G, Koenker H, Kilian A, Hunter G, Acosta A, Scandurra L, Selby RA, Mulondo K, Fotheringham M, Lynch M. Impact of a behaviour change communication programme on net durability in eastern Uganda. Malaria Journal 2015;14:366. 14. Bennet A, Smith SJ, Yambasu S, Jambai A, Alemu W, Kabano A, Eisele TP: Household possession and use of insecticide-treated nets in Sierra Leone 6 months after a national mass-campaign. PLOS One 2o12, 7:e37927 15. Stevens ER, Aldridge A, Degbey Y, Pignandi A, Dorkenoo MA, Hugelen-Padin J: Evaluation of the 2011 long-lasting, insecticide-treated net distribution for universal coverage in Togo. Malaria Journal 2013, 12:162 16. Yukich J, Bennet A, Keating J, Yukich RK, Lynch M, Eisele TP, Kolaczsinki K: Planning long lasting treated net campaigns: should household’s existing nets be taken in to account? Parasites and Vectors 2013, 6:174. 52 17. WHO: Recommendations for achieving universal coverage with long-lasting insecticidal nets in malaria control. Global Malaria Program September 2013 (revised March 2014). http://www.who.int/malaria/publications/atoz/who_recommendations_universal_coverage_llins.pdf 18. Mappin B, Cameron E, Dalrymple U, Weiss DJ, Bisanzio D, Bhatt S, Gething PW: Standardizing Plasmodium falciparum infection prevalence measured via microscopy versus rapid diagnostic test. Malaria Journal 2015, 14:460. 19. Koenker H, Kilian A: Recalculating the net use gap: a multi-country comparison of ITN use versus ITN access. PLOS ONE, (2014), e97496. 20. Kilian A, Balayo C, Feldman M, Koenker H, Lokko K, Ashton RA, Bruce J, Lynch M, Boulay M: The effect of single or repeated home visits on the hanging and use of insecticide-treated mosquito nets following a mass distribution campaign- a cluster randomized, controlled trial. PLOS-ONE, (2015), DOI: 10.1371/journal.pone.0119078. 21. Koenker H, Kilian A, Zegers de Beyl C, Onyefunafoa EO, Selby RA, Abeku T, Fotheringham M, Lynch M: What happens to lost nets: a multi-country analysis of reasons for LLIN attrition using 14 household surveys in four countries. Malaria Journal (2014), 13:464 22. Kibe LW, Kamau AW, Gachigi JK, Habluetzel A, Mbogo CM: A formative study on disposal and re￾use of old mosquito nets by communities in Malindi, Kenya. Malaria World Journal 2015, 6:9. 23. Bowen HL: Impact of a mass media campaign on bed net use in Cameroon. Malaria Journal 2013, 12:36. 24. Boulay M, Lynch M, Koenker H: Comparing two approaches for estimating the causal effect of behaviour-change communication messages promoting insecticide-treated bed nets: an analysis of the 2010 Zambia malaria indicator survey. Malaria Journal 2014 13:342. 25. Yukich JO, Lengeler C, Tediosi F et al.: Costs and consequences of large-scale vector control for malaria. Malaria Journal 2008, 7:258. 26. Smith Paintain L, Awini E, Addei S et al.: Evaluation of a universal long-lasting insecticidal net (LLIN) distribution campaign in Ghana: cost effectiveness of distribution and hang-up activities. Malaria Journal 2014, 13:71. 27. Kolaczinski JH, Loaczinski K, Kyabayinze D, Strachan D, Temperley M, Wijayanandana N, Kilian A: Costs and effects of two public sector delivery channels for long-lasting insecticidal nets in Uganda. Malaria Journal 2010, 9:102. 28. Müller DH, Wiseman V, Bakusa D, Morgah K, Daré A, Tchamdja P: Cost-effectiveness analysis of the Togo integrated child health campaign. Malaria Journal 2008, 7:73. 29. WHO: World Malaria Report 2015, Global Malaria Programme, Geneva 2015 http://apps.who.int/iris/bitstream/10665/200018/1/9789241565158_eng.pdf 30. Bhatt S, Weiss DJ, Cameron E et al: The effect of malaria control on Plasmodium falciparum in Africa between 2000 and 2015.Nature 2015, 526:207-211. 31. Chizema-Kawesha E, Miller JM, Steketee RW, Mukonka VM, Mukuka C, Mohamad AD, Miti SK, Campbell CC: Scaling up malaria control in Zambia: progress and impact 2005-2008. American Journal of Tropical Medicine and Hygiene 2010, 83:480-488. 32. Lindblade KA, Mwandama D, Mzilahowa T et al.: A cohort study of the effectiveness of insecticide treated bed nets to prevent malaria in an area of moderate pyrethroid resistance, Malawi. Malaria Journal 2015,14:31. 33. Aregawi M, Lynch M, Bekele W et al.: Time series analusis of trends in malaria cases and deaths at hospitals and the effect of antimalarial interventions, 2001-2011, Ethiopia. PLOS One 2014, 9:e106359. 34. Karema C, Aregawi MW, Rukundo A et al.: Trends in malaria cases, hospital admissions and deaths followibg scale-up of anti-malarial interventions, 200-2010, Rwanda. Malaria Journal 2012, 11:236. 53 ANNEXES Annex I: Evaluation Statement of Work Statement of Work (SOW) for Evaluation of Universal Coverage Campaign for Long Lasting Insecticide Treated Nets in Uganda - Phase II 1. Introduction Background Malaria remains one of the greatest public health problems in Uganda. Uganda records one of the highest (45%) prevalence rates of malaria in children in the world; significantly higher than in neighboring countries. Uganda’s tropical climate and seven months of rainfall allows perennial malaria transmission across 95% of the country. The country records thousands of malaria related deaths per year; children account for the majority of these. According to the 2011 Uganda Demographic Health Survey (DHS), the proportion of households with at least one long lasting insecticide treated net (LLIN) increased to 59% in 2011, from 47% in 2009. Reducing high levels of morbidity and mortality associated with malaria in Uganda requires sustained delivery of proven prevention and treatment strategies to scale. The three primary interventions recommended by the World Health Organization (WHO) Global Malaria Program to tackle malaria are: a) distribution of insecticide-treated nets (ITNs), more specifically LLIN, to achieve universal coverage; b) indoor residual spraying (IRS) of houses to reduce and eliminate malaria transmission; and, c) diagnosis of malaria cases and treatment with effective medicines. Uganda’s inability in the past to take these proven anti-malarial interventions to scale is the primary reason why malaria prevalence and malaria related deaths has remained so high. LLIN lifespan is about 3 years; and Uganda last had a mass campaign to distribute LLINs to children and pregnant women in early 2010. Lot Quality Assurance Sampling surveys (LQAS) conducted in 60 key districts on uptake of nets and malaria prevalence rates over the past four years shows increasing, but still rates low rates of LLIN/net use. The National Malaria Control Program (NMCP) of the Government of Uganda (GoU) and its partners launched a major initiative to achieve universal coverage (UC) of LLINs in Uganda between October 2012 and August 2014. As per the WHO recommendations LLINs were distributed at no cost to all Ugandan households through a mass-distribution campaign. To undertake this initiative, Uganda needed at least 20 million LLINs. The Global Fund for AIDS, TB and Malaria (GFATM) provide 15 million of the LLINs required, Department for International Development (DFID), the US President’s Malaria Initiative (PMI) and World Vision supported the procurement and distribution of the remaining 5.5 million LLINs. The mass distribution campaign was government led and implemented in partnership with PMI, DFID, GF, World Vision, district and sub-county task forces, and Village Health Teams. This was the first time that Uganda attempted universal coverage of LLINs, and the first time any country has attempted to roll out LLINs at such a scale; therefore understanding the outcomes of the program and learning lessons from it for future programs is essential. If correct and consistent used LLINs protect those sleeping under them but also have a protective “herd￾effect” on others sleeping nearby when coverage rates reached 60% or higher. Higher coverage rates are achieved through universal coverage of LLINs for all people at risk of malaria to achieve community benefits. In order to achieve and maintain universal coverage, countries apply a combination of mass free 54 distributions and continuous distributions through multiple channels, in particular antenatal and immunization services. The Ministry of Health adopted use of LLINs as one of the major interventions in the National Malaria Control Policy and Strategic Plan, with the aim of having the whole population of the country sleeping under LLINs by December 2013. In the past, the focus was on protecting the most vulnerable people (pregnant women and children under five), and distributions of LLINs was targeted at these groups. The revised WHO and National Malaria Control Program policies now aim at achieving “Universal Coverage” for all interventions (prevention and treatment), including LLINs. Universal coverage involves reaching 100% of the population at risk of malaria with prevention and treatment. For Uganda, universal coverage for LLINs is defined as one LLIN for every two people. The UC was implemented by the NMCP and the District Health Teams as well as PMI Uganda’s implementing partners (Stop Malaria Project and the Malaria Consortium). This activity was coordinated by the National Coordination Committee, which is chaired by the Ministry of Health (MOH). Significant financial support and technical assistance was realized from several campaign donors (The GF, UKAid through the DFID, CDC and USAID through PMI). The UC involved a number of activities that were implemented The UC involved a number of activities that were implemented 1. Procurement of LLINs 2. Storage, and positioning of LLINs at distribution-points in sub-counties 3. National behavior change communication 4. Registration of households and allocation of bed nets 5. Distribution of bed-nets to registered households 6. Data center for distribution data capture The UC was implemented alongside other components of malaria prevention which include indoor residual spraying (IRS), Intermittent Preventative Treatment of Malaria in pregnant women (IPTp), and routine antenatal care (ANC) bed net distribution implemented by the government and its development partners. While evaluating the effectiveness of anti-malaria programs has become of considerable interest to host country governments, donors and implementing partners globally; the UC in particular because of its scale and scope provides unique opportunities for demonstrating effectiveness, outcomes and learning that will inform future policy interventions. The UC evaluation was designed as a two pronged with two phases. Phase I, as a process evaluation that focused on the implementation processes of the campaign in 2014. This part of the evaluation was completed in December 2014. Phase II of the evaluation to focus on the outcomes of the campaign, cost efficiency and other issues noted as requiring attention in phase I. 2. Purpose of the Evaluation Phase II is a build on of the phase I process evaluation that is already completed. The purpose of phase II is to articulate the outcomes of UC on malaria morbidity, burden of malaria in Uganda. It will also assess the effectiveness of different approaches as used under the campaign. 3. Audience The primary audience of the evaluation findings includes government of Uganda, development partners and civil society, to inform planning and programming of subsequent large scale universal net coverage campaigns. • Development partners i.e. DFID, PMI, Global Fund for Global Fund for AIDS, TB and Malaria (GFTAM) and World Vision 55 • Government of Uganda including the Ministry of Health as the technical lead agency and other arms concerned with monitoring government programs and developing policies and programs10. 4. Evaluation Questions 1. Was the UC campaign implemented in the most cost effective manner? If not, what alternative means could have been pursued by the program to accomplish the same outcomes at a lower cost in comparison with phase 1 and phase II of the campaign? What is the comparative cost of phase 1 and phase 2 LLIN delivery models? 2. How have the different behavior change communication channels affected ownership and use of nets? How many channels of BCC are there? Were different BCC communication channels applied in the different regions of Uganda? 3. What measureable health outcomes in terms of morbidity and overall malaria disease burden have been contributed to the universal coverage LLIN campaign? What is the outcome of the nationwide LLIN distribution on malaria burden (prevalence, incidence, and anemia)? 4. What is the quality of household registration data that was collected during the UC campaign and used to inform procurement and/or distribution of the LLINs? The evaluation team is encouraged to make suggestions/amendments to the evaluation questions that would yield a better study, in such a case the suggested questions will have to be approved by the multi￾stake holder group composed of PMI, DFID, and NMCP. The evaluation team will also be required to make another table showing the data source, the year, the variables of interest and a brief description of how the data set will be used in the analysis. In addition, a detailed methodology of the primary data collection for the BCC question will also be included. 5. Evaluation Design and Methodology The evaluation will use a combination of before and after designs and time series analyses11 to establish the effectiveness and outcomes of the universal coverage campaign. In order to respond to the evaluation questions, the evaluators will deploy a mixed methods approach, i.e., using both quantitative and qualitative data collection methods. They will conduct both a systematic review of different data sets, evaluations and studies on the one hand, as well as primary data collection to understand behavior change communication channels. The evaluation team is required to disaggregate data by age and specific interest groups which include for children under five (5) years and pregnant women. The use of GIS, data visualization techniques and maps is also envisaged. Suggested data sources/collection methods include: • Review of Program Documents and Related Literature: The evaluation team will review documentation of universal campaign process, budgets and results including the Phase 1 evaluation report and draft Malaria Indicator Survey Report (2014)12. The team will also look at similar assessments of effects and outcome of net utilization from other surrounding countries within East Africa and Asia. 10 Including the Ministry of Finance, Planning and Economic Development, Office of the Prime Minister, Parliament and civil society. 11 The evaluation team will review and determine based on the HMIS data available whether they can conduct an interrupted time series. We recognize potential limitations in using an experimental design through the construction of a true counter-factual group because the UC was implemented country wide. 12 http://dhsprogram.com/what-we-do/survey/survey-display-484.cfm 56 • Secondary data: Some of the data that should be reviewed include: o UC budget and expenditures, o Household registration data, o HMIS and DHS2, on malaria incidence, prevalence, morbidity and mortality o Sentinel data from the Uganda Malaria Surveillance project o Preliminary Uganda National Population and Housing Census (2014) o Uganda MIS (2014) data set; and o Data from other PMI countries that have done UCC • Key Informant Interviews (KII): Interviews with UC stakeholders as well as researchers and analysts involved in the MIS to clarify data and findings. • Focus Group Discussion (FGDs): FGD with different sections of stakeholders especially beneficiaries and lower local leaders to triangulate both distribution/coverage and ownership information. The evaluation team may also suggest other rigorous methods for achieving results for the evaluation. The evaluation team will propose a sound sampling strategy and data analysis strategies. The team is expected to study existing datasets such as the Malaria Indicator Survey (MIS), DHIS2 conduct trend analysis over the period 2010–present; A comparison of changes in health outcomes between Uganda and other countries that have implemented UC could highlight presence or lack of cause effect between program interventions and health outcomes, where possible. Data disaggregation and analysis by region, gender, and age (especially the youth) to establish the differential effects of the intervention on men, women and different age groups will also be expected. The team will propose other analysis approaches. In addition, the evaluation team shall describe the type of software for data analysis they propose to use. All proposed data collection instruments (methods), sample selection, data analysis and guides will be approved by the government and PMI/DFID team prior to the beginning of fieldwork. Both the evaluation process and products should meet the Uganda Evaluation Standards and evaluation requirements of both USAID and DFID (See annex 4). Prospective evaluators are encouraged to propose additional, alternative methodologies that they believe can yield stimulating, robust evidence in answering the evaluation's questions. Data Limitations: Availability and quality of expenditure data, as well as HMIS data will pose limitations during this evaluation. Though capacity of local governments and their partners to compile and make available routine health services and use data has improved, completeness and accuracy of initial data is not guaranteed. The evaluation team will provide recommendations on how they will address these limitations. 6. Deliverables The evaluation team is expected to deliver on the following: 1) Inception report should include: a. the evaluators’ interpretations of the different evaluation questions, b. the evaluation design and methodologies to address the evaluation questions including the cost efficiency assessment component, c. sampling strategy, data collection tools, instruments and detailed survey protocol where appropriate, detailed data analysis strategy and plan, d. outline and structure for the final evaluation report, 57 e. proposals for other communication products and ways for presenting and sharing findings with different stakeholder groups, f. detailed evaluation work plan, plan for the management of the evaluation, as well as identify risks and challenges the evaluation may potentially face and how they will be addressed, g. The evaluation team will present a completed “getting to the answers matrix” (Annex I). 2) Oral presentation of findings (debrief): the team will make Power Point presentation (including hand-outs) for: i) Debriefing: The team will present major findings of the evaluation to the joint evaluation management team from USAID/PMI, DFID and NMCP/MOH before embarking on writing the draft report. The debriefing will include a discussion of findings, emerging conclusion, lessons learnt as well as any recommendations the team has for possible future designs and implementation. The presentation will normally be for 60 minutes covering the major findings, conclusions, and lessons learned, and allowing for about 30 minutes of discussion and feedback. i) Debriefing with other Stakeholders (Optional): The team may have to present major finding of the evaluation to wider partners (as appropriate and as may be defined by The GF, PMI and DFID) through a PowerPoint presentation. This debriefing will include a discussion of findings only, with no recommendations. The team will consider other Stakeholders’ comments to write the draft report accordingly. 3) First Draft Evaluation Report: The team will submit a rough draft of the report to the USAID/Learning Contract, who will provide preliminary comments prior to submission the multi-stakeholder group including The GF, PMI, DFID, World Vision, NMCP and other development partners. The content should cover all the main elements of the report including major findings, conclusions, lessons learned, and relevant annexes. The input from the oral presentation sessions should also be incorporated in this report. USAID/PMI, DFID, The GF, World Vision and the MoH will provide comments on the final draft report within two weeks of submission. The first draft report should be 15-25 pages excluding the executive summary and the annexes. The report should meet both USAID and DFID standards. 4) Final Draft Evaluation Report: A complete report presented in the agreed-upon format and incorporating comments from NMCP/MOH, USAID/PMI, DFID and other development partners. The written report should clearly describe findings, conclusions, and recommendations. USAID/PMI, DFID, The GF, World Vision and the MoH will provide comments on the final report within two weeks of submission. 5) Policy Brief – It is a summary and analysis of current policy issues and developments to help policymakers and development practitioners improve development outcomes through better evidence. It should be written in non-technical language and be delivered in a short and engaging format relevant to the target audience, most preferable in visual format. 6) Final Report: The team leader will submit a final report to QED within one week of receiving final comments from USAID/PMI, DFID, The GF, World Vision, MoH including those from other stakeholders. The final report should be 15-25 pages excluding the executive summary and the annexes. The format will include an executive summary, table of contents, methodology, findings, and recommendations. The report will be submitted electronically in both word and pdf versions. 58 7) Cleaned labeled and ready to use electronic copies of datasets collected through evaluation must be separately delivered to the Learning Contract 7. Anticipated Level of Effort (LOE) Required The task is estimated to begin on or about the beginning of July 2015 and be completed no later than September 2015. The anticipated major tasks below are not necessarily presented in the order of execution. Contract may provide for protracted timeline outside the performance time to provide for billing and closure. Anticipated Major Tasks Expected Level of Effort (Days) 1. Review of program documents (out of country) 3 2. In briefing and team building 1 3. Inception Report (including presentation) 3 4. Field work (data collection) 14 5. Draft Report writing 7 6. Debriefing 1 7. Final Draft Report 4 Total Days 33 8. Team Composition The evaluation will be conducted by an external evaluation team, consisting of international and Ugandan experts. The evaluation team must include a team leader and three senior experts and may include research assistants/middle level evaluators as need be. The following are the key evaluation team members: 1) Team Leader – (international) a senior evaluator 2) Evaluator – (National) 3) Malaria Technical Expert 4) Health Economist 5) Statistician 1. Team leader shall have: • A minimum of a Master’s degree in development studies, health management, public health, applied research or related fields. PhD is preferred • Significant expertise in conducting evaluations with a minimum of seven years’ experience in health evaluation/ management • Must have played significant role designing, managing or executing a minimum of seven evaluations, at least five for public sector health activities and at least two in Africa or similar contexts. • Should have been team leader in a minimum of five evaluations, with at least one evaluating activities of similar scale • Skills in quantitative and qualitative evaluation design and methods. 2. Evaluator shall have: • A minimum of a Master’s degree in development studies, health management, public health, applied research or related fields. PhD is preferred • Significant expertise in conducting evaluations with a minimum of five years’ experience in health evaluation 59 • Must have played significant role managing or executing a minimum of five evaluations, at least five for a public sector health activities. • Skills in quantitative and qualitative evaluation design and methods. 3. Malaria Technical Expert (National) shall have: • A minimum of a Master’s degree in, health management, public health, applied research or related fields. • Significant expertise in conducting evaluations with a minimum of five years’ experience in evaluating/implementing malaria control and management program • Must have played significant role in evaluating at least five for a public sector health activities and at least two in Africa or similar region. • Skills in quantitative and qualitative evaluation and methods. 4. Health Economist (National) shall have: • A minimum of a Master’s degree in health economics. • Significant expertise in efficiency and cost benefit analysis with a minimum of five years’ experience in health evaluation management • Should have been a team in a minimum of five evaluations, with at least one evaluating activities of similar scale • Skills in efficiency analysis and report writing 5. Statistician (International / national) shall have: • A minimum of a Master’s degree in epidemiology, medical statistics, biostatistics or applied statistics. • Significant expertise in analyzing large datasets health datasets in developing countries • Should have been a statistician / principal investigator on a minimum of five research studies, with at least one relating to malaria intervention of a similar scale • Skills in quantitative data analysis and report writing. 9. Evaluation Management The USAID Program Office will have primary administrative and technical responsibility for the evaluation process. This also includes making the necessary arrangements for PMI/DFID inputs and briefings. The Program Office M&E Specialist will provide coordination and clarification of USAID requirements and standards for maintaining effective communication on what is to be learned. The USAID Monitoring, Evaluation and Learning Program i.e. the Learning Contract has overall responsibility for the coordination and management of this evaluation. This will include the day-to-day management of the evaluation team, logistical support, quality assurance and submission of relevant deliverables to PMI/DFID. The Learning Contract will also organize relevant stakeholder in briefs and de-briefs and overall ensuring that the key requirements of the SOW are being met. PMI/DFID will work with the different partners to set up a multi-stakeholder core evaluation management committee composed of PMI, DFID, The GF, World Vision, MoH/NMCP, and the Office of the Prime Minister. This committee will have a primary responsibility in coordinating necessary inputs, review and clearance of evaluation products in line with the requirements and standards established in the SOW. The Ministry of Health and Office of the Prime Minister may each nominate a senior staff member to join the evaluation team and participate in critical field data collection activities, analysis and reporting. The evaluation contractor/s will be responsible for making transport and other logistical / management 60 arrangements and will also cover all relevant facilitation for evaluation team members and government staff members, if and when necessary. 10. Logistical Support The Learning Contract (TLC) will provide equipment necessary for performance of this activity including transport hire (within Kampala and upcountry), office supplies, communication costs, Performance Period The evaluation is scheduled to begin in July 2015. The end of the performance period will be determined by PMI/DFID. 61 Annex II: Getting to Answers matrix Final “getting to answers” matrix Evaluation Questions Sub questions to help answer the main question Data Collection Method(s) Data Source(s) Data Analysis Method(s) 1. Was the UCC campaign implemented in the most cost effective manner? And if not, what alternative means could have been pursued by the program to accomplish the same outcomes at a lower cost? • What was the economic cost for the UC? Total and per LLIN delivered Compilation and aggregation of secondary data , KIIs Budgets and expenditure data from UCC procurement and implementation Cost effectiveness analysis • What were the cost drivers and how did cost/LLIN delivered differ between waves KIIs Cost-effectiveness data, KII results Comparative and sensitivity analysis • How does the cost of implementing UC in Uganda compare with UC costs in other countries? Literature search Cost-effectiveness data, literature review Comparative analysis • What were the outcomes of the UCC regarding ITN ownership? Preparation of secondary data MIS data, UCC distribution database Univariable and multivariable statistical analysis, mapping • Could there have been significant savings without loss of outcome? KIIs,, FGDs, literature search Cost-effectiveness data, registration quality data, MIS data, UCC distribution data, literature review, KII and FGD results Comparative analysis, sensitivity analysis, data triangulation, modeling, mapping • What is the ratio of benefits to costs? None Cost-effectiveness data, malaria burden analysis data Cost-benefit analysis 2. How have the different behavior change communication channels affected use of nets? • Which communication channels were used where in which intensity used during the UCC? KIIs, Compilation and aggregation of secondary data KII results, MoH, UC implementing agencies), literature review Descriptive analysis, BCC mapping • What was the quality of messages delivered or materials used? None Samples of BCC materials from BCC mapping Qualitative content analysis • What was the use pattern of nets by age, gender and other factors? Who used the nets from the campaign? Preparation of secondary data MIS data Univariable and multivariable statistical analysis, mapping • Which BCC channels reach which type of households? Preparation of secondary data MIS data Univariable and multivariable statistical analysis • Is there a link between exposure to BCC and net use after the UCC? Preparation of secondary data, FGDs MIS data, FGD results Statistical modelling (treatment effects), Qualitative analysis, triangulation, mapping 62 Evaluation Questions Sub questions to help answer the main question Data Collection Method(s) Data Source(s) Data Analysis Method(s) 3. What measureable health outcomes in terms of morbidity, parasite prevalence and anemia can be found and can they be attributed to the UCC? • What have been the trends in malaria morbidity before and after the UCC by region, age, gender? Compilation and aggregation of secondary data HMIS data, Stratified Trend analysis with adjusted morbidity data • What have been trends in parasite prevalence and anemia in children under five before and after UCC Compilation and aggregation of secondary data MIS data, previous DHS and MIS data Comparative analysis, triangulation, mapping • To which extent can changes in incidence, prevalence and anemia be attributed to UCC? None Morbidity analysis data, MIS analysis data Trend analysis, statistical modeling • How do trends in malaria morbidity, parasite prevalence and anemia in Uganda after UCC compare to other countries in Sub Saharan Africa? Literature search Morbidity analysis data, MIS analysis data, Literature review Qualitative comparison 4. What is the quality of household registration data that was collected during the UC campaign and used to inform procurement and/or distribution of the LLINs? • What is the validity and completeness of the registration data? Compilation of secondary data UCC registration database Consistency analysis against demographic standards, search for outliers • How do the registration data compare against previous and recent census data? Compilation of secondary data UCC registration database, census data Comparison, mapping • What is the overall quality of the UCC registration data? KIIs, literature search UCC registration database, census data, literature review, KII results Descriptive comparison 63 Annex III: Evaluation Methods and Limitations Additional Information on Quantitative Data and Analysis The phase II of the Uganda UCC evaluation provided a unique opportunity to address the key evaluation questions due to the occurrence of two independent data collections that directly relate to the campaign and hence can be used to assess the effects and data quality of the UCC. These are the National Census, which took place in August 2014 at the end of the UCC implementation, and the Malaria Indicators Survey (MIS) undertaken between December 2014 and January 2015, 5-15 months after the core of the UCC waves (Figure 13). FIGURE 13: TIMING OF THE UCC DISTRIBUTION WAVES, NATIONAL CENSUS AND MALARIA INDICATOR SURVEY In addition to the data sources mentioned in the main report, the following secondary or contextual data were used: • Historical root of current 112 districts and year of separation since 2001 (56 districts) to allow comparison of current with historical data • List of LLIN brands and specifications (material, yarn strength [denier], mesh size, shape, color and size) and which brand was distributed where (wave or sub-region) • Dates of actual distribution by district to allow a more precise estimation of time since distribution in the MIS 2014/15 data set • List of IRS activities by district and month of spraying since 2009 64 Methods UCC Registration Data Definition of the data quality criteria evaluated was as follows: Criterion Definition Timeliness Data should be available sufficiently timely to influence management decision making Completeness Results are appropriately inclusive: it represents the complete list of eligible units and not just a fraction and all data fields have valid entries Validity Data should clearly and adequately represent the intended result Reliability Data should reflect stable and consistent data collection processes and analysis methods over time Integrity Data collected should have safeguards to minimize the risk of transcription error or data manipulation Accuracy13 Accurate data are considered correct: the data measure what they are intended to measure. Accurate data minimize error (e.g., recording or respondent bias, transcription error) to a point of being negligible 13 In general, accuracy is often also described as validity in the statistical and evaluation literature, but because in the USAID data quality guidance validity is defined otherwise, we avoid this use here. 65 FIGURE 14: REGISTRATION AND DISTRIBUTION DATA FLOW AND POTENTIAL SOURCES OF ERRORS Data entry Process, Checks and Integrity for UCC Database • SOP’s for UCC data collection processes were not available in written format for review although procedures were well known and articulated by the Director of the MOH Data Resource Center and Data Manager. • In Feb 2013, registration list for Soroti was completed. The database was created at MOH in March and launch of UCC began with Soroti on April 10. Wave 1 districts (Soroti and Busia) were the only districts where collected registration information was entered into the database with enough time to generate allocation lists for distribution. Following wave 1 districts, system changed and distribution immediately followed registration. Completed forms including both registration and distribution data were sent to MOH for data entry. • Database was in a dashboard interface “LEDMSS LLIN electronic management software system” and the database server was a PC laptop computer. Dropdown menus captured data by sub￾county, parish and district. • Data entrants were recruited interns that were primarily university students who used personal laptop computers for data entry. These were solicited by newspaper advertisements and interviews until SMP was able to manage the recruitment process several months after data entry had already begun. A data entry supervisor managed access to the server. • At Data Resource Center there was a system administrator, verification officer, supervisor and data entrant. As forms came in per district (or in bulk for numerous districts, e.g. information from SMP) the production officer (in charge of verification officer) verified actual forms that came in (in the presence of the person who brought forms). Production officer would capture 66 parish, name of sub-county, and number of villages. • One data supervisor managed ten data entrants. System recorded data entrant log in and log off and quantity of records entered. Data records captured VHT and LC1 chair for follow-up of potential errors, which would be carried out by NMCP staff. • There was no clear mechanism to capture transcription errors from the field, e.g. when ample forms for registration were not available, photocopies were made which resulted in duplicate serial form numbers that were later discovered at the central level during data entry. • During data entry process, if forms were not complete clarification was sought from NMCP for resolution with calls or visits to the relevant district. Data was aggregated at village, parish, sub￾county and district level. • Funds for data management were limited and became available months into the UCC. • All source documents are available in hard copy at the MOH Data Resource Center and filed by district. • No clear review for distribution data for completeness and accuracy. Calculation of Variances and Comparisons Population variance was calculated as: (Population UCC database - population census) / population UCC database *100 For comparison between UCC database and UCC records the census was replaced by the UCC records. For comparisons between UCC records and census the UCC database was replaced by the UCC records. Population per net distributed (or allocated) was calculated as: Population UCC records/nets distributed UCC records Methods ITN Ownership The core net and ITN ownership indicators were: • Proportion of households with any net/ITN • Proportion of households with at least one ITN for every two people (also referred to as “enough ITN” • Proportion of the de-facto14 population with access to an ITN within the household (also referred to as “access”. Two regional categorizations were used to geographically stratify data: the four major census regions (Northern, Eastern, Central, Western) and the 10 sub-regions as defined for the MIS 2014/15. As these differed slightly from the sub-regions previously used in the DHS and MIS surveys, the sub-regions in the older survey data sets were recalculated to match the 2014/15 categories. A comparison of these regional definitions is shown in Figure 15. 14 The de-facto population includes all people that were present in the household the night before the survey, including visitors but excluding usual household members who slept elsewhere that night. 67 FIGURE 15: DIFFERENCES IN DISTRICT ALLOCATION TO SUB-REGIONS IN DHS 2011 AND MIS 2014 THE COLORED ZONES REPRESENT THE FOUR CENSUS REGIONS Additional indicators were calculated which are not commonly used in DHS/MIS surveys: • ITN ownership coverage with and without the UCC nets and only for specific sub-types of nets. This was done by defining net types by the source (e.g. UCC, health facility or private sector) and type of net (LLIN, conventionally dipped, untreated), collapsing the net data to households level (number of nets of each type in household) and then calculating the ownership indicators based on only the nets of the category of interest. • Under- and oversupply with nets was defined similar to the indicator of at least 1 ITN per two people but instead of a dichotomous outcome (yes/no) five categories were used: i) no ITN, ii) less than 1 ITN for every 3 people, iii) 1 ITN for every 3 people; iv) 1 ITN for every 2 people; v) 1 ITN per person or more. For single-person households ownership of one net was recoded as “1 ITN for every two people” as one net is not an oversupply. • In order to provide estimates of the intra community coverage with ITN an LQAS-based approach was used applied to the survey clusters as previously described for the 2007 Mozambique MIS [10]. In this case a variety of possible community level coverages were explored for “any ITN” and “enough ITN” and details are given in Annex E. • Equity assessment used wealth quintiles derived from the assets-based wealth index calculated in the original DHS/MIS data sets using principle component analysis (PCA). However, in order to exclude regional differences in ITN distribution, quintiles were first calculated for each of the 10 68 MIS regions before being aggregated at national level15. Three outcome majors were used: i) equity ratio equity ratio, i.e. the ratio (range zero to infinity) between proportion of households owning a net/ITN in the poorest quintile compared to that in the least poor quintile, where a ratio of 1.0 describes perfect equity, values below 1.0 inequity towards the least poor and values above 1.0 in favour of the poor; ii) the concentration index [11] which includes not only the highest and lowest wealth quintiles but takes into account the full data set. A value of zero represents perfect equity, while a negative value indicates disproportionate concentration of net ownership among the poor (pro-poor), and a positive value disproportionate concentration among the wealthier (pro-rich); iii) the concentration curves (or Lorenz curves) which plot the cumulative distribution of wealth quintiles in the overall sample against the cumulative distribution of wealth quintiles in the sub-group of interest (e.g. households owning any ITN). Calculation of indicator “1 ITN/2 people or better” Two variables were used to construct this indicator, i) the number of nets/ITN etc. owned by the household and ii) the number of de-jure household members, i.e. those usually living there without visitors. The number of nets of a specific definition (e.g. UCC nets) was obtained from the net file by calculation the number of nets within each household and then collapsing the net data set to household level and importing it into the household file. In a first step an interim variable was created that expresses the “number of nets per person” in the household by dividing the number of nets in the household by the number of people. This approach is preferable over the calculation of “persons per net” because the number of people in a household is always greater than zero and hence gives a valid ratio even if the number of nets is zero. In the case of “persons per net” any household with no nets would result in a missing value. The second step then is to create a dichotomous variable (yes/no) that is determined as positive if the “nets per person” ratio if equal to or larger than 0.5 (corresponding to 1 net per 2 people). For the supply indicator with five categories the respective cut-offs were: Less than 1 net/3 people: ratio nets/person > 0 and < 0.30303 1 net /3 people: ratio ≥ 0.303 and <0.5 1 net/2 people: ratio ≥ 0.5 and <1.0 1 net/person or more: ratio ≥ 1.0 For households with just one person and one net the variable was recoded as “1 net/2 people”. Calculation of “population access to ITN within household” with adjustments The calculation of this indicator is done in the household member file and uses as input the number of nets/ITN owned by the household imported from the household file, the variable “person used ITN last night” and the information whether the person spent the night in the house (de-facto population). The first step is to create a variable of the “potential net users” in the household assuming two people share a net by multiplying the number of nets by 2.0. In the second step household members are allocated the “has access status” based on the number of de-facto members and potential users ensuring that the people with access do not exceed the number of people in the household. The adjustment for actual net use is made by allocation the access first to all actual users and then giving access to non-users based on remaining 15 Since the distribution differed in Kamapala/Wakiso by design and the highest proportion of “wealthy” people live here, the equity outcome would not be biased: wealthy people did not get nets not because they are wealthy, but because they lived in Kampala and were not targeted. The same applies to regional differences in quality of distribution during UCC or previously. 69 potential users in that household. LQAS-based approach to calculate intra-community coverage The community level coverage is calculated by applying the decision rules of LQAS to each survey cluster based on the number of actual interviewed households in that cluster which for the MIS 2014/15 was between 12 and 28 with median of 26. The decision rules were obtained for each potential cut-off level of community coverage with a 20% margin target to minimal coverage and each potential cluster size from a USAID sponsored web site16. The procedures were implemented as macros (do-files) in Stata. Methods Malaria Morbidity Preparation of HMIS/DHIS2 data on malaria morbidity The first step in data preparation was the merging of the HMIS data 2005 to 2011 with the DHIS2 data set 2011 to June 2015. For the transition phase between the two systems in 2011 care was taken that no duplicate records by month and district were present. Next, a history of districts was established for the period 2005 to 2010 during which the number of district increased from 70 to 112. Based on government records of the establishment of new districts17 each of the current districts was linked to its “mother” district of 2005 to allow proper grouping by census and MIS 2014 regions. Because data for 2005 was very limited, this year was omitted from the analysis. Cleaning of the data started with screening of all variables for outliers and correction of obvious errors. For missing monthly reports of a district an empty rows was added if the district already existed after cross-check with history of districts. There were two levels of adjustments done. The first was for missing facility reports within a month for which data was reported. Here the proportion of actual out of expected reports was calculated and all variables were inflated to 100% using this percentage. In cases where no expected and actual facility reports were available as well as for missing months, the results were estimated from preceding and following months by interpolation. Next, the population for each district was imported from the 2014 census data and projected backwards using the district specific growth rates 2002-2014. To allow the calculation of monthly and quarterly incidence rates population estimates for each month or quarter were calculated as linear growth between the annual estimates. Population sub-groups by age (under five and five or more years of age) were calculated using the proportion of under-fives from the 2011 DHS of 18.9% applied to the annual population. The gender allocation was based on the 2014 census reporting 48.6% males. Based on these completed and reporting-adjusted data, OPD attendance (visits per person per year), proportionate malaria rate (reported malaria cases per 100 new OPD attendances), malaria incidence (reported malaria cases per 1,000 population), test (microscopy and RDT) positivity rate and test to diagnosis ratios were calculated. To adjust for changes in OPD attendance over time an adjusted malaria incidence was calculated by applying a standardized OPD utilization rate of 1.50 for children under five and 1.00 for over five year olds. In a final step the OPD adjusted malaria incidence rate was multiplied with the slide positivity rate to get an estimate of a “confirmed” malaria incidence. To explore trends over time in the outcome variables monthly and quarterly data was aggregated by regions, UCC waves, IRS and Non-IRS districts (Mid-North region only) and plotted using moving 16 Food and Nutrition Technical Assistance II Project (FANTA-2) Bridge LQAS sampling calculator at: http://www.brixtonhealth.com/hyperLQAS.html 17 Obtained from the following website: http://www.statoids.com/uug.html 70 averages to stabilize monthly fluctuations of data. Methods Cost The provider perspective was used for the cost analysis. Travel or time costs to users, or other household￾level costs or cost savings have not been measured or included. All direct and shared costs of the LLIN program to the provider were included, including those of the commodities and their delivery, health promotion, as well as direct costs for commodities and transport payment of salaries and overheads. Costs represent the marginal cost of adding the Uganda UCC intervention onto an existing malaria control program and health system. These expenses include the cost of training health workers in LLIN distribution and use and the use of facilities, transportation, payment of personnel, and the commodities themselves. Costs required to build and train the network of health workers and teachers necessary or to develop a network of health facilities and schools have not been included. Cost Data Collection Costs were collected retrospectively from financial and operational records kept by Malaria Consortium, Population Services International, John Snow/DELIVER and WorldVision, and through interviews with program managers and implementers involved in the deployment of the UCC campaign in Uganda. Costs were collected by examining agencies' financial records, including budgets, expenditure records, reports, receipts and invoices. In addition, cost and activity information were collected through stakeholder interviews where costs were not reflected in the financial records. Where possible, the ingredients approach was used, meaning that unit activities/inputs were first determined and then a unit cost was determined for these activities/inputs, allowing for the establishment of a total cost. Where this approach was not possible, either because the information was deemed too sensitive or was not available in adequate detail, aggregated expenditure was generally used. Resource use was valued based on national and international aggregate information. Resources were valued based on the reported expenditures or budgets, and in the case of shared personnel, on salary plus any fringe benefits. Capital goods were valued based on their procurement costs. Cost Classification and Adjustments Costs have been divided into capital and recurrent costs18 based mainly on the lifetime of the goods being purchased (in this case capital items were limited to LLINs as most other costs (buildings and vehicles were treated as rental costs and treated as recurrent). Capital costs have been discounted in the economic analysis using lifetimes and discount rates determined through stakeholder interviews, expert information, and past literature. Varying discount rates and lifetimes were examined in the sensitivity analysis. Both financial and economic analyses were conducted, in order to show i) what the actual expenses of running a program would be (financial costs19), as well as ii) the level of all resources used over the project period (economic costs20). In the financial analysis, capital costs have not been annualized and are instead incurred in full at the time of the purchase. Costs were collected in USD or in Ugandan Shillings. Costs collected in Shillings were converted to USD based on yearly average interbank exchange rates for the period during which the costs were incurred, and all costs were incurred during 2014 thus 2014 USD are 18 Capital costs are costs incurred to purchase items with a lifespan greater than one year. Recurrent costs are costs incurred for goods or services lasting less than one year. 19 Financial costs are a measure of expenditures as they were incurred by the provider of the intervention. 20 Economic Costs are a measure of resource used by the provider of the intervention rather than financial expenditure. As such they include a valuation of donated goods as well as annuitization of capital goods. 71 the final currency used for valuation. In the economic analysis, economic cost occurring in future years (such as those due to future net use were discounted to 2014 USD). Outputs and Outcomes Generally, one main output measure was used: number of nets delivered. This figure was reported by the program as an output of the UCC. The main output indicator was used to calculate a several combined output measures, including: Person Years Protected (PYP), which assumes that one net can cover two persons for a period of three years in baseline calculations, Treated Net Years of protection assuming that each net provides three years of protection in baseline calculations, this was equivalent to Child Years of Protection which assumes that each net protects one child for three years in baseline calculations, and Deaths and DALYs averted which assume that each child year of protection prevents 5.5 / 1000 all cause child deaths (Cochrane review). These assumptions were examined in sensitivity analysis. Base scenario In this analysis, the base case scenario relies on the following set of assumptions. A discount rate of 3% has been applied. LLINs have been assumed to last for three years and, as noted above, to provide three years of protection to two persons each. The cost of nets was based on the commodity, insurance, and freight (c.i.f.) price of the nets. Net Coverage Simulation Simulation of net coverage results using a simulation package for net coverage developed in [R] was conducted to estimate the additional person years of protection and coverage provided by LLINs distributed through the UCC above those delivered through the previous distributions. Simulation was conducted using the [R] package for the stochastic simulation of Net Coverage based on survey data [16] and the NetCALC. The base scenario assumed that no UCC distribution activities were conducted that the only nets in Uganda were those from previous distributions. Two alternative simulations were conducted, one including only the nets distributed during the study period in addition to those in the base line scenario and one alternative scenario assuming the UCC campaign had distributed a smaller volume of nets more closely aligned with 2014 census data as opposed to the 2014 registration data. Limitations Survey Data All survey data used for this evaluation was from nationally representative household surveys undertaken by the Uganda Bureau of Statistics with technical support from international partners such as the USAID funded DHS Measure Project. Sampling procedures followed state of the art two stage cluster sample with probability proportionate to size for the clusters and equal probability at household level. For the analysis sampling weights according to the survey design were applied ensuring that results remained representative at national level as well as by sampling domain. Sample sizes of these surveys were sufficient to provide enough precision even at domain level for the purpose of this evaluation. As for all respondent-dependent household surveys potential biases are to be taken into account, especially recall bias for critical responses such as net ownership and use. However, the questionnaires used have been tested over many years to provide the most reliable methodology (e.g. for net use) and the constancy of results from multiple, independent surveys suggests that even though some biases cannot be excluded, it is very unlikely to have distorted the results of this evaluation. NetCALC NetCALC is a compartmental model and as such has its limitations. One of the most important is that nets within the model are always optimally allocated to households, i.e. irrespective of the physical location of the household. This is unrealistic in reality and hence the model leads to overestimation of coverage in 72 situations of “excess” nets” and underestimation if targeted distributions with less than 1 ITN per 2 people are implemented. However, to the extent possible these factors have been adjusted for in the input into the model and by undertaking sensitivity analysis. Also, the limitation affects all scenarios equally so that the relationship between them (e.g. UCC versus continuous distribution) remains valid. HMIS and DHIS2 The major limitation of the HMIS/DHIS2 data is that it only covers data from the public sector and not the for-profit hospitals and clinics which provide a significant proportion of fever of malaria treatments in Uganda. However, from all available contextual information it is highly unlikely that trends in these data – if available – would be essentially different from those of the public sector. Therefore, the absolute level of malaria incidence from the public sector alone is lower than the “true” incidence, but the trends over time are valid. 73 Annex IV: Data Collection Instruments and Summary of Field Data Guide for Key Informant Interviews and Focus Group Discussions Focus Group Discussions: Community Beneficiaries Opening Statement: Good morning and welcome. Thanks for taking the time to join our discussion about the mosquito nets recently distributed. My name is …….., and I will serve as the guide for today’s discussion. Assisting me is …….. The purpose of today’s discussion is to get information from you about what you feel about the mosquito nets recently distributed. You were invited because the mosquito nets are meant to reduce malaria in your individual households. There are no right or wrong answers to the questions I am about to ask. We expect that you will have differing points of view. Please feel free to share your point of view even if it differs from what others have said. If you want to follow up on something that someone has said, you want to agree, disagree, or give an example, feel free to do that. Don’t feel like you have to respond to me all the time. Feel free to have a conversation with one another about these questions. I am here to ask questions, listen, and make sure everyone has a chance to share her views. We’re interested in hearing from each of you. So if you’re talking a lot, I may ask you to give others a chance. And if you aren’t saying much, I may call on you. We just want to make sure we hear from all of you. Feel free to get up if you would like. We will both be taking notes to help us remember what is said. No names will be included in our reports. First we would like you to introduce yourselves. Would you tell us: 1) your first name, 2) how many young children you have, 3) what you do during the day, 4) why you decided to participate in this focus group 5) how many people reside in your household_______ ? ***Please ensure that you have selected at least 2 groups with large households >7 persons and at least 2 groups with small households ~3-4 members*** 1. Now we would like you to think back on your experiences regarding the recent mosquito nets distributed by government. Can you describe that experience for us? Probes a. How did you decide to get the mosquito nets? b. What were your expectations and reactions to getting the mosquito nets? c. Who talked to you? d. What questions did you have? 2. Now, we want to ask you about the campaign in your community. What, if anything, do you know about this campaign? Probes a. How was it that you first learned about the mosquito net campaign? b. What did you think about the registration process before the campaign? c.Were there any problems with the process of registration? d. Were people that you know missed during registration? What did they do? e.What were your initial thoughts about the nets? f. When you received new nets from the campaign, did you get enough nets to cover all the sleeping places in your house? g. Can all family members use a net if they wanted to? h. Which useful information was left with you during or after the campaign? 3. Are there difficulties you find in using the mosquito nets? 74 Probes a. What aspects of using the mosquito net are you most comfortable with? b. What are disappointments you've had with mosquito nets? c. Do you recall a specific event or message that convinced you to use a net? If yes, what was it? d. Are nets that you have being used for alternative activities and if so, which nets, the new ones or those already old and torn? 4. In what way is your life different because of the mosquito net you received? Probes a. Did you receive enough mosquito nets for your family? What did you do if not? b. How and when do you use a mosquito net? c. When do you NOT use it? d. When you received nets from the campaign, did you already have some nets at home from another source? What did you decide to do with these? e. What do you do with old or unused mosquito net? 5. What other thoughts do you have that we should have talked about, but didn’t? 6. Suppose that you were in charge of mosquito net distribution and could make one change that would make the program better. What would you do? Closing Statement: Recap discussion, thank participants, and summarize any next steps. In-depth Interview: District Health Officer The focus of this evaluation will be to better understand how the universal bed net campaign was done in your district/community as well as to learn what effect it has had on beneficiaries. Input from this evaluation will be shared with staff and funders in order to make improvements in program implementation. My purpose in talking with you today is to learn more about your thoughts, feelings, and experiences with the universal bed net campaign. Anything you tell me will not be personally attributed to you in any reports that result from this evaluation. All of the reports will be written in a manner that no individual comment can be attributed to a particular person. Your participation in this interview is completely voluntary. Are you willing to be interviewed? Do you have any questions before we begin? 1. How was UCC implemented in this district? Probe a. Are there peculiar conditions to this district that affected implementation? b. Where there any local modifications made? If so, why? 2. Thinking generally about UCC, did it work well? Probe: a. What do you think was the aim or purpose of UCC? b. What was it meant to achieve in this district? c. Were goals met in a timely manner? 3. Thinking now specifically about the UCC registration phase, what issues during the phase could hinder the impact of UCC? 75 Probe a. Household numeration and registration issues b. Awareness creation and social mobilization issues c. What could be the reason that in this district the results from registration were much higher than from the census that was done just a few months later? 4. Thinking now specifically about the UCC distribution phase, what issues during the distribution could hinder the impact of UCC? a. Delivery of nets and materials b. Reach to all populations in rural/urban, difficult to reach areas c. Ensuring quality data d. IEC and social mobilization 5. Thinking now specifically about the UCC post-distribution phase, what issues during the post￾distribution could hinder the impact of UCC? a. Data quality b. IEC/BCC on net hanging/use, maintenance and replacement c. High risk people in the household 6. What changes to health services do you think were the due to UCC? And why? 7. What can be done to ensure the UCC works more effectively? What factors would encourage more effective performance? 8. Finally, is there anything we haven’t covered that you’d like to tell me about UCC? Key Informant Interview: District Focal Point/VHT/LC The focus of this evaluation will be to better understand how the universal bed net campaign was done in your district/ community as well as learn what effect it has had on beneficiaries. Input from this evaluation will be shared with staff and funders in order to make improvements in program implementation. My purpose in talking with you today is to learn more about your thoughts, feelings, and experiences with the universal bed net campaign. Anything you tell me will not be personally attributed to you in any reports that result from this evaluation. All of the reports will be written in a manner that no individual comment can be attributed to a particular person. Your participation in this interview is completely voluntary. Are you willing to be interviewed? Do you have any questions before we begin? 1. How were UCC communication activities implemented in this district? 2. What were the different BCC approaches conducted in the district related to the UCC and how effective were they? Probe a. Please give examples of advocacy at local government levels b. Please give examples of community mobilization to increase awareness and correct misconceptions about LLINs and malaria c. Please give examples of behavior change communication to influence use of LLINs in malaria control 3. Please describe the different communication channels used during the UCC? Probe Radio, film vans, pictorial posters, interpersonal communication, religious platforms, music, dance and 76 drama, hotline 4. What do you think of the effectiveness of these BCC approaches towards Probe a. Net retention and use b. Need to use the nets every night, irrespective of the season c. Correct way to hang and use the nets d. Who should be given priority for sleeping under the nets (under the age of five, pregnant women) 5. Were these the most appropriate channels for each target audience in the district? Probe: Rural/ Urban, literate/illiterate 6. Was the timing and frequency of the BCC sufficient? 7. What was the quality of messages delivered? 8. To what extent were the communication channels suitable in influencing daily use of nets by all people? 9. Were there “hang-up” campaigns with focused messages accompanying mass distribution campaign to maximize ITN use in the community? 10. Did primary school teachers promote LLIN use among the primary school children in their respective schools? And if so, how effectively did they perform their roles? 11. What IEC materials were used to disseminate information on malaria prevention? Ask to see them. 12. Probe for posters, leaflets, laminated sheets and fact sheets. 13. What would you recommend to be changed in future? 77 SUMMARY OF FIELD DATA District Interview Thematic summary BCC Registration and Distribution Household Supply Apac KII Value of nets is clear Need better district engagement Net use messaging is pervasive (beyond UCC) 6 FGDs, 54 respondents Clear messages to avoid net misuse Some households missed Some had pre-existing nets but large households did not receive adequate nets Was unclear how many nets were to be received Registration sheets not always available for verification Not enough nets at distribution Messages through LC and radio Distribution at village level preferred Inadequate nets Hoima KII Net use is established Lack of district engagement Missed households IPC and radio Not enough time for full registration 3 FGDs Many channels Registration errors yielded additional nets for households Soroti KII VHT IPC and radio were preferred Respondents felt figures were exaggerated Some households missed 4 FGDs Multiple channels and net use appreciated Registration was not transparent and had errors Jinja KII Decentralized approach will be more effective 4 FGDs, 49 respondents Radio, loudspeakers, IPC with VHTs Do not bypass DHO in future UCC Ghost registrations Sensitization on UCC activity details was limited Compensation was not enough Insufficient time to complete activities Over-registration occurred at benefit of LCs District data management skills not used 78 Annex V: Sources of Information Documents and materials considered • Uganda Malaria Control Strategic Plan 2010-2015 • Uganda Malaria Reduction Strategy (UMRS) 2014-2020 • LLIN Distribution Detailed Implementation Guidelines, Ministry of Health • LLIN Distribution Training Manual, Ministry of Health • LLIN Process Evaluation Draft Report Phase I • National Communication Strategy for Malaria Control in Uganda 2012 -2015 - Working draft • MAAD Advertising Campaign Proposal • MAAD Advertising Campaign Plan • MAAD Net Collection Announcements • MAAD Registration and Distribution Talk Show Guide • MAAD Net Usage Talk Show Guide • MAAD Net Usage DJ Mention Guide • Stop Malaria Project Home Visit Guide • Stop Malaria Project LLIN Job Aid • Universal Coverage of LLINs in Uganda - Insights into the Campaign Implementation (Oct 2014) • Field Reports from BCC Evaluation Team • Stakeholder Interviews (Aug 2014) People interviewed Organization Name Affiliation QED/The Learning Contract Jennifer Dahnke Chief of Party, Ag. Ruth Nanyonga Activity Manager Augustine Wandera M&E Specialist Cliff Musimenta Director, Finance USAID/PMI May Mwaka Program OfficeR, M&E Dr. BK Kapella PMI Resident Advisor/CDC Dr. Gloria Sebikaari PMI Program Specialist Joel Kisubi PMI Program Specialist National Malaria Control Program Dr. Albert Peter Okui Program Manager, Ag. Dr. Denis Rubahika Malaria Officer Dr. Henry Katamba M&E Officer Ministry of Health Dr. Jim Arinaitwe GF Coordinator Malaria Consortium Dr. Godfrey Magumba Country Director Dr. Sam Gudoi Technical Advisor Uganda IRS Project/ABT Dr. JB Rwakimari Country Director Betty Tukei Program Officer MOH Data Resource Center Dr. Eddie Mukooyo Asst. Commissioner Key Informants during field visits Organization Name Affiliation 79 Jinja District John Rex Aachilla Regional District Commissioner District Internal Security Officer Olive Hope Nakyanzi Chief Administrative Officer Jane Francis Mirembe Deputy District Health Officer (DDHO) Dr. Gilbert Baayenda District malaria focal person Dorothy Buwenge sub-county health assistant Alfred Makoka Walukuba sub-county health assistant Faisal Baaka LCIII chairperson Mutai Stephen Natswenda Buwenge Subcounty coordinator Apac District DDHO Ayo Francis Chair, LC III CAO District malaria focal person Soroti District DHO Elwongu Julius LC chair Achol Michael VHT coordinator Hoima District Tivu Mark Deputy CAO Dr. Ruyonga Joseph DHO Thomson Isingoma District malaria focal person Sande Judith VHT- Dwola Hanifa Kabayaga VHT-Kahoora Aidah Birungi VHT-Businsi 80 Annex VI: Evaluation Team Composition Evaluation team composition All team members declare that they have no conflict of interest for this evaluation. Detailed description of roles and responsibilities TABLE 1 - EVALUATION TEAM MEMBERS’ ROLES Evaluation Team Leader • lead technical discussions with USAID program office and the multi stakeholder core evaluation management committee • lead methodology finalization • ensure evaluations components are done according to agreed methodology and timeline and ensure overall consistency • lead the statistical modeling and guide technically all other data analysis • facilitate synthesis of findings • lead writing of the evaluation report • quality assurance of all evaluation outputs In-country Evaluation Team Malaria Technical Expert • support the Team Leader in overall evaluation methodology development and implementation • lead and technically coordinate in-country secondary data collation and primary data collection • conduct part of KII • lead literature review on other country data • contribute to data analysis • contribute to report and other deliverables writing National Evaluator, Statistician and Researcher 81 • secondary data collation and primary data collection for health and campaign data (cost, BCC strategy and materials, etc.) through data and document review, KII and FGD • contribute to data analysis under the guidance of the Team Leader and Malaria Technical Expert Desk based technical experts LLIN cost effectiveness and BCC experts • provide short term high level inputs in the areas of cost-effectiveness and BCC Management Program Manager • lead contractual discussions with USAID program office, ensure efficient evaluation implementation and financial management Research Assistant • support to plan and organize the field work, in particular the FGD 82 Annex VII: Detailed Findings Due to the significant space limitation in the main body of the report it was not possible to present the evidence produced by the team in sufficient detail to allow the reader to follow all the inferences made. This annex therefore presents the findings for the four evaluation questions in more detail. Quality of UCC Registration and Distribution Data Accuracy The quantitative analysis of the UCC data for accuracy was done in two steps; first, the population registered was considered and compared with projections from the 2002 census (which had served as benchmark for the procurement of LLIN), the 2014 census results, and between the two UCC data sources (records vs. database) for the 19 districts of the UCC database extract. The findings were triangulated with the three demographic measures of “% of children under five”, “% of currently pregnant women” and “mean household size” from MIS 2014 and census 2014 in order to assist the interpretation of findings; second, the number of nets allocated and distributed were explored for agreement and/or variation between the two UCC data sources and in relation to the registered population (people per net distributed). Including the four pilot districts the population registered for the UCC according to the UCC records (Table 9) was 41.0 million and 6.2 million more than the 34.8 million counted in the census of 2014 which had been done immediately after the end of the UCC in August 2014. This is a significant number and a variance of plus 15.2%. When analyzed by census region the difference was largest in the North with 19% and very similar in the other three regions with around 14%. 83 TABLE 9: REGISTERED POPULATION FROM UCC RECORDS, DATABASE EXTRACT AND CENSUS 2014 Category Sub category Population Relative variance UCC records Census 2014 UCC database Records vs census in % of records Database vs records in % of database Database vs census in % of database UCC total 41,034,354 34,813,459 15.2% Regions Central 11,122,119 9,579,119 13.9% Eastern 10,521,225 9,029,408 14.2% Northern 8,938,752 7,230,661 19.1% Western 10,452,258 8,974,271 14.1% UCC database extract 7,764,316 7,186,478 7,933,165 7.4% 2.1% 9.4% Variance group High 2,846,208 2,221,131 2,842,500 22.0% -0.1% 21.9% Slightly high 3,233,838 3,045,037 3,194,171 5.8% -1.2% 4.7% Low 1,684,270 1,920,310 1,896,494 -14.0% 11.2% -1.3% The 2014 population projected from the 2002 census was 35.6 million which was close to the actual 2014 census results. But mapping the variance between the 2014 and 2002 census data shows that the fit by district was very poor (Figure 16C) , especially in the North where most districts were significantly lower in the census 2014 than projected suggesting that the growth rates decreased compared to the previous census period, most likely due to out-migration. This over-estimation by projections from the 2002 census in the North was also true when comparing to the UCC registration results (Figure 16B), while for most other districts the UCC registration was higher. This confirms that using over 10 year old census data for the calculating the procurement need for a UCC always has a significant risk of error. The mapping of the variance between the UCC registration and the 2014 census population (Figure 16A) shows that there were only seven districts (6%) where the UCC registration was below the census; for 24 (21%) it was less than 10% above the census and in 81 (72%) the variance was above 10% and up to 57% (Kaabong District). The largest proportion of high positive variance was in the North (83% of districts) followed by the East (78%), West (65%) and Central (58%). 84 FIGURE 16: VARIANCE BETWEEN POPULATIONS A) UCC RECORDS VS. CENSUS 14; B) UCC RECORDS VS. PROJECTION FROM CENSUS 2002; C) CENSUS 14 VS. PROJECTION FROM CENSUS 2002 Table 10 presents the population results from the UCC database extract in total and by the three variance groups in comparison to UCC records and 2014 census. The total population was lower by 169,000 in the UCC records by district but most of the difference came from those districts that had low populations compared to the census and these were 11% higher in the UCC database than in the UCC records. Comparison of the UCC records by sub-county with the UCC database provides two important observations; first, among the 255 sub-counties there was not a single one where the data in the UCC records and UCC database was exactly the same and this was true for all variables (population, households, nets allocated and nets distributed). While some differences are to be expected from corrections of entry error or omissions (see also Figure 13), this 100% variation appears too high and would imply that in each sub-county there was at least one parish or LC1 that had incorrect data which would be worrying; second, the UCC records by sub-county also did not agree always with the UCC records by district which was especially true for the low variance group and shows that there was considerable inconsistency or confusion in managing the aggregation of the various forms which could have been avoided had the UCC database been available during the distribution process. TABLE 10: REGISTERED POPULATION AND VARIATION BETWEEN SOURCES BY DISTRICT 85 District Population Relative variance UCC records Census 2014 UCC database Records vs census in % of records Database vs records in % of database Database vs census in % of database Amuru 335,013 190,516 299,808 43.1% -11.7% 36.5% Apac 443,637 368,786 447,881 16.9% 0.9% 17.7% Hoima 699,295 573,903 709,597 17.9% 1.5% 19.1% Iganga 616,089 506,388 614,438 17.8% -0.3% 17.6% Kiruhura 396,188 328,544 398,057 17.1% 0.5% 17.5% Soroti 355,986 252,994 372,719 28.9% 4.5% 32.1% Isingiro 499,323 492,116 513,364 1.4% 2.7% 4.1% Kibaale 862,832 788,714 837,711 8.6% -3.0% 5.8% Kisoro 307,739 287,179 309,854 6.7% 0.7% 7.3% Masindi 304,290 292,951 329,128 3.7% 7.5% 11.0% Mityana 357,860 331,266 357,099 7.4% -0.2% 7.2% Mukono 624,816 599,817 566,952 4.0% -10.2% -5.8% Ssembabule 276,978 252,994 280,063 8.7% 1.1% 9.7% Adjumani 171,808 232,813 197,435 -35.5% 13.0% -17.9% Alebtong 206,876 225,327 283,647 -8.9% 27.1% 20.6% Bududa 181,886 211,683 179,625 -16.4% -1.3% -17.8% Jinja 462,639 468,256 548,683 -1.2% 15.7% 14.7% Masaka 281,596 296,649 312,097 -5.3% 9.8% 4.9% Yumbe 379,465 485,582 375,007 -28.0% -1.2% -29.5% In order to see at which level registration data differed from the expected, demographic variables derived from the registration data of the UCC database extract were compared with results from the 2014 MIS and the 2014 census and results are shown in Figure 17 and Table 11. For all three groups of variance between UCC registration and census, the proportion of children under five and pregnant women among the total population was higher in the registration data by 3-4%-points for children under five and 0.5- 0.8% points for pregnant women. In contrast, the mean number of people per household was very similar between registration and MIS 2014 and slightly lower in the census, but this difference was not sufficient to explain the variance in population suggesting that most of the variance came from a higher number of households and only to a lesser degree to more people per household even though the specific risk groups were exaggerated compared to the representative household survey. 86 FIGURE 17: COMPARISON OF DEMOGRAPHIC VARIABLES COMPARISONS MADE BETWEEN UCC DATABASE EXTRACT, MIS 2014 AND CENSUS 2014 BY VARIANCE GROUP OF POPULATION REGISTER VS. CENSUS. TABLE 11: DEMOGRAPHIC METRICS COMPARISON BETWEEN SOURCES Variable Variance category UCC database MIS 2014/15 (95% CI) Census 2014 Persons/household High 4.91 5.05 (4.51 – 5.59) 4.79 Slightly high 4.62 4.60 (3.92 – 5.28) 4.49 Low 4.79 4.98 (4.40 – 5.55) 4.66 Children under five High 23.7% 19.3% (17.6 – 21.0) Slightly high 24.4% 20.3% (18.1 – 22.8) Low 24.7% 19.9% (17.5 – 22.4) Pregnant women High 2.9% 2.0% (1.3 – 3.0) Slightly high 3.0% 2.2% (1.5 – 3.1) Low 2.9% 2.3% (1.5 – 3.3) Another approach taken to explore the way in which the data differed was to plot the variances of different variables against each other expressed as percentage as shown in Figure 18. Data along the diagonal in these graphs are linearly correlated, i.e. very similar for both plotted relationships. Data points in the upper right and lower left quadrants show same direction of variance while those in the upper left and lower right quadrant indicate opposite directions of variance. First (panel A) the population variance between UCC database extract and census is compared to the population variance between UCC database and UCC records. This shows that even though variance was higher between registration database and census, there was still considerable variance, i.e. inconsistencies, between the database and the records and both variances were not well correlated meaning they differed in different ways. In contrast, the variances in populations UCC database to census and UCC records to census (panel B) correlated reasonably well with only 4/19 districts far from the diagonal and only 4/19 showing divergent direction of variance. This implies that in about 80% records and database varied in similar ways from the census data, but with the majority of data points above the diagonal, the variance in the database was larger than in the records. Finally (panel C) the household variance UCC database against census is compared to the population variance UCC database to census and here the linear correlation is even more evident. Most data points (13/19) are found below the diagonal meaning that variance for households was lower than that for the population supporting the previous observation that the difference in population was mostly 87 an increase in households but to some extent also an increase in people per household. FIGURE 18: CORRELATION OF VARIANCES BETWEEN UCC DATABASE, RECORDS AND 2014 CENSUS How does the Uganda experience of a significantly higher population in the registration than in a census done at the same time compare with previous campaigns in Uganda or in other countries? Even though in most of Uganda the 2010 mass distribution of LLIN was targeted to children and pregnant women, in some districts in the West a UCC was undertaken as part of a project funded by the British INGO Comic Relief and implemented by Malaria Consortium. The Registration records for Hoima, Kiboga (including Kyanwanzi at the time) and Buliisa were used to compare to census and 2014 UCC registration records21. While for two districts the results were closer to the 2014 census with a variance of -2% (Hoima) and +4% (Kiboga) and considerably far from the UCC records (+19% and +10% respectively) suggesting that here the census is probably the more correct result, the third district was closer to the UCC records with +3% (Buliisa) and far above the census with +29%. 21 In order to adjust for population growth 2010 to 2014 the registered population was projected forward using a 3.0% growth rate as indicated by the 2014 census. 88 Experience from many other countries exists, but is poorly documented. The NetWorks project attempted to explore the discrepancies between the projected procurement need based on the 2002 census and the 2010 campaign registrations in 18 districts in Senegal (unpublished data from Eric-Marie Dupuy). Overall the variance registration to census projection was +7% but varied significantly between districts from - 60% to +26%. Another example comes from the 2010 UCC in Tanzania where sleeping places rather than population were registered [8]. At national level the variance of registered sleeping places to projection from the 2002 census was +5% but varied between the seven zones from -13% to +30%. In Senegal and Tanzania the census was 8 years before the UCC and differences are to be expected. In the Uganda case with the census done immediately following the UCC one would expect variances to be lower rather than larger. TABLE 12: DISTRIBUTED AND ALLOCATED LLIN IN UCC RECORDS AND UCC DATABASE Category Sub category LLIN distributed during UCC Relative measures UCC records UCC database % of allocated distributed UCC records % of allocated distributed UCC database Persons per LLIN distributed UCC records Persons per LLIN distributed UCC database UCC total 22,267,777 99.3% 1.84 Regions Central 5,673,744 99.5% 1.96 Eastern 5,746,076 98.7% 1.83 Northern 5,021,127 98.7% 1.78 Western 5,826,830 100.0% 1.79 UCC database extract 4,491,081 4,349,273 99.5% 98.9% 1.73 1.82 Variance group High 1,599,265 1,538,489 99.5% 98.1% 1.78 1.85 Slightly high 1,839,988 1,768,413 100.3% 99.4% 1.76 1.81 Low 1,051,828 1,042,371 99.5% 98.1% 1.78 1.85 The qualitative data from the field visits largely confirms that in some cases registration data was inflated and also confirmed the two major mechanisms suggested by the quantitative analysis. In all four districts KII as well as FDG respondents mentioned that the numbers of household were exaggerated both, by households themselves as well as by LC1. At the same time some households registered multiple times or “ghost households” were created thereby increasing the number of households in the registration data. Another aspect that was consistently mentioned in the interviews in all districts and which is likely to have contributed to issues around accuracy of the registration data is the fact that district teams were not involved in the processing of the data as the forms when directly from sub-county level to the center. 89 TABLE 13: LLIN DISTRIBUTED AND VARIATION BETWEEN SOURCES BY DISTRICT District LLIN distributed during UCC Relative measures UCC records UCC database % of allocated distributed UCC records % of allocated distributed UCC database Persons per LLIN distributed UCC records Persons per LLIN distributed UCC database Amuru 185,760 165,153 99.1% 99.4% 1.80 1.82 Apac 248,823 245,318 99.9% 98.4% 1.78 1.83 Hoima 386,298 382,562 99.8% 97.6% 1.81 1.85 Iganga 345,120 336,867 99.0% 99.4% 1.79 1.82 Kiruhura 219,387 216,968 99.4% 99.8% 1.81 1.83 Soroti 213,877 191,621 100.0% 93.5% 1.66 1.95 Isingiro 288,074 282,429 103.8% 99.9% 1.73 1.82 Kibaale 473,463 462,714 99.7% 99.3% 1.82 1.81 Kisoro 171,208 170,150 99.8% 99.3% 1.80 1.82 Masindi 178,813 179,843 99.9% 98.1% 1.70 1.83 Mityana 203,921 202,373 100.0% 99.9% 1.75 1.76 Mukono 368,309 315,471 98.2% 99.3% 1.70 1.80 Ssembabule 156,200 155,433 101.9% 99.6% 1.77 1.80 Adjumani 114,840 108,521 100.0% 99.9% 1.50 1.82 Alebtong 154,190 156,024 90.6% 99.8% 1.34 1.82 Bududa 99,494 98,994 97.5% 99.8% 1.83 1.81 Jinja 301,107 300,900 100.0% 99.7% 1.54 1.82 Masaka 172,315 174,183 97.5% 98.7% 1.63 1.79 Yumbe 209,882 203,749 101.8% 99.4% 1.81 1.84 The second part of the data quality assessment focused on the nets allocated and distributed in relation to the population registered between the UCC database and UCC records. Data for nets distributed is shown in Tables 12/13 and indicate that at national level 99.3% of the allocated nets were distributed with 1.84 persons per net distributed which is close to the 1.78 estimated for UC [6]. The comparison for the UCC database extract with the UCC record indicates that while the database population had been higher than the records, it was the other way around for the nets with 141,808 less nets in the database. This was true for all three variance groups. As a result the population per net distributed was lower in the UCC records than in the database. This can also be seen in Figures 18 and 19 where the population per net from the UCC database is plotted against UCC records (left panel). While 6/19 data points were less than 1.80 in the records indicating some level of oversupply compared to the registered population, all these values were “corrected” in the database to very close to 1.80 with one now a bit too high with 1.95. The phenomenon can also be seen when the variance in population per net between UCC database and records is plotted against the population variance (Figure 19, right panel): for 16/19 data points the population per net variance is negative (i.e. lower in the database) in 15/19 data points the population variance is positive (i.e. higher in the database) indicating that generally population was increased and nets reduced resulting in a minimized variance around the ideal 1.78 people per net. The same was seen for the data at sub- 90 county level (Figure 20): while the relative standard deviation22 for population per net allocated or distributed in the records was around 18%, it was only 1.7% and 1.9% in the database. FIGURE 19: POPULATION PER NET DISTRIBUTED FROM UCC DATABASE AND RECORDS FIGURE 20: POPULATION PER NET DISTRIBUTED BY SUB-COUNTY FROM UCC DATABASE AND RECORDS Even though the lack of variance is a bit suspicious, this does not necessarily mean that the “corrections” were not reflecting a true correction in the records during data entry. However, the case of one of the selected districts, Soroti, suggests that this certainly was not always the case. First, Soroti together with Busia District comprised wave 1 of the UCC, the only occasion where the electronic database was used for net allocation and still the records and database disagree with 1.66 population per net in the records (suggesting some oversupply) but +4.5% population, - 4.3% nets allocated and -11.6% nets distributed in 22 The standard deviation expressed as % of the mean (also called coefficient of variance) 91 the final database resulting in a people per net ratio of 1.95. Data from the MIS 2014 (see also next section) as well as the qualitative data from that district suggest that indeed, there was some oversupply: 18 months after the UCC still 80% of people had access to a campaign net, 4.6% of households reported having some "extra" nets" and 26% had some of their nets not hanging even though 92% of those with access to a net used it that night before. So there is good evidence that indeed there was some oversupply and the original record was more accurate than the final database. In summary, the analysis of data quality of the UCC showed that there was an issue with timeliness of the electronic database and some reliability issues around the metric “nets allocated and delivered to sub￾county” which was inconsistently defined making it impossible to use as a management tool to improve distribution efficiency. However, the main problem was found in the accuracy of the data which came from three major sources: i) an exaggeration of the registered population which was predominantly an increase in number of households and to a lesser extent an increase of the people per household; ii) inaccuracies in data aggregation and/or transcription leading to significant differences between the different versions of data available for the UCC; iii) accuracy issues around the nets distributed in comparison the population which at least in part are suspected to be an “over-correction” of data in the UCC database to match the expected “need”. A critical question is what the likely “true” population of Uganda was at the time of the UCC and hence how many LLIN would have been needed. It is – of course – impossible to provide a definite answer in the context of this evaluation, but comparing all five census results since 1969 with the result of the UCC registration (Figure 21) shows that the 41.0 million clearly appears too high as it would imply a growth rate of 4.5% since 2002 which is unlikely as the 2011 DHS [7] reports a reduction in the fertility rate from 6.9 in 2000 to 6.7 in 2006 and 6.2 in 2011. Such a reduction is also consistent with a reduction of the population growth rate reported in the census [5] from 3.2% 1991 to 2002 to 3.0% from 2002 to 2014. However, it is also possible and even likely that each census under-estimates the true population to some extent. So if we assume that 5%-points of the 15% variance between UCC registration and 2014 census was due to census errors and 10%-points due to registration errors, one can estimate the “true” population to be 36.9 million. This would imply a need of 20.7 million LLIN for the UCC meaning that in this scenario 1.5 million LLIN more than necessary had been distributed. FIGURE 21: POPULATION GROWTH FROM CENSUS IN COMPARISON TO UCC REGISTERED POPULATION 92 II. Targeting and Quality of BCC and Effects on Net Use BCC Strategy for UCC The BCC strategy was taken from the National Communications Strategy for Malaria 2012-5 and further informed by the UC pilot conducted in 4 districts in August/September 2012. A number of lessons were learnt during the pilot that informed the main UC strategy, but not all activities could be replicated at scale. It appeared that the pilot implementation featured a number of BCC activities including local radio, banners, film-vans and T-shirts for voluntary health team members (VHT) who undertook interpersonal communication (IPC) using job aids. There was a sufficient budget for both-above-the-line and below￾the-line media and IPC during the pilot and this was well coordinated with the registration process and net distribution. Follow-up with households having received their nets was also made possible, requiring the VHTs return to give further IPC around net use, termed the ‘Hang-Up’ campaign. The pivotal role of the VHTs was acknowledged during the pilot. However, when attempting to replicate this activity at scale for the main UCC, a number of problems in management, coordination and funding prevented full implementation. The Implementation Guidelines drawn up for the UCC list a number of activities under strategies for Advocacy, Social Mobilization and BCC (p23-24) at national and sub-national level. The strategies outlined aimed to co-opt a number of different bodies to support the campaign including faith-based organizations, telecoms companies, oil and gas companies, etc. but this did not materialize in any systematic way in practice. The UCC had a number of strategic aims at sub-national level which were not possible in practice. These included: mass media campaigns led by the District Health Educators, supported by the district leaders including the Regional District Commissioner, Local Council Chairperson District Level (LC5) and District Health Officer (DHO). These activities were to focus on local radio translated into local languages. This laudable aim was not undertaken during the campaign. Many stakeholders interviewed for the evaluation, including District Health Educators (DHE) in some districts, would have welcomed this initiative. It was acknowledged that this would have enabled a greater degree of flexibility in campaign execution at local level and a wider coverage of local radio stations, as well as allowing some sense of ownership of the UC campaign at district level. As it was, the District level stakeholders felt that this was a top-down directive from central government with very little input allowed from district level leaders. A further strategy planned, but with patchy implementation, was the demonstration of LLIN use at 93 distribution points with IEC sessions. IPC led by VHTs was planned and implemented. However, the use of IEC materials including posters, leaflets, laminated sheets and fact sheets, was not fully executed and relied on variable supplies from previous campaigns. During the pilot program, the implementers found that leaflets were not useful for the UCC. A final strategy articulated in the guidelines was the inclusion of primary schools to be sensitized and co-opted within the campaign. This strategy, although strongly desired by a number of stakeholders, was not executed systematically during the campaign. However, those districts that did manage to organize such activities e.g. Soroti, stated that teachers may not have had full information, but they still believed that the impact on children was positive in encouraging net use. The training content, contained in the Training Manual (p11-21) prepared especially for the UC C, was thorough in addressing the information required for the various bodies involved in the campaign at District level down to VHT IPC messages for households, including ‘Take Home Messages’ and answers to frequently asked questions. There was a significant amount of information to retain, so it is not clear whether VHTs with variable levels of education would have necessarily retained the information in enough detail to pass on comprehensively, hence making the use of job aids necessary. The evaluation team was not able to access any other strategy documents or implementation plans. With the management of BCC activities falling to the INGOs, rather than being coordinated from government departments a highly centralized implementation resulted. This led to a reliance on mass media which could be placed and bought by one agency from Kampala. From stakeholder interviews it is clear that many believe, both at central level and district level and below, that the BCC strategy would have benefitted greatly from being decentralized. There would still be issues of coordination, which would have required close links to the logistics sub-committee particularly. However, even with the centralized structure, the coordination between radio airing and actual dates of registration or net distribution were on occasion misaligned, sometimes by two weeks, or more when nets failed to be delivered due to late procurement problems affecting wave 6 of the campaign by two months. The target audience selection was taken from the National Communications Strategy document. The primary target audience was the entire population in the UC campaign, reached in some cases through a door-to-door exercise of registration by VHT’s. Although not all of the secondary audiences were targeted during the UC campaign, a key feature of success was the mobilization of District and sub￾county level personnel behind this campaign. The social mobilization effort was a huge task and was undertaken by enthusiastic people recognizing the importance of what they were being asked to do. From interviews, it was clear that many of the sub-county personnel had not been required to work together in such a way before, e.g. District Internal Security Officers and Sub-county Internal Security Officers at village level and that the UCC forged new and unexpected partnerships at the district and sub-county levels. Further the importance of the VHTs, an often overlooked and under-resourced element in the health care arsenal, was essential and key to the success of an exercise of this scale. A follow-on post-campaign strategy was mentioned by stakeholders as essential. The evaluation team have not been made aware of such a plan. VHTs would be vital to this program. To this effect it may be useful to consider the cost/benefits of a communication campaign using the database of mobile phone numbers taken from net recipients during household registration. The Household Registration Form contains telephone numbers of LC1, VHT and residents – this is a potentially powerful database for a future e-campaign if the technology is viable and funding available. Implementation Teams Due to the pooled nature of the BCC funding (PMI and Global Fund), there were two implementing agencies appointed, the INGOs: Stop Malaria Uganda and Malaria Consortium, responsible for BCC. There were some critiques from stakeholders of these organizations not having the capacity to handle the requirements of a national BCC campaign of this scale and duration. However it was apparent that both agencies were required to step up to cover for the breach in management and coordination at the central 94 level and inevitably some problems remained. Each agency was responsible initially for different regions of the country as specified in their operating agreements; but this was then changed during the execution to focus activity on each wave of distribution. Under the INGO’s teams of supervisors and coordinators were also assigned to the districts to carry out training for social mobilization efforts among District Leaders and trainers to be taken down to village level throughout the country. Implementation Implementation was not carried out as fully as planned. Mass media, below-the-line materials and IPC were all anticipated. However, during implementation problems arose. A key constraint in implementation was an insufficient budget to cover all planned activities. Along with the lack of effective leadership and management, this meant that compromises had to be taken in implementation e.g. the decision to centralize all mass media purchase. An advertising agency was commissioned to create and air radio advertisements and talk show formats. A media plan was made available to the evaluation team, but it is not clear if indeed this plan was implemented, nor is it comprehensive for the whole campaign. From interviews with the advertising agency, it is likely that the weight of the radio campaign was lighter than planned, since it was not possible to fund the planned heavy-weight blitzing of advertising at the key times of registration and distribution. There were occasional problems in misalignment of advertising and on-the-ground registration or distribution. Coordination of these activities was more difficult in practice at scale, than in the pilot. A further constraint on the radio campaign was that it had to be centrally bought with donor funds which required greater accountability restrictions on choice of media used. This placed a requirement on station selection - only stations monitored by the Ipsos monitoring agency could be used. Ipsos only monitor larger accountable radio stations and so small local stations could not be used for the campaign. It is likely that the expenditure on BCC was heavily weighted toward mass media rather than interpersonal communication (IPC). In retrospect a number of stakeholders felt that the emphasis should have been placed on IPC for a campaign of this nature. Mass media has a role in awareness generation, but without sufficient funding for heavy-weight targeted activity, it loses its power to effect the population at large. Since the campaign reached down to householder level – it is imperative to capitalize on this interaction to enable effective face-to-face IPC during exchanges through a trained cadre of VHT’s with individual householders during registration and net distribution, as well as post campaign. TV was not used for the UCC specifically; however, previous national malaria television advertising may have been running as public service messages. TV was not integrated into the UCC. Film vans were also not used systematically – there was no NMCP coordination of this resource for the campaign. Below-the￾line materials created were limited to UCC T-shirts for VHTs. There was no public relations and PR management – any free press came from the media, rather than campaign initiated. Faith groups and Schools were not co-opted systematically, as these were not encouraged effectively from the center as planned. There was however, top-level advocacy coming from the office of the President. Both at launch and at campaign completion, President Museveni attended the events organized and spoke passionately and on￾message about malaria and net use. At local level also, implementation of key activities were encouraged by the presence of the Medicines and Health Service Delivery Monitoring Unit. This is a specialist unit, recently set up under the offices of the President to monitor all health expenditure in the country. A team of monitors were present during all waves of registration and distribution which assisted with advocacy for activities at local level, as they also assisted in ensuring participation at local level, since this Presidential focus was evident to local stakeholders in civic office which added to the importance placed on the exercise. It is important, particularly with budget constraints, to utilize materials and resources already available to save unnecessary expenditure. It was clear from respondents that some districts had materials and 95 resources (usually manpower or vehicles etc.) that they were able to utilize, whilst others did not have sufficient materials left from previous campaigns. Therefore there were not consistent universal BCC activities generated for each area of the country. It was variable. A mapping of BCC activities by region would be useful to see how differing BCC activities may have affected the campaign, but there was insufficient data to undertake such an exercise. Quality of BCC Materials A script was made available for one radio advertisement and two talk show guides (see appendix). The evaluation team was not given access to any pre-testing results for development of these advertisements, so it cannot be stated with any confidence how effectively the target audience would have received them. However, a number of concerns arise from the scripts alone. Firstly, the advertisement and script open with scare tactics and contains a number of negative assertions, rather than offering a positive behavior change message. Positive messages, pertaining to a behavior someone should be enacting and why it is good for them, tend to work better for a less knowledgeable general population than negative scare tactics. This is particularly pertinent to malaria, which is not a new disease to Ugandans. It is also pertinent to the UC campaign, which from interviews with stakeholders and recipients, was considered a highly positive initiative. Receiving a free public health commodity from the government was a very welcome exercise and more could have been made about the positive nature of this in the advertisement message. The number of people who went to the trouble of attending registering and net distribution events is testament to the effect of such an event. The advertisements and scripts also state inaccurate statistics about the number of people dying from malaria. Otherwise, the scripts for the talk shows appear comprehensive in the information that they contain. However, for this quantity of information, the interpersonal element of communication would be more effective in ensuring appropriate behavior. This could have more effectively been built on by VHTs during hang-up campaigns after the net distribution. The job aid and home visit script were developed for the targeted campaign prior to the UCC. The sample given to the evaluation team is written in English, it is not known how many languages it was translated into, or how widely this was distributed to, and used by, VHTs during the UCC. Outcomes Regarding Net Use When evaluating the use of nets and ITN it is essential to differentiate between the non-use of nest due to behavioral reasons and non-use because of the absence of a net that could be used which means that net use has to be related to the access indicator. Table 14 presents the use of ITN from the four representative household surveys 2006 to 2014 and shows a steady increase of both, the ITN use last night and access to an ITN within the household. Accordingly ITN use provided there was access was high with 75% in the first three surveys and then increased to 84% after the UCC. Although the surveys were done at different times of the year, there was no indication that ITN use if access varied significantly with the rains (Figure 22). 96 FIGURE 22: ITN USE IF ACCESS AND RAINFALL TABLE 14: ITN USE IF ACCESS TO ITN BY YEAR AND ITN USE BY TARGET GROUPS Indicator Population group DHS 2006 MIS 2009 DHS 2011 MIS 2014/15 Used ITN last night 6.9% 24.2% 34.1% 68.6% Had access to ITN in household 9.9% 33.8% 47.0% 82.1% Use of ITN if access 74.7% 75.7% 74.6% 83.6% Children <5 yrs 9.6% 32.8% 42.8% 74.3% Children 5-14 yrs 4.4% 17.6% 29.0% 62.4% Women 15-49 pregnant, not head 9.3% 46.1% 46.6% 74.7% Women 15-49 not pregnant, not head 9.9% 30.6% 39.5% 72.7% Males 15-49 not head 4.9% 13.6% 21.4% 51.5% Head of household 9.6% 30.2% 29.6% 76.5% Age 50+ not head 5.0% 24.6% 26.8% 69.8% Table 14 also presents ITN use by specific population sub-groups including the target groups of children under five and pregnant women. Even when population access to an ITN was still low, young children and pregnant women always had higher use rates with women in reproductive age but not pregnant and heads of household close behind. However, as access increased from 47% to 82% after the UCC the previous “non-users”, namely older children, men in reproductive age and persons 50 years or older, showed the steepest increases in use suggesting that the main reason for non-use had been lack of access and not a principle unwillingness to use. This is more clearly seen in Figure 23 where ITN use by age￾group, gender and household supply with ITN is compared between the four surveys. When there are not enough ITN in the household there is i) a significantly lower ITN use for age groups 5-19 years and 50+ years; and ii) a clear gender imbalance favoring women in the age bracket 10-49 years. When there are enough ITN as defined by the 1 ITN per two people indicator, the gender difference is much lower although still favoring women between aged 10 and 49 years and differences in use by age groups have largely disappeared. More importantly, in the group of households with at least 1 ITN per 2 people the age pattern equalizes increasing with increasing time suggesting that over the last eight years a strong net use culture has been built in Uganda. 97 FIGURE 23: ITN USE BY AGE, GENDER AND HOUSEHOLD SUPPLY WITH ITN TABLE 15: NUMBER OF USERS PER USED ITN BY HOUSEHOLD SUPPLY WITH ITN Background MIS 2014/15 DHS 2011 Mean users per net 95% CI Mean users per net 95% CI All ITN used last night 1.83 1.80 – 1.86 1.82 1.78 – 1.85 Less than 1 ITN/ 3 people in HH 2.33 2.25 – 2.41 2.12 2.07 – 2.17 1 ITN/ 3 people in HH 2.17 2.11 – 2.22 2.06 2.01 – 2.11 1 ITN/ 2 people in HH 1.75 1.71 – 1.78 1.69 1.64 – 1.73 1 ITN per person of more in HH 1.37 1.33 – 1.41 1.21 1.17 – 1.25 The allocation of LLIN during the UCC assumes that on average two people will share a net and this assumption has been shown to be generally true [6]. However, there are also indications that as ownership coverage increases and people such as adolescents gain access to an ITN there is an increasing proportion of nets that is only used by a single person. This can also be shown for Uganda (Table 15) where over time the mean users remained constant at 1.8 but if around or above 2.0 if there are less than 1 ITN per 2 people and drops to 1.2-1.4 if there is at least 1 ITN for every person in the household and 41% of nets being used by a single person. This implies that as universal coverage is reached the assumption of 2 persons sharing a net is no longer true and some of the users now covered cannot share a net for cultural reasons. Multi-variable logistic regression models with ITN use as the dependent variable confirm the picture presented above from the uni-variable analysis (details see Table 16). If all households owning any ITN are considered, the intra-household supply with ITN is by far the most important determinant with an Odds-Ratio (OR) of 6.4 comparing ITN use when there are less than 1 ITN per 3 people to that when 98 there is 1 ITN per 2 people and even an OR of 9.1 for 1 ITN per person. However, using males age 15-49 not being head of household as reference, pregnant women are 3.4 times more likely to use an ITN, children under five 2.9 times and women in reproductive age not currently pregnant 2.2 times. When only households with at least 1 ITN per 2 people are considered, the difference are smaller with OR 2.6 for pregnant women, 2.0 for young children and 1.7 for women in reproductive age. Heads of household and people from the poorest wealth quintile also had consistently higher ITN use and people from households with a female head had a lower likelihood of using an ITN (all other things being equal) with an OR of 0.62. Compared to the East Central region as reference the North East and Mid-West had significantly higher ITN use and Central 1, Central 2 and especially the South-West had lower ITN use. The latter was mainly driven by the high altitude areas in the South-West: in areas below 1,350 meters ITN use if access was 78%, between 1,350 and 1,550 meter 74% and above only 61%. Other variables such as urban residence, household size or education of the women interviewed for the BCC section of the MIS questionnaire were not significant in the models. 99 TABLE 16: MULTI-VARIABLE REGRESSION MODELS ON ITN USE Independent variable People from households with any ITN (N=24,731) People from households with 1 ITN / 2 people or better (N=14,028) Adjusted OR 95% CI Adjusted OR 95% CI HH ownership of ITN < 1 ITN / 3 people 1 ITN / 3 people 1 ITN / 2 people 1 ITN / person or more Reference 3.07 6.36 9.20 2.60 – 3.65 5.46 – 7.39 6.96 – 11.9 Age and gender Men 15-49 yrs Child < 5yrs Child 5-14 yrs Female 15-49 not pregnant Female 15-59 pregnant Age 50+ yrs Reference 2.89 1.40 2.38 3.14 1.14 2.44 – 3.41 1.21 – 1.61 2.04 – 2.77 2.25 – 4.39 0.94 – 1.38 Reference 1.97 1.22 1.79 2.28 1.02 1.57 – 2.48 1.01 – 1.47 1.48 – 2.17 1.43 – 3.64 0.80 – 1.30 Is head of household 3.08 2.69 – 3.52 2.41 2.05 – 2.82 Household from lowest quintile 1.23 1.06 – 1.43 1.20 0.94 – 1.53 Head is female 0.62 0.52 – 0.72 0.69 0.57 – 0.84 MIS Region East Central Central 1 Central 2 Kampala Mid-North Mid-Western Mid-Eastern North East South-Western West Nile Reference 0.67 0.72 0.99 1.15 1.34 1.03 1.98 0.45 0.94 0.48 – 0.91 0.56 – 0.92 0.73 – 1.34 0.87 – 1.52 0.96 – 1.84 0.76 – 1.39 1.63 – 2.39 0.34 – 0.60 0.72 – 1.26 Reference 0.97 0.81 1.73 1.28 1.78 1.17 2.49 0.50 0.97 0.57 – 1.62 0.53 – 1.25 1.07 – 2.79 0.85 – 1.93 1.12 – 2.83 0.75 – 1.81 1.64 – 3.76 0.33 – 0.77 0.64 – 1.48 100 FIGURE 24: ITN USE AND IRS IN MID-NORTH REGION There was also no indication of a lower use of ITN in the households that received IRS in the Mid-North region. Although in the DHS 2011 there had been a marginal difference of about 7%-points (73% vs. 81%) in ITN use if access, this difference was not visible in the MIS 2014/15 (see Figure 24). The results from the UCC evaluation on ITN use are consistent with what has been reported in other countries when the “use gap” was considered [19]. A process of growing net use culture has been seen in similar magnitude following the UCC in Tanzania (A Kilian unpublished) and has also been described for Kamuli district in Uganda as part of a cluster-randomized trial on the effects or repeated home visits on net hanging and use. While the visits had no measurable effect, the general exposure to a socio-cultural environment that accepted and promoted ITN use created a positive effect on ITN use in all study arms including the control [20]. Exposure to BCC and attributable effects During the 2014/15 MIS 93% of women in reproductive health were interviewed regarding exposure to any net related BCC in the last six months and their knowledge about malaria transmission and ways to prevent malaria (N=5,322). In total 64% of women responded that they had heard any messages about nets and this was highest in Kampala with 81% and varied in the other MIS regions between 71% in the North East and 56% in the Mid-East. Those women who had been exposed to any message mentioned on average 2 sources of information of which radio was the most common (82%) followed by community health workers (CHW) (34%), poster or billboard (21%), television (19%) and community event (17%). “From anywhere else” was mentioned by 25% of women. When this data was grouped into the major groups of media (radio, TV, poster, billboards) and interpersonal communication (IPC, CHW, community event, other) it showed that exposure to these major BCC channels differed significantly by background characteristic of the women. While exposure to IPC was very similar across education or wealth background, exclusive exposure to IPC only was more common among women with poor education and from the lower wealth quintiles (Figure 25). In contrast, exposure to media channels significantly increased with increasing 101 education, wealth. This pattern was confirmed in multi-variable regression models which showed that media exposure also increased with increasing age, but IPC exposure did not. Variables that did not influence exposure to either channel were urban residence, gender of head of household, family size, ownership of any ITN or any nets from UCC and the different UCC waves23. Exposure to media was particularly high in Kampala, Central regions and South-West and particularly low in North East and Mid-East. IPC was strongest in North East and West Nile and lowest in Central regions. FIGURE 25: EXPOSURE TO BCC CHANNELS BY EDUCATION AND WEALTH Having any correct knowledge of malaria transmission was high with 91% and 57% of women only mentioned correct responses while another 34% mentioned correct and incorrect or wrong responses. The proportion with only incorrect responses was 4%. Wrong responses were low with 6% of all women while incorrect responses were given by 36% of women. Not surprisingly, educational status was the most important determinant for this outcome and women with higher education had 82% correct only responses and less than 1% incorrect only while women without schooling had 43% correct only and 8% incorrect only responses. There was also some indication that media exposure was working somewhat better in creating correct knowledge on malaria transmission that IPC although the effect was programmatically small even if statistically significant (Figure 26). FIGURE 26: KNOWLEDGE ON MALARIA TRANSMISSION BY EDUCATION AND BCC CHANNEL EXPOSURE 23 Only waves 7, 8a, 8b and part of 6 fell into the 6 months bracket for the BCC question in the MIS 2014/15 102 TABLE 17: IMPACT OF BCC CHANNELS AND MALARIA KNOWLEDGE ON ITN USE OF WOMEN Independent variable* Women in households with any ITN (N=4,961) Women from households with At least 1 ITN / 2 people (N=2,972) Adjusted OR 95% CI Adjusted OR 95% CI BCC channel exposure None Media only Both IPC only Reference 0.98 1.07 1.16 0.77 – 1.24 0.84 – 1.38 0.82 – 1.63 Reference 0.73 0.91 1.06 0.53 – 1.00 0.66 – 1.27 0.62 – 1.80 Malaria transmission knowledge None Correct only Both Incorrect or wrong only Reference 0.94 0.95 1.05 0.63 – 1.41 0.62 – 1.46 0.61 – 1.81 Reference 0.83 0.82 1.07 0.46 – 1.54 0.44 – 1.51 0.46 – 2.49 Net/ITN to prevent mentioned 1.09 0.78 – 1.52 1.08 0.66 – 1.76 * model was adjusted for age, household supply with ITN, education, wealth and MIS regions However, when impact on ITN use of BCC channel exposure, knowledge on correct malaria transmission and knowledge that nets or ITN are best to prevent malaria was tested in a multi-variable regression model that also adjusted for other key determinants, there was absolutely no effect among women from households with any ITN (Table 17). When the model was restricted to households with at least 1 ITN per 2 people there was some indication that exposure to media only was associated with a lower ITN use, but this association was not very strong statistically (p=0.05). The results did not look any different when all people in households where a woman was interviewed were considered. This shows that neither recent exposure to net related BCC messages nor good knowledge on malaria transmission or prevention are directly associated with ITN use in the current situation in Uganda. This is in keeping with the steady increase in net use since 2006 (see Figure 22) and suggests that the build-up of a net use culture is a gradual change of social norms following repeated exposure through multiple channels rather than the 103 effect of one single message. The finding from the Uganda UCC seems to somewhat contradict findings from other countries [23, 24] that did find a significant impact of BCC on ITN use, but in these cases the assessments were done earlier in the development of a net use culture. In addition, the capture of BCC exposure in the 2014/15 MIS was limited in the time scale (six months) and scope of BCC channels and cannot be considered a comprehensive assessment of BCC activities. Evidence from the FGD confirms the existence of a strong net use culture, namely that “people use the nets” which was found in all four sites visited. With respect to potential misuse of nets respondents mentioned that other uses (such as making ropes to tether goats, reinforce roof or fences, constructing bath shelters, protecting chicken or in some cases fishing) are predominantly for old nets which is in keeping with other quantitative [21] and qualitative studies [22]. There also was repeated mention of a certain dislike of net users of what obviously referred to Olyset nets (rough texture, too large mesh, difficult to wash, shrink when washed). However, this had no correlation in the MIS data where there was no difference in use of any of the LLIN brands. In summary, a BCC strategy for the UCC existed, but was not followed during an implementation which was highly centralized circumventing existing district structures and relying heavily on media channels (radio) rather than interpersonal communication. The media campaign in turn was delayed only covering from UCC wave 6 onwards and the radio message scripts were in part of questionable quality using scare tactics rather than a positive or encouraging attitude. Nonetheless, ITN use among those who could have used an ITN was high with 84%, an increase of 10%-points from before the UCC and at a very high level in international comparison. While children under five and women in reproductive age (and especially those pregnant) had always been and still are prioritized when ITN in the household are scarce, the UCC enabled groups such as older children and men, who had shown lower ITN use rates before, to catch up and gender and age differences were found to be minimal if a household had ITN for all members. A side effect of this increased access and use is that there is an increased proportion of nets used by a single person which lowers the average number of people sharing a net. About two third of women had been exposed to any BCC message, mainly radio. However, the data suggests that media channels preferably reach the better educated and wealthier households and that interpersonal communication is more suitable to evenly reach the population. While knowledge on malaria transmission and the preventive potential of ITN was very high, neither this knowledge nor recent exposure to BCC messages was associated with increased ITN use supporting the notion that over the past 10 years a strong net use culture has developed in Uganda which does not depend on a single message but rather is a result of positive experiences with nets use and reduced morbidity and a gradual change of social norm. III. Health Outcomes of Malaria Morbidity, Parasite Prevalence and Anemia Malaria cases before and after UCC The number of health facilities in the HMIS system expected to submit reports increased from 2,979 in January 2006 to 3,561 in 2010 and 4,499 in 2015. The increase from 2010 to 2015 was in part due not to creating new facilities but due to the shift from the HMIS to the DHIS2 system which included that all health facilities in a district including the one run by NGOs or faith based organizations were in the DHIS2 which had not always been the case before. The reporting completeness increased during this time from 76% in 2006 to 88% in 2010 and 95% in 2015. But as shown in Figure 27 (left panel) there was considerable fluctuation in the reporting completeness and a significant drop to as low as 55% in November 2011 during the transition from HMIS to DHIS2. Not surprisingly, the lowest reporting completeness and largest fluctuation was found in Kampala which makes the data from this area somewhat “unstable”. In general the transition to the electronic, district-based DHIS2 system improved data quality and completeness significantly. 104 In total the database reported between 2006 and 2015 253 million new OPD visits in the public sector which after adjustment for under-reporting increased to 298 million. The OPD attendance rate was essentially identical for boys and girls less than five years of age, but differed between males and females in the age-group above five years with male attendance being only about two-thirds of the female attendance (Figure 27, right panel). For children under five the utilization rate was between 1.5 and 1.75, i.e. at or above the WHO recommended level of 1.5 visits per child and year. Similarly, the utilization of public facilities was around the recommended 1.0 or above for women, but only 0.6 to 0.8 for men. There was a marked increase of all utilization rates during 2011 which is the artefact described above of inclusion of more reporting non-government facilities in the DHIS2. Interestingly, there was a slight decline in the utilization rates 2012 to 2015 for all age groups and for men and women. The proportion of reported malaria cases out of all new OPD attendances was initially around 60% for children under five and between 40% and 50% for patients above five years. The ratio between OPD attendance and malaria cases was very similar for males and females indicating that the lower number of malaria cases in the database was not a result of a lower malaria incidence in men but due to their lower attendance. The OPD-adjusted malaria incidence which assumes a constant utilization rate over time which is the same for men and women was almost identical for men and women and tended to be slightly higher for men (Figure 28). This suggests that the exposure to malaria is the same for men and women but that men either do not attend health facilities as often, or obtain their treatment more frequently from the private sector not captured in the database. FIGURE 27: TREND IN REPORTING COMPLETENESS AND OPD ATTENDANCE FIGURE 28: QUARTERLY OPD-ADJUSTED MALARIA INCIDENCE RATE BY AGE AND GENDER 105 Between 2006 and mid-2010 the malaria incidence for children was around 900 reported cases per 1,000 children per year or close to one episode per child and year. Considering that this comprises only those cases attending public facilities and that DHS surveys suggest that these are only 30-40% of all fever cases in children, the overall clinical malaria incidence in this period can be estimated to be between 2.0 and 2.5 episodes per child per year. Based on the slide positivity rate of around 50% during this period, the confirmed malaria incidence in children under five is estimated to be 1.0 to 1.25 per child and year. Incidence for adults in the HMIS database is about half that of the children but projection to national level is not possible as the proportion attending the public sector is not known for this age group. The trend in OPD-adjusted malaria incidence over time is shown in Figure 29 in connection with the two national LLIN distribution campaigns: the targeted campaign of 2010/11 and the UCC. Although there is significant seasonal variation in the monthly and quarterly data, the trend in the moving averages show a declining trend which is more clearly visible in the quarterly data. Starting in mid-2010 there was a modest decline in incidence to about 750 cases per 1,000 for children and 400 for persons above five until about mis-2013. Thereafter, the decline was steeper for both age groups reaching about 600 per 1,000 for children and 300 for older persons by early 2015. Plotting the moving average of the quarterly incidence data by census region (Figure 30) shows that incidence is highest and incidence decline the lowest in the East followed by the North while Central region and West are significantly lower and show a more consistent long-term decline in malaria incidence. These relationships reflect the situation in malaria transmission intensity and epidemiology in Uganda from historical data [1]. However, in all regions there an acceleration of decline that starts between 2013 and 2014. Table 18 presents the proportionate changes in malaria incidence by age-group and region for four distinct time periods: i) from 2007 to 2010 representing a phase of focal distribution in the North; ii) 2010 to 2012 covering the national targeted mass distribution to children and pregnant women; ii) 2012 to 2015 representing the period of the UCC; iv) the overall trend 2007 to 2015. At national level the trends were very similar for children and older persons with a 33% to 37% reduction in malaria incidence 2007-15. The UCC period showed consistently the most pronounced decline of between 24% and 25% meaning that 67% to 72% of the long-term decline came during the UCC period. In contrast, decline in the first time period was moderate (1.5-6.0%) while the period of the targeted campaign showed around 10% 106 decline, less than half of the decline seen during the UCC period. Linear regression analysis confirmed that the downward trend in malaria incidence in the three phases was statistically significantly different from each other and increased over time indicating an acceleration of morbidity decline after the UCC from -5.1 cases/1,000/month to -6.3 for children under five (p<0.05). 107 FIGURE 29: TREND IN MONTHLY AND QUARTERLY OPD-ADJUSTED MALARIA INCIDENCE BY AGE PALLID LINE MONTHLY/QUARTERLY DATA, BOLD LINE 6-MONTHLY OR 3-QUARTERLY MOVING AVERAGE, U5&PW TARGETED CAMPAIGN FIGURE 30: TREND IN QUARTERLY OPD-ADJUSTED MALARIA INCIDENCE BY CENSUS REGION 3-QUARTERLY MOVING AVERAGE OF ADJUSTED INCIDENCE 108 TABLE 18: CHANGES IN OPD-ADJUSTED MALARIA INCIDENCE OVER TIME Area & age group Reduction in malaria incidence during time period* QI/2007-QI/2010 QI/2010-QI/2012 QI/2012-QI/2015 QI/2007-QI/2015 Age <5yrs National 1.5% 11.1% 24.0% 33.4% Northern -9.5% 19.5% 13.6% 23.8% Eastern -7.6% 5.1% 12.5% 10.6% Central 8.1% 2.4% 42.7% 48.6% Western 17.3% 22.6% 29.6% 54.9% Age 5+yrs National 6.0% 9.5% 25.3% 36.5% Northern -5.9% 18.1% 13.2% 24.7% Eastern -3.5% 3.3% 15.3% 15.2% Central 4.7% 5.5% 42.2% 48.0% Western 20.1% 15.9% 26.5% 50.5% * calculated from 3-quarter moving average The trends in changes in reported malaria incidence by census region shows that with the exception of the North which was influenced by the impact of IRS 2009-2014 (see below), all regions had their most pronounced decline in the UCC period. The Western region had the longest continuous decline while the Central region had a modest decline until the UCC but then showed the strongest effect during the UCC period. With only 11% to 15% decline in malaria incidence the Eastern region had the least decline which corresponds not only to the highest transmission found here but also the lowest community level ITN ownership. In order to obtain a clearer picture of the impact of the UCC malaria incidence data for each wave was grouped into the 12 months preceding the UCC and the 12 months after. Results for the aggregate data are shown in Figure 31 and demonstrate a statistically significant six fold increase of the rate of decline (coefficient from regression analysis). 109 FIGURE 31: MALARIA INCIDENCE BY AGE COMPARING 12 MONTHS BEFORE AND AFTER UCC* * Numbers show gradient (coefficient) from linear regression analysis with 95% CI in brackets and p￾value refers to the change in gradient before against after UCC TABLE 19: CHANGE IN MONTHLY MALARIA INCIDENCE BEFORE AND AFTER UCC FOR CHILDREN UNDER 5 Wave Malaria incidence rate (cases/1,000/year) 6-months moving average Change in % Before￾after* 12m before At UCC 12m after before after p-value 0 (pilot) 1,035 1,101 1,011 +6.5% -8.2% <0.0001 1 1,047 1,075 906 +2.6% -15.7% <0.0001 2 903 911 838 +0.8% -8.0% 0.05 3 946 924 836 -2.3% -9.5% 0.003 4 775 764 567 -1.4% -25.8% <0.0001 5 689 683 488 -0.9% -28.6% <0.0001 6 531 533 457 +0.5% -14.3% 0.001 7 619 574 558 -7.2% -2.9% 0.4 8a 1,006 931 792 -7.5% -14.9% 0.008 8b** 496 433 336 -12.8% -22.3% 0.3 All 839 793 717 -5.5% -9.6% <0.0001 *Significant change in time trend as assessed by interaction term time and before/after UCC in regression model ** only 10 months before and after UCC 110 TABLE 20: CHANGE IN MONTHLY MALARIA INCIDENCE BEFORE AND AFTER UCC PERSONS 5 AND OLDER Wave Malaria incidence rate (cases/1,000/year) 6-months moving average Change in % Before￾after* 12m before At UCC 12m after before after p-value 0 (Pilot) 548 554 574 +1.2% +3.6% 0.03 1 467 543 417 +16.3% -23.2% <0.0001 2 498 499 456 +0.3% -8.7% 0.02 3 470 481 409 +2.3% -14.9% <0.0001 4 433 422 318 -2.6% -24.7% <0.0001 5 407 426 307 +4.8% -27.9% <0.0001 6 284 292 230 +2.5% -21.1 <0.0001 7 296 274 276 -7.5% +0.9% 0.7 8a 411 397 336 -3.5% -15.8% 0.004 8b** 267 224 166 -16.0% -25.8% 0.3 All 424 411 369 -3.0% -10.2% <0.0001 *Significant change in time trend as assessed by interaction term time and before/after UCC in regression model ** only 10 months before and after UCC However, as shown in Tables 19 and 20 and Figures 32-34, the situation varied by wave or region and the declines induced by the UCC were less pronounced in the East (pilot, waves 1-3) and North (waves 7, 8a) than in the West and Center (waves4-6). With the exception of waves 7 (affected by IRS as shown below) and the Kampala/Wakiso wave (decline started before UCC), there was a significant acceleration of the malaria incidence trends for all waves. 111 FIGURE 32: MALARIA INCIDENCE BY AGE COMPARING 12 MONTHS BEFORE AND AFTER UCC, WAVES 0-3 112 FIGURE 33: MALARIA INCIDENCE BY AGE COMPARING 12 MONTHS BEFORE AND AFTER UCC, WAVES 4-7 FIGURE 34: MALARIA INCIDENCE BY AGE COMPARING 12 MONTHS BEFORE AND AFTER UCC, WAVES 8A-8B 113 FIGURE 35: MAP OF IRS AREAS SUPPORTED BY PMI IN NORTH AND EAST UGANDA Ten districts in the Mid-North region received two annual rounds of Indoor Residual Sparaying (IRS) between 2009 and 2014 (Figure 35), initially using the pyrethroid alphacypermethrin and due to increasing resistance switching to the carbamate bendiocarb at the end of 2010. As shown in Figure 36, there was little change of incidence in the first year of IRS followed by a rapid, about 50% decline of reported malaria incidence compared to the other districts in the Mid-North MIS region. After the initial sharp drop incidence curves for the IRS and non-IRS districts seems to run in parallel and both show a second very clear decline that coincides approximately with the UCC in this area. In the second quarter of 2015 (more specifically in the months of May and June) a significant increase of malaria cases can be observed that was more pronounced in the IRS districts than the non-IRS districts (see quarterly data in Figure 36). This also coincides with the end of the IRS program in the 10 districts and could be interpreted as a “rebound” of malaria incidence to “LLIN only” levels of protection. However, during this time there was a significant “upsurge” of malaria cases observed also in other regions of Uganda, namely West-Nile, Mid-West and Central 1 and 2 (Figure 37) while it was absent in the North East, Mid-East, East Central and Kampala. The increase of cases (and test positivity rates) is most likely an effect of above normal rainfall in the Western, Northern and Central areas of Uganda in early 2015 due to the El Niño weather phenomenon (Figure 38). Since the increase of cases is only seen in the last two months of the HMIS data time series available for this evaluation, no further assessment on the magnitude or duration can be made. But is it very likely that the increase seen in the IRS districts in Mid-North at the end of the IRS program is a combination of the two described mechanisms. 114 FIGURE 36: TREND IN QUARTERLY MALARIA INCIDENCE IN IRS AND NON-IRS AREAS IN MID￾NORTH REGION PALLID LINE QUARTERLY DATA, BOLD LINE 3-QUARTER MOVING AVERAGE OF OPD-ADJUSTED INCIDENCE 115 FIGURE 37: MONTHLY TREND IN MALARIA INCIDENCE BY MIS REGION AND IRS 116 FIGURE 38: DEKADAL RAINFALL ANOMALY EAST AFRICA MARCH TO MAY 2015 117 The results of microscopy as documented in the HMIS data suggest a slide positivity rate of approximately 50% in 2006/07 for under-fives and 48% for persons five and above (Figure 39, left panel). In the following years there was slight general decline except for a temporary increase in 2009/10 which could correspond with a period when there was a severe shortage of Artemisinin-based Combination Therapy (ACT) in Uganda. From about mid-2013 the slide positivity rates begins to drop more clearly reaching 34% for under-fives (32% reduction to 2006/07) and 27% for those five years and older (44% reduction). Rapid Diagnostic Tests (RDT) were introduced at the end of 2011 and since 2012 are test results are routinely reported in the DHIS2 system. Test positivity rates are usually a bit higher than for microscopy as the Histidine-rich Protein based RDT also capture antigenemia without viable parasites [18] and in 2012 was around 60% for under-fives and 49% for older persons. However, there also was a decline in the RDT positivity rates to 48% and 37% respectively in early 2015 before they increased again in May and June 2015 together with the previously described increase of cases. FIGURE 39: TRENDS IN SLIDE POSITIVITY AND TEST TO MALARIA DIAGNOSIS RATIO PALLID LINE MONTHLY DATA, BOLD LINE 6-MONTHLY MOVING AVERAGE The HMIS data in Uganda does not distinguish between “suspected” and “confirmed” malaria cases but rather reports malaria diagnosis and test results separately. However, relating the number of malaria tests done (microscopy and RDT) to the number cases reported of can give an idea of the level of testing. This is shown in Figure 39 (right panel) and indicates initially the proportion of suspected cases tested was low, around 20% slowly increasing to 40% in 2011. With the introduction of RDT in the peripheral health facilities without microscopy the test to diagnosis ratio rapidly increased to above 100% by early 2014 indicating that there were more tests done than malaria cases diagnosed. As the testing rate will depend on how patients are selected to be tested no exact estimate can be made, but it is very likely that now more than 80% of suspected cases attending government facilities get tested and confirmed. In an attempt to estimate the incidence of “confirmed“ malaria cases the OPD-adjusted incidence was multiplied with the test-positivity rate. Results are shown in Table 21 and Figure 40) and suggest an even more pronounced decline of incidence following the UCC. But these data also have a lot of fluctuations and appear less stable. Since it is also not clear how the testing strategy of patients had changed over time and to which extent the reported cases in the later years are already reflecting the confirmed cases (which would lead to a bias if the positivity rate is applied), this data is not used to estimate the effects of UCC on malaria morbidity in this evaluation. 118 TABLE 21: CONFIRMED MALARIA INCIDENCE ADJUSTED FOR UNDER-REPORTING AND OPD ATTENDANCE Area & age group Reduction in malaria incidence during time period* QI/2007-QI/2010 QI/2010-QI/2012 QI/2012-QI/2015 QI/2007-QI/2015 Age <5yrs National 5.5% 19.9% 39.3% 54.1% Northern -24.4% 44.8% -10.0% 24.6% Eastern -6.3% 13.9% 30.0% 36.0% Central 15.6% 6.8% 64.5% 72.1% Western 29.3% 30.3% 33.2% 67.1% Age 5+yrs National 16.8% 21.8% 44.9% 64.2% Northern -6.3% 46.0% 5.7% 45.9% Eastern 6.8% 9.4% 40.8% 5’.0% Central 16.5% 15.9% 64.9% 75.4% Western 34.4% 34.3% 26.2% 68.2% * calculated from 3-quarter moving average FIGURE 40: TREND IN QUARTERLY “CONFIRMED” MALARIA CASES PALLID LINE QUARTERLY DATA, BOLD LINE 3-QUARTER MOVING AVERAGE 119 FIGURE 41: TREND IN ANEMIA CASES BY AGE AND GENDER The reported rate of anemia per 1,000 OPD attendances showed a continuous decline from 2006 to 2015 of about 60% for children and older persons with the exception of a peak in 2009/10 which had already been described in the malaria incidence data and most likely corresponds to a period of ACT stock-out in Uganda (Figure 41). There was no specific effect visible that could correspond to the UCC. Anemia rates were slightly higher for boys than for girls and higher for women compared to men. Qualitative data from the KII as well as FGD showed that health staff and the communities are aware that that malaria has come down. In Apac the Chief Administrative Officer mentioned also the increase of malaria in early 2015 but reported in December 2015 had “malaria is now down a bit”. Changes in Parasite Prevalence and Anemia Malaria parasite prevalence in children 6-59 months reduced at national level by 55% from the 2009 MIS to the 2014 survey (Table 22) with the largest reduction observed in the Central region (63%) followed by Western (56%), Northern (52%) and Eastern (49%). Reductions occurred at all ages within the 6-59 months bracket as shown in Figure 42. When analyzed by MIS region (Figure 43) three observations can be made: i) in keeping with the reduction seen in malaria incidence in the IRS districts in Mid-North there was a dramatic 87% reduction in parasite prevalence in this area from 63% to just 8%; ii) in the Kampala area where in 2009 parasite prevalence was already low with 6% a further dramatic reduction to only 0.4% was seen which mean that in Kampala malaria is at negligible levels; ii) the least reductions were seen in East Central region where also the lowest community and household level ITN ownership coverage had been observed. 120 TABLE 22: CHANGES IN MALARIA PARASITEMIA AND ANEMIA BY AGE OF CHILD Age group Malaria parasitemia Any anemia Moderate/severe anemia 2009 2014 2009 2014 2009 2014 6-59 (95% CI) change 44.7% (39.9 – 49.7) 20.1% (17.3 – 23.2) -55% 62.2% (58.9 – 65.4) 53.0% (50.5 – 55.4) -15% 9.6% (8.0 – 11.6) 4.6% (3.8 – 5.7) -52% 6-11 31.8% 11.2% 80.8% 80.0% 20.5% 10.3% 12-23 37.1% 14.7% 74.2% 68.3% 15.5% 6.6% 24-35 45.1% 22.2% 64.8% 53.3% 9.9% 4.2% 36-47 49.4% 21.5% 52.3% 40.8% 6.2% 3.2% 48-59 53.1% 26.0% 49.1% 38.2% 2.1% 2.2% FIGURE 42: CHANGES IN MALARIA PARASITE PREVALENCE BY AGE AND REGION FIGURE 43: CHANGES IN MALARIA PARASITE PREVALENCE BY MIS REGION AND IRS For anemia the overall reduction of any anemia was only moderate from 62% to 53% but when 121 hemoglobin levels of less than 8g/dL were considered, the reduction from 2009 to 2014 was in a similar range as parasite prevalence with 52% (Table 22) with the biggest gains made in the IRS districts in the Mid-North and in the Central Regions and the least again in East Central (Figure 45). Analysis by age group (Figure 44) shows that overall the reduction of any anemia was mainly in older children and due to a reduction in moderate anemia while infants and a significant reduction in severe anemia but not so much in moderate anemia. FIGURE 44: SEVERITY OF ANEMIA BY AGE AND YEAR FIGURE 45: MODERATE/SEVERE ANEMIA 2009 AND 2014 BY MIS REGION AND IRS The decline of malaria incidence by 33-35% observed in Uganda is very similar as the decline of 37% reported in the most recent World Malaria Report [29]. Declines of parasite prevalence of 40% for Africa 122 have been estimated for the period 2000-20015 [30]. Country specific data Malawi showed a 30% incidence reduction that attributable to ITN [32] and from Zambia a 53% decline in parasite rates was reported [31], very similar to Uganda. Decreases in malaria mortality and morbidity from selected health facilities of 70-80% have been reported from Ethiopia [33] and Rwanda [33] but these are countries with much lower transmission potential than Uganda. Considering historical data from Uganda such as those from Kamwenge District (Figure 46, Albert Kilian unpublished) show that tremendous reduction in parasite prevalence have occurred in Uganda since the 1990’ies and which are not even captured in the MIS data. FIGURE 46: HISTORICAL TRENDS IN MALARIA PARASITEMIA FROM KAMWENGE DISTRICT 1994- 2008 In summary, data from the routine health services indicate that after correction for under-reporting and adjustments for changes over time in OPD attendance there was a clear downward trend between 2006 and 2014 in reported malaria incidence, malaria test positivity and reported anemia for children as well as for older persons, for males as well as for females. The reduction in malaria incidence was overall around 35% but three distinct phases of change could be identified: a modest decrease up to the targeted LLIN mass campaigns in 2010/11 of approximately 5%, a 10% reduction following the targeted campaign up to the UCC and a 25% reduction after the UCC. This pattern was seen in all regions except the North where the IRS had created significant morbidity reductions between 2010 and 2014. There also was some moderately strong evidence that the decline followed the implementation timeline of the UCC waves. Both malaria parasite prevalence and moderate/severe anemia in children 6-59 months reduced by about 50% between 2009 and 2014 with strong impact in the IRS districts in the North and an reduction of parasite prevalence below 1% in Kampala. The least reductions were observed in the East Central region where also the lowest ITN ownership levels had been seen. IV. Effects, Cost and Cost-effectiveness The Outcome of UCC regarding Net Ownership As the implementation of the UCC was done by waves and stretched over a considerable time period, it is important to keep in mind the geographical spread and time since distribution at the time of the MIS 123 2014/15. As shown in Figure 47, the UCC waves rolled essentially from the East to the central area, to the West and South-West and then to the North ending in the Kampala and Wakiso area. The time since distribution was longest for the pilot with 26 months, followed by waves 1 and 2 with 15-18 months. Waves 3-5 had a 10-12 months period between UCC and MIS, waves 6-8a around 5 months and wave 8b less than 4 months (details see Table 23). FIGURE 47: MAPS OF UCC WAVES, BRANDS USED AND DISTRIBUTION DENSITY Since there is a potential that the type of LLIN distributed might impact on user acceptability and use and/or the durability of the product, it was attempted to map out the various brands used in the UCC. Overall two polyester-based brands (Permanet 2.0 and 3.0, Dawa Plus 2.0) and two polyethylene-based LLIN brands (Olyset, Royal Sentry) were distributed. However the bulk of the nets were Permanet 2.0 (46%) and Olyset24 (32%) and only smaller contributions from Dawa Plus (17%), Royal Sentry (5%) and 24 Most likely the new version with improved knitting pattern to avoid “unraveling” of small holes. 124 Permanet 3 (1%). As far as could be established from distribution records and triangulated by the brand identification in the MIS (Table 23), the North and East received predominantly Olyset, central region Permanet, a mix of Permanet and Dawa in the South-West, West and West-Nile and a mix of Permanet, Olyset and Royal Sentry in the West/South-West in wave 6. TABLE 23: TIME BETWEEN MIS 2014 AND UCC AND MAIN BRANDS BY UCC WAVE Wave Avg. time since distribution (months) Brands (implementation records) MIS 2014 brand for campaign nets Polyester Polyethylene Permanet Dawa Olyset Royal Sentry Pilot 26.4 Permanet 42% 52% 1 18.6 Permanet (incl. PN 3.0) 78% 13% 2 15.0 Olyset (Permanet) 25% 72% 3 12.8 Olyset 71% 23% 4 11.6 Permanet, Olyset 91% 5 10.5 Permanet 36% 30% 13% 6 5.5 Dawa (Permanet) 32% 59% 7 6.0 Dawa, Permanet, Royal Sentry 92% 8a 5.5 Dawa, Permanet, Royal Sentry 63% 21% 8b 3.6 Dawa, Permanet, Olyset 36% 46% 11% Distribution density of the UCC (Figure 47) is primarily a function of population density and demonstrates that – apart from the Kampala and Wakiso area – the South-Eastern part of Uganda has not only the highest malaria transmission levels, but also the highest population density. Other population focus areas are the South-West, West-Nile and the southern part of Mid-North. The household ownership of any net at national level had increased from 34% in 2006 to 74% in 2011 and to 94% in 2014/15 (Figure 48). The proportion of ITN among all nets was only 45% in 2006 and 65% of the ITN were LLIN, 35% conventionally treated. This changed dramatically in favor of LLIN and in 2011 78% of all nets were ITN, and 98% of the ITN were LLIN meaning that by that time “dipping” of nets had essentially disappeared. In 2014/15 95% of all nets were ITN and 99.8% of ITN were LLIN showing that now untreated nets only play a very marginal role in Uganda. And untreated nets are mainly found in the central area where they comprise 8-13% of the net crop while in all other regions their contribution is between 0.5% (North East) to 3.7% (South-West). Nationally only 3.5% of households owned only untreated nets and outside the central region only 1.5%. While increase of household ownership of any ITN and population access to ITN increased in parallel over time (Figure 48), there was initially a slower growth of households with enough ITN for all members which is a reflection of the targeted distribution strategy during that time which favored geographical spread (at least one ITN) over intra-household net saturation (enough ITN). FIGURE 48: TREND IN ITN OWNERSHIP INDICATORS 2006 TO 2015 DASHED LINES REPRESENT COVERAGE WITHOUT THE UCC NETS 125 126 FIGURE 49: TREND OF POPULATION ACCESS TO ITN BY MIS REGIONS The trend over time by the MIS regions for the most important of the ITN ownership indicators, population access, is shown in Figure 49 and reflects the history of ITN distribution in Uganda: before the targeted campaign of 2010 the North, including Mid-East, had the highest coverage due in part to the focus on internally displaced people. Also initially high was Kampala where availability of nets from the retail market was high (subsidized and at cost). During the 2010/11 campaigns West-Nile and Mid-West and Central 125 made the biggest gains while Mid-East and North East stagnated or even dropped. It is worth noting that throughout the time period East Central had the lowest rates and this was also true in 2014/15 when only 72% had access to an ITN, 10%-points lower than the national average. Without the contribution of the UCC only 16% of households would have had any ITN in 2014/15 and only 11% of the population would have had access (Figure 48). Conversely, with only the UCC nets considered, i.e. ignoring any net from other sources which is the underlying assumption for the quantification of the UCC nets when calculating it as population/1.8 [6], 83% of households had at least one ITN, 54% enough ITN and 75% of the population had access to an ITN within the household. This implies that there were some households that had both, UCC nets and non-UCC nets. Nationally these were 8% with another 8% having only non-UCC nets leaving 74% that had UCC nets only. The distribution by MIS region is shown in Figure 50 with details for all ownership indicators presented in Table 24. By far the largest contribution of non-UCC nets was in Kampala where 45% of households owned any non-UCC nets and only 41% had exclusively UCC nets. This was followed by the North East (23%), Central 1 (21%) and East Central (20%). The lowest contribution of non-UCC nets was found in Mid-West (85) and Mid-East (9%). FIGURE 50: CONTRIBUTION OF UCC AND NON-UCC NETS TO ITN HOUSEHOLD COVERAGE BY MIS REGION 25 In four districts of Mid-West and Central 1 the campaign actually was UCC instead of targeted due to a project funded by the British INGO Comic Relief. These were Hoima, Kiboga, Kyankwanzi and Buliisa. 127 128 TABLE 24: HOUSEHOLD ITN OWNERSHIP AND POPULATION ACCESS TO ITN WITHIN HOUSEHOLD Background All ITN Only nets from UCC Any ITN 1 ITN for 2 people or better Population access to ITN Any UCC net 1 UCC net for 2 people or better Population access to UCC net National 90.2% 62.0% 82.1% 82.5% 53.9% 74.5% MIS-Regions Central 1 80.8% 57.7% 74.1% 67.0% 45.3% 60.8% Central 2 81.6% 54.2% 74.5% 74.7% 48.4% 68.0% East Central 82.1% 45.5% 71.8% 72.3% 38.4% 62.0% Kampala 86.3% 66.6% 80.6% 61.8% 36.7% 53.6% Mid-North 94.3% 65.7% 87.2% 90.2% 60.0% 83.6% Mid Western 93.6% 64.0% 85.0% 89.7% 60.2% 81.8% Mid Eastern 94.6% 61.9% 82.6% 88.7% 56.8% 77.5% North East 97.0% 59.1% 86.4% 85.2% 47.9% 73.5% South Western 96.9% 75.8% 90.7% 95.4% 71.6% 88.4% West Nile 96.3% 71.5% 86.7% 93.8% 64.7% 83.2% Months since distribution 3-5 90.5% 65.6% 83.8% 81.8% 54.7% 75.3% 6-11 91.5% 66.6% 84.2% 86.3% 61.0% 80.1% 12-17 89.4% 56.9% 80.2% 82.5% 50.7% 73.1% 18-23 98.9% 62.2% 90.2% 87.9% 49.8% 80.4% 24-28 79.0% 33.7% 60.9% 56.1% 19.4% 37.5% FIGURE 51: CONTRIBUTION OF UCC AND NON-UCC NETS TO ITN HOUSEHOLD COVERAGE BY UCC WAVE When the analysis was done by waves (Figure 51) the largest contribution was in Kampala/Wakiso (wave 8b) where by design distribution was limited to one or two LLIN per household. But it was also very high in the pilot districts with 39% of households having any non-UCC nets and only 40% owning UCC nets 129 only. On the other hand, the ownership of UCC nets for waves 1 and 2 was still very high with 89% even though this was 15-18 months after distribution, i.e. half-way towards the next scheduled distribution. Coverage at community level is critical for the development of a community or mass-effect of ITN that provides protection for those households and people without an ITN and results are shown in Table 25. At national level 94% of households resided in communities with at least 80% ITN coverage and still 71% in communities with 95% coverage. This is very high and much higher than what was found in Nigeria in the MIS 2010 for stated that had had a recent campaign where the rates were 66% and 31% respectively (A. Kilian, unpublished). This suggests that spatial coverage was very high after the UCC. Coverage for communities with enough ITN for all household members was also high, 77% of households lived in communities with at least 60% coverage and still 44% with 75%. The lowest community level coverage was seen in East Central where still 94% of communities had 80% of households covered with at least one ITN but only 35% at the 95% level and only 7% had a coverage level of 75% with enough ITN. This area was covered by the pilot and wave 3. On the hand, South-Western (waves 5 and 6) and West Nile (wave 8a) were particularly high. TABLE 25: COMMUNITY LEVEL COVERAGE WITH ANY ITN AND ENOUGH ITN (1ITN/2 PEOPLE) Region Proportion of households where community coverage is at least Any ITN 80% Any ITN 95% Enough ITN 60% Enough ITN 75% National 94.0% 71.3% 76.8% 44.2% Central 1 86.3% 23.6% 77.6% 30.2% Central 2 77.8% 50.2% 64.3% 6.1% East Central 93.9% 34.9% 40.9% 7.1% Kampala 90.5% 62.3% 88.0% 57.0% Mid-North 100% 83.1% 80.7% 54.8% Mid Western 94.9% 88.8% 85.9% 54.3% Mid Eastern 100% 87.2% 78.0% 47.0% North East 98.8% 96.8% 68.0% 33.8% South Western 100% 100% 94.6% 85.3% West Nile 100% 93.7% 93.7% 71.6% Some indication of a possible oversupply of LLIN through UCC had already arisen in the analysis of UCC data quality with a much higher population registered than expected and a low population to nets distributed ratio in some areas (at least in the UCC implementation records). Further evidence of potential over and under-supply was seen in the ITN ownership analysis above. To explore this aspect further, the comparison with expected coverage as a function of the time since distribution from the NetCALC model is useful and results are shown in Figure 52 for the indicators “households with any ITN” (left panel) and “population access to ITN within the household” (right panel). FIGURE 52: ITN OWNERSHIP COVERAGE IN COMPARISON TO ESTIMATES FROM NETCALC ERROR BARS SHOW 95% CONFIDENCE INTERVALS 130 Even if one considers a median survival of the LLIN of 3.5 years and UCC nets only, there is strong evidence of an oversupply for waves 1, 5 and 8a. Waves 2, 6 and 7 are on the high end but still plausible and waves 3 and 4 are below the expected although the 95% confidence interval does include the expected coverage under a 2.5 year median LLIN survival assumption. Wave 8b was far below universal coverage but as, previously mentioned, this was by design. The pilot was below or at the lower limit of the expected which fits with the previously presented data and also with the population per net distributed in the UCC records (2.35) with only one of the four districts reaching a value close to the expected 1.8 (Kaliro 1.87) while the other three are between 2.5 and 2.6. It is not clear whether this was by design as the Phase I report does not mention a targeted distribution in the pilot or any other difficulties, but clearly the pilot districts were under-supplied. When household supply with ITN or UCC nets is categorized into five groups to consider under-, sufficient and oversupply, it can be seen that in addition to the 62% of households with at least 1 ITN for every 2 people another 19% (81% in total) had almost enough ITN for all members and 21% actually had more than they would need based on the allocation algorithm. For UCC nets only the figures were lower with 72% having almost enough and 16% “too many”. The most important determinant of the intra-household supply with ITN was household size as shown in Figure 53 and here two opposite trends can be seen. On the one hand the likelihood of having no ITN at all was highest among small households and decreased with increasing household size. This could be a reflection of the probability of being registered as for large households it is more likely that somebody was present at the day of registration than for a household of three or less persons. On the other hand small households had the highest rate of oversupply (26%) while large households were almost never oversupplied and only 33% had enough nets for all members. In a logistic regression model region was the only other variable associated with a household having no ITN at all (likelihood in the Central region higher than North and West) while oversupply was – in addition to household size – positively associated with being in the highest wealth quintiles and negatively with having children under five in the household. FIGURE 53: INTRA-HOUSEHOLD SUPPLY SITUATION WITH ITN AND UCC NETS BY HOUSEHOLD SIZE 131 The Phase I report had already stated that some informants reported that for large households the number of nets given was sometimes limited to a maximum of four. This was also confirmed in the FGDs undertaken for this evaluation where many of the large households in all four visited districts reported that they received less that the 1 ITN for every 2 people. The FGDs also confirmed that some households were missed entirely because people were absent at the time. The FGDs also revealed that in all visited districts many people had misunderstood the concept of universal coverage and had expected each person to receive their own ITN: The intra-household supply by region and UCC wave is presented in Figures 54 and 55 and confirms the picture already seen in the previous analysis that East Central region and the pilot and wave 8b were the most under-supplied. 132 FIGURE 54: INTRA-HOUSEHOLD SUPPLY WITH ITN BY MIS REGION FIGURE 55: INTRA-HOUSEHOLD SUPPLY WITH ITN BY UCC WAVE If there is oversupply with nets one would expect that households have unused nets which they have not hung and which they use for visitors or keep for a later time when the current nets are torn. In the MIS 2014/15 the reasons for non-use were explored for those nets not used the previous night. These were overall 26%, i.e. 74% of the ITN were used the previous night. But in households with too few ITN only 16% were not used compared to 26% if there were enough ITN in the households and 41% not used in households with too many nets. Among the unused ITN by far the most common stated reason was that the net was not hung (72%) followed by “it is an extra net” (10%), subjective reasons such as heat or no perceived mosquitoes (7%) and the net being too old (4%). Comparing the proportion of nets not hung between households without and with enough ITN for all members (9% vs. 20%) suggests that about half of the nets not hung were unused because they were more than needed. At household level 5% owned at 133 least one “extra” net but this was negligible among households that did not have enough ITN (< 1%) and reached 12% among households that had more than enough. Similarly, 31% of households owned any ITN that were not hung the night before the survey: 15% if there were not enough ITN, 34% if enough and 53% if there were more than enough. Analysis by region and wave confirmed the previous findings showing that especially in South-West (wave 5) and wave 1 there was evidence of oversupply considering the time since distribution. The results from the UCC as documented in the MIS 2014/15 compare favorably with results from other countries that have undertaken UCC. The 2011 MIS in Tanzania, undertaken just after the “top-up” campaign showed 91% of households had any ITN, 55% had at least 1 for every 2 people and 80% of the population had access to an UTN within the household. A similarly high access rate as the 82% found in Uganda were also documented in the 2013 MIS in Rwanda (72%) and the 2011 MIS in Madagascar (72%) and in a post-campaign survey in Eastern Ghana (77%). Other post-campaign surveys only reported the coverage with any ITN and these were 88% in Sierra Leone [14] and 97% in Togo [15]. Contribution of Other Sources We had previously established that 16% of households had nets not from the UCC and the question is where did these come from. Of all the nets recorded in the MIS 2014/15 84% were from the UCC, 3% from a previous campaign, 2% from a public or NGO health facility and 9% from the retail market (shops and market places). Looking at sources by region (Figure 56) shows that the highest contribution from private sector nets was in Kampala (42%) and Central 1 (19%) (including Wakiso) which is not surprising as here the UCC did not supply sufficient nets and LLIN are available for purchase. But nets from the private sector were also found to some extent in all other regions, especially the North East (8%) and East Central (9%) which includes the pilot districts. When analyzed by wave (see Figure 57) the pilot area had a 21% contribution of private sector nets suggesting that households filled the existing gaps by buying additional nets where possible. Interestingly, the majority of nets from the private sector where also LLIN (63%) and only in Central 1 and 2 as well as South-West were there more untreated nets than LLIN among the nets bought from the markets. At household level 14% owned at least one net and 9% at least one LLIN from the private sector. As expected this rate was very high in Kampala (55% and 41% respectively) but also in the North- East (14% and 13%) and East Central (14% and 12%). Among households for which the UCC was more than one year ago, 38% owned a net that was less than one year old providing further evidence that some gaps from the UCC were filled by other means. 134 FIGURE 56: SOURCES OF NETS FROM MIS 2014/15 BY MIS REGION Only 4% of households owned an ITN that was obtained from a health facility and most likely obtained through ante-natal care services. This rate was highest in the North East (10%) and East Central (5%) and lowest in the South-West (2%). There was some evidence that these nets were targeted to pregnant women as the proportion of households with an ITN from a health facility was twice as high among households that had either a pregnant woman or a child less than two years old (5% vs. 2.5%). But overall this distribution channel was far below its potential of reaching up to 30% of households which shows that to date delivery of ITN through routine services in not yet working in Uganda. 135 FIGURE 57: SOURCES OF NETS BY UCC WAVE Issues of Equity Measures of equity were taken for all net ownership indicators and compared across the most recent four representative household surveys. Equity was also compared between nets obtained by the public or private sectors with the exception of the DHS 2011 data for which this distinction was not possible. Results are shown in Table 26 and for the population access to ITN indicator in Figure 58 and clearly demonstrate a clear trend towards perfect equity over time. As the ownership of nets and ITN increases, the initial slightly pro-rich distribution is increasingly shifted towards equity for both, the equity ratio and the concentration index (Figure 59). The development is somewhat slower for the indicator of “enough” ITN which stagnates between 2009 and 2011, but reaches almost perfect equity following the UCC in 2014. Overall, ownership of any net obtained from the public sector (UCC, previous campaigns or through health facility) showed a slightly pro-poor tendency in 2014 with a equity ratio above 1.0 and a concentration index below zero, but statistically the latter was not different from zero. This was due to a slightly lower ownership of public nets in the highest wealth quintile while are other quintiles had about the same ownership rate (Figure 60). In contrast, ownership of nets purchased from shops or the market showed a pro-rich gradient which did not change dramatically over time even though the level of private sector nets had decreased from 2011 to 2014 which is most likely a “crowding out” effect of the UCC on the retail market. 136 TABLE 26: EQUITY RATIO AND CONCENTRATION INDEX FOR ITN OWNERSHIP 2006 TO 2014 Indicator Equity measure DHS 2006 MIS 2009 DHS 2011 MIS 2014 Any ITN Equity Ratio 0.28 0.57 0.77 1.00 Concentration Index 0.259 0.097 0.038 -0.002 Enough ITN Equity Ratio 0.13 0.47 0.46 0.97 Concentration Index 0.404 0.144 0.162 0.0001 Access to ITN Equity Ratio 0.28 0.57 0.72 0.99 Concentration Index 0.288 0.102 0.068 0.003 Any public net Equity Ratio 0.38 0.73 n.a. 1.10 Concentration Index 0.188 0.054 n.a. -0.018 Any private net Equity Ratio 0.30 0.28 n.a. 0.18 Concentration Index 0.244 0.252 n.a. 0.335 FIGURE 58: TREND IN EQUITY OF POPULATION ACCESS TO ITN 137 FIGURE 59 TREND IN CONCENTRATION INDEX OF ITN OWNERSHIP VRIABLES 2006-14 FIGURE 60: OWNERSHIP OF PUBLIC AND PRIVATE SECTOR NETS BY WEALTH QUINTILE Outlook for Future Distributions The second NetCALC model intended to project ownership of any ITN and population access to an ITN within the household based on past net distributions and purchases including the UCC to address two critical questions: How does the projection of ownership indicators under various assumptions of median net survival and underlying population compare with the results from the household surveys and can this additional data point help to assess the most likely “true” population size at the time of the UCC? What is the likely ownership coverage at the time of the next UCC currently planned for 2016/17, what net crop (number of ITN) does this coverage translate into and how does this compare to estimates from the MIS 2014/15? The population size in the model was varied between the census estimates for 2014 of 34.8 million, the result of the UCC registration records of 40.0 million and the estimate of 36.9 million based on a 5% 138 under-counting in the census. The first observation is that when the 22.3 million LLIN distributed in the UCC are entered on top of the previous distributions, the model indicates an excess of nets in the system that varies between 0.6 and 7.3 million with a median of 4.3 million nets (Table 27). As was previously shown (Figure 48) the ITN coverage with any ITN was 60% in 2011 with 47% of people having access to an ITN. Coverage would be expected to decline further until the UCC slowed down a bit by additional uptake of nets from ANC distributions and market sales. At the time of the UCC these nets were still in the households and some may have been discarded early, others may have been kept for later use. In the NetCALC model all nets that mathematically exceed a 100% universal coverage are marked as “excess” at the time but will be redistributed in later years wherever a gap in coverage appears. Since this does not really happen in reality in 100% of the excess nets, all excess nets in the model were removed in order to allow a conservative estimate of future coverage and net crop. TABLE 27: EXCESS NETS ESTIMATED FOLLOWING UCC Median net survival Population estimate for 2014 34.8 million 36.9 million 41.0 million 3.5-years 7.3 m 5.7m 3.5 m 3.0-years 5.8 m 4.3 m 2.1 m 2.5-years 4.3 m 2.9 m 0.6 m Based on the three population scenarios and considering the two indicators “Household has any ITN2 and “population access to ITN within the household” six projections were made each with the expected coverage curves for median net survival varied between 2.5 and 3.5 years and these curves were compared to the respective survey estimates and their 95% confidence intervals. The projections are shown in Figure 61. For all estimates it was observed that “any ITN” coverage for 2009 and 2011 was underestimated in NetCALC while the “access” indicator showed a much closer fit. This is caused by the previously described effect (Figure 44) that during the phase of primarily targeted distributions before the UCC there was a disconnect between increase in geographical coverage of “at least one ITN” and intra￾household saturation with ITN (enough ITN) which cannot be represented in NetCALC as the model always assumes a balanced and simultaneous increase of these two indicators. Considering not only the coverage results from the four DHS and MIS but also the estimate of coverage in 2014/15 without the UCC nets and assuming further that the most realistic median survival in Uganda is between 2.5 and 3.0 years at the moment [13], the best fit between model and data is achieved based on the census population in 2014 of 34.8 million while a population estimate of 36.9 million still appears plausible. However, the results suggest that a assuming a population of 40.0 million at the time of the UCC would only be possible if median survival of nets in the past 10 years was consistently at 3.5 years or above. 139 FIGURE 61: PROJECTED POPULATION ACCESS 2006-2018 COMPARED TO SURVEY RESULTS RESULTS FROM NETCALC WITH AND WITHOUT UCC AND ASSUMING DIFFERENT POPULATION ESTIMATES The expected ITN coverage with any ITN from the models varies between 42% and 55% for 2016 and 21% and 37% for 2017. This translates to an ITN crop of between 10 and 13 million at the end of 2016 and between 6 and 8 million in 2017 (Table 28). In order to see how this compares to estimate of numbers of ITN in the population from the MIS 2014/15 the mean number of ITN owned per household was multiplied with the total number of households assuming 4.9 persons per household as reported by the 2014 census and the NIS 2014/15. As shown in the table the figures for the 2014 ITN crop compare quite 140 well with the estimates from the survey with the best fit again for a population of 34.8 million in 2014. This suggests that the models have a reasonably good fit and that the likely median survival of nets is probably just below 3.0 years on average. TABLE 28: EXPECTED NET CROP BY YEAR ASSUMING A 3 YEAR SURVIVAL AND ESTIMATES FROM MIS 2014/14 2014 2015 2016 2017 NetCALC Population 34.8m 18.4m 14.7m 10.4m 6.4m Population 36.9m 19.6m 15.8m 11.4m 7.1m Population 41.0m 21.8m 17.8m 12.9m 8.1m MIS 2014/15 Population 34.8m 17.7m Population 36.9m 18.7m Population 41.0m 20.8m Given that the concept of the repeated mass campaigns is to distribute the full amount of the need for universal coverage calculated as population divided by 1.8, i.e. assuming each time that there is no viable net left in the population, this approach by design will replace some nets that are still in usable condition and may provide one or two years of useful life more. It has previously been estimated from modelling that if there still is a coverage of 20% or more of nets that are at two years old or less, a campaign is not cost-effective [16]. Alternatively, one can estimate the need for ITN in a hypothetical system of comprehensive continuous distribution (CD) through a mix of channels as suggested by WHO as a medium-term objective [17]. This is presented in Table 29 based on NetCALC projections for two campaign cycles assuming that in 2014 full universal coverage is achieved and maintained at 100% and that median survival is 3.0 years. Because in the first years after a UCC all nets are new and initial loss slow, the replacement need in the first three-year cycle from NetCALC is lower than the steady state which is reached in the second cycle. Nonetheless, the model shows that the need for ITN would be 10% lower in the first 6 years under CD compared to UCC which would translate to about 9 million ITN under the assumption of a population of around 35 million in 2014. TABLE 29: PROJECTED NEED FOR ITN COMPARING UCC AND CONTINUOUS DISTRIBUTION Population size Distribution 2015-17 2018-20 Total 34.8m CD 13.1m 21.4m 34.5m (- 9.2m) UCC 20.9m 22.8m 43.7m 36.9m CD 14.0m 22.7m 36.7m (-9.8m) UCC 22.2m 24.3m 46.5m 41.0m CD 15.5m 25.2m 40.7m (-10.8m) UCC 24.6m 26.9m 51.5m Interestingly, several respondents in the qualitative KII and FGD (e.g. the deputy CAO in Hoima and communities in Soroti) mentioned that they thought that a “continuous” distribution would be preferable, particularly to replace nets when needed and to ensure that all households are reached as in such a system the likelihood that a household might be missed, would be much lower. In summary, the analysis of ITN ownership from survey data found that the UCC in Uganda had achieved full universal coverage with population access to ITN of over 80% which is as high as any other country has ever reported following a mass campaign. The resulting ITN ownership was highly equitable are achieved a high community-level coverage that should ensure optimal effect on malaria transmission in the absence of insecticide resistance. However, there was evidence that the UCC left some gaps which 141 geographically were around Kampala (wave 8b) and in the North East and East Central (pilot). In addition, small households were shown to have been more likely to be missed while large households were systematically under-supplied. There also was some evidence of oversupply in some waves (1, 5 8a) and generally among those smaller households that were reached. The data suggests that households tried to fill gaps left by the UCC by obtaining nets from the markets including LLIN while routine distribution of nets through health facilities was very low and far below its potential. Modeling supports the previous finding that the “true” population in 2014 is more likely to be closer to the census results or slightly above but below the numbers obtained in the UCC registration process. Modelling also suggests that by the time of the next planned UCC there will be a crop of ITN of around at least 6-10 million and that a hypothetical continuous distribution instead of UCC could save around 9 million ITN over a six year period. Cost and Cost-effectiveness Economic and Financial Cost Base case cost analysis resulted in an estimate of approximately 95 Million USD in financial costs and 100 Million USD in economic costs for the distribution of ~22 Million LLINs across all waves of the UCC campaign in Uganda. Table 30 shows the total costs of the program by line item category for both the economic and financial analysis as well as the relative contribution of each line item. Not surprisingly, the net procurement was the largest cost driver with 83% of the financial and 91% of the economic cost. But also within the delivery cost alone the line item of procurement and supply management was the largest contributor comprising30% of the delivery cost. So in total, 95% of the economic cost of USD 100 million was linked to the procurements of the nets. 142 TABLE 30: ECONOMIC AND FINANCIAL COST OF UCC Line Item Financial Economic Total cost USD % of total % of delivery Total cost USD % of total % of delivery Human Resource 2,054,187 2.1% 15.4% 2,054,187 2.1% 15.4% Training 1,004,292 1.0% 7.5% 1,004,292 1.0% 7.5% Procurement and supply management 4,053,283 4.1% 30.4% 4,053,283 4.1% 30.4% Communication material 1,177,482 1.2% 8.8% 1,177,482 1.2% 8.8% Technical assistance 69,532 0.1% 0.5% 69,532 0.1% 0.5% Monitoring and evaluation 2,264,340 2.3% 17.0% 2,264,340 2.3% 17.0% Planning and Administration 2,447,230 2.5% 18.3% 2,447,230 2.4% 18.3% Overheads 283,747 0.3% 2.1% 283,747 0.3% 2.1% Nets 81,678,867 82.8% 90,499,036 90.5% Grand Total 98,682,965 100.0% 99,981,976 100.0% Distribution Only 13,354,097 13.5% 100.0% 13,354,097 13.4% 100.0% Human Resource 2,054,187 2.1% 15.4% 2,054,187 2.1% 15.4% Table 31 shows the outputs and base case scenario estimated outcomes. TABLE 31: TOTAL OUTCOMES AND OUTPUT ACHIEVED Output or Outcome Total achieved Nets Distributed 22,377,772 Net Years 67,133,316 Children Years Protected 67,133,316 Child Deaths Averted 369,233 DALYs Averted 12,184,697 Table 32 shows the total cost per outcome in base-case analysis for both financial and economic analyses as well as for distribution only, i.e. excluding LLIN commodities. TABLE 32: ECONOMIC AND FINANCIAL COST PER OUTCOME Output or Outcome Cost per outcome in USD Financial Full Cost Economic Full Cost Financial Cost Distribution Economic Cost Distribution Nets Distributed 4.25 4.49 0.60 0.60 Net Year 1.42 1.49 0.20 0.20 Person Years Protected 0.71 0.74 0.10 0.10 Children Years Protected 1.41 1.49 0.20 0.20 Child Deaths Averted 257.38 270.78 36.17 36.17 DALYs Averted 7.80 8.21 1.10 1.10 Sensitivity Analysis In order to test the basic assumptions of the cost and effectiveness models presented above to changes a one way, scenario analysis and probabilistic sensitivity analysis. The main assumptions investigated were surrounding the numbers of LLINs distributed and their cost, as well as the translation of numbers of nets distributed to the number of deaths and DALYs averted. Under standard NetCALC assumptions and using census data for population estimates as opposed to 2014 registration data the standard NetCALC 143 formulation estimates that approximately 5 Million excess nets were distributed during the 2014 UCC. We estimated used the NetCALC implementation to determine the excess number of Net Years of Protection provided by distributing those 5 million nets and subsequently the potential cost savings and cost effectiveness when these 5 million fewer nets were distributed. The results of alternative simulations with five million less nets indicates that the coverage under the population scenario using census data would still be expected to have achieved greater than 80% coverage in the majority of simulations and achieve nearly 100% coverage in at least 10% of simulations. Figure 62 shows the summary of these simulation results. These results indicate that significantly fewer LLINs may have been able to have been distributed without significant losses of coverage. Consequently we also examined the cost implications of distributing 5 million fewer LLINs in the UCC. Distributing 5 million fewer LLINs during the UCC would result in a total financial cost estimate of approximately 77 Million USD as opposed to 95 Million in the base scenario, resulting in a cost savings of approximately 18 Million USD. Cost per outcome would increase but only to ~4.64 USD per Net Distributed (economic Analysis). Estimates of cost per other outputs or outcomes would similarly increase only slightly. We also varied the discount rate applied in the analysis from 10% to 0%. This had little effect on estimates of the Cost per outcome or output, partially because the duration of the program was very short. Reducing the cost per Net purchased from ~3.65 USD to ~ 3 USD reduced the cost per outcome and output universally and resulted in cost per net distributed of under 3.78 USD in both economic and financial analysis. In all cases reducing the assumptions around the usage and durability of LLINs used in the program significantly increased the cost per outcome in terms of child years of protection, treated net years and deaths and DALYs averted. However, even in the case where worst case scenario approaches were taken (e.g. LLINs were assumed to last only one year, and be relatively infrequently used, estimates still indicated that cost per DALY averted and death averted fell into a range which would be considered highly cost effective by WHO standards as applied to Uganda (Cost per DALY averted ≤ GDP per Capita). 144 FIGURE 62: NETCALC SIMULATION RESULTS (SMALLER UCC CAMPAIGN IN A) AND NO UCC IN B) The economic cost per LLIN delivered of USD 4.49 or average annual economic cost of USD 1.49 compares very favorably with reports from other countries. Early reports (2008) from six countries reported annual cost between USD 2.75 and 4.80 [25]. More recent data from Ghana [26] found annual economic cost of USD 2.90 and in Uganda cost of between USD 3.55 and 2.88 were found for previous and local campaigns [27]. The only report with a similar total economic cost is from Togo [28] with USD 4.41 per LLIN delivered, but this was calculated by applied shared cost in an integrated campaign. There is no doubt that economic cost in recent years have come down significantly due to lower prices of the LLIN which were only USD 3.65 (c.i.f.) on average for the UCC which is USD 1.31 less than paid in Ghana in 2012 [26]. But one also must keep in mind that possibly some cost were not included in this evaluation due to the retrospective approach to collecting data as the relative cost of net procurement is somewhat higher (91%) than was found in Ghana (60%) or Togo (61%). In summary, cost and cost-effectiveness analysis indicates that the UCC campaign in Uganda was an efficient use of health resources considering the health benefits like to have been achieved. The overall cost of the campaign was large, ~100 Million USD. In light of the large absolute investment necessary, areas of efficiency should be identified which could result in significant cost savings. In Uganda, the cost of 5 million potentially excess nets identified by comparison of 2014 census data with UCC registration data through NetCALC, were conservatively estimated to have added ~18 Million USD to the cost of the campaign. While the cost of the planning and registration for the campaign was estimated to be approximately 2.4 Million USD. It appears possible, therefore, that for relatively minor investments in 145 improved registration it might be possible to achieve significant cost savings through better alignment of net procurement and distribution with actual need. 146 Annex VIII: Sample of Radio Script NET USAGE TALK SHOW GUIDE SCRIPT CLIENT: Stop Malaria Project DATE: 23rd/05/2014 CAMPAIGN: Net Usage Presenter: What brings you to the station? Guest: • To tell people about Malaria prevention by sleeping under insecticide-treated mosquito nets every night Presenter: The Government of Uganda and its partners gave long lasting, insecticide-treated mosquito nets to people. Can you explain to our listeners, why? Guest: • To prevent Malaria • Every day about 360 people die from Malaria in Uganda. On average, 15 Ugandans die from Malaria per hour • Malaria is the leading cause of illness and death in Uganda. In fact, Uganda is considered one of the countries in the world most affected by malaria • That is why the Ministry of Health in partnership with the government of Uganda gave mosquito nets to people to make sure everyone in every family sleeps under an insecticide-treated mosquito net and is protected from Malaria • Presenter: Why should everyone sleep under a mosquito net? Guest: • Studies have shown that sleeping under a long lasting insecticide-treated mosquito net (LLIN) is one of the most effective ways to prevent malaria, because the mosquitoes that transmit Malaria bite at night. • The protection that a mosquito net provides against Malaria is also doubled when the net is treated with insecticides 147 • Mosquito nets have been proven across different countries to prevent sickness and deaths from Malaria, particularly among those biologically most vulnerable: children and pregnant women • The nets can cut down the number of Malaria attacks by more than a half • Sleeping under a net also enables a restful night sleep. It ensure less disturbance from mosquitoes • Sleeping under a mosquito net saves life and money • It’s very cheap to prevent Malaria than to treat it • Sleeping under an insecticide-treated mosquito net every night costs you nothing. But a dose of Malaria treatment costs over 20,000/=. If you and your family members don’t sleep under mosquitoes, you’ll find yourself spending a lot on treating Malaria. That’s not all. When you’re sick, you can’t work. When your wife suffers from Malaria, she can’t look after the children. When your children suffer from Malaria, they miss school. Malaria can even kill you or your family members. That’s why you should sleep under an insecticide-treated mosquito every night. Presenter: What are long lasting, insecticide-treated mosquito nets (LLINs)? Guest: • Long lasting, insecticide-treated nets are mosquito nets which are treated with safe insecticides that don’t easily wash off the net. Presenter: How safe are the long lasting, insecticide-treated mosquito nets (LLINs)? Guest: • There are several brands of long lasting, insecticide-treated mosquito nets (LLINs) recommended by the World Health Organization • These nets are proven to be safe for everyone, including pregnant women and children Presenter: There have been cases of people misusing mosquito nets. What do you have to say about this? Guest: Some people in our communities misuse mosquito nets in the following ways: • They misuse them as fishing nets • They misuse them as wedding gowns • People misuse mosquito nets as curtains on windows and doors for their houses • Others misuse mosquito nets as sieves to make passion juice 148 • Some people misuse mosquito nets to cover utensils, tables and chairs in their homes • There are some people who misuse mosquito nets to make veils • There are also reports of people who are misusing mosquito nets to make cages for rearing chicken • Some people are misusing mosquito nets to cover nursery beds • There are people who are misusing people as curtains on their bathrooms and latrines • Others misuse mosquito nets as goal nets People misuse mosquito nets in many ways. These mosquito nets are meant to protect you and your family members from Malaria. Don’t misuse them. Sleep under the mosquito nets every night to prevent Malaria. Presenter: What are the correct/proper ways of using a mosquito net? Guest: • When you get a mosquito net, hang it in air to balance the concentration of the insecticide • Hang the mosquito net over your bed or the bed of your family member • At night, properly tuck the mosquito net under the mattress. Make sure there is no any open space that can let the mosquitoes inside the net. • Do not hang items, like clothes and bed sheets, on top of the mosquito net. The items weaken the net, making it to sag and tear • Wash your long lasting, insecticide-treated mosquito net when it gets dirty using mild soap and water • After washing your net, lay it flat in a shaded area to dry. Do not put it in direct sunlight, or hang it off a line • Sew your net when it gets holes, just the way you would for your clothes • Avoid getting the net close to open fires such as candles. The net can catch fire Presenter: How do the long lasting, insecticide-treated mosquito nets (LLINs) prevent Malaria? Guest: • An insecticide-treated mosquito net acts as a physical barrier to mosquitoes. The insecticide repels and kills the mosquitoes Presenter: Let’s have a few questions from our listeners. 149 Presenter: Do you have any concluding remarks? Guest: • Everyone—young and old—should sleep under an insecticide-treated mosquito net every night. It’s the most effective and safe way to prevent Malaria • Sleeping under a mosquito every night save life and money • Don’t misuse the mosquito nets. Don’t use them as fishing nets, wedding gowns, curtains, sieves, table clothes and veils. The mosquito nets were given to you to prevent Malaria. Don’t use them to cover utensils, make passion juice or make cages for rearing chicken. • The mosquito nets are safe. Sleep under them every night to protect yourselves from Malaria. This is an initiative supported by the Government of Uganda and partners. Prevent Malaria. Sleep under an insecticide-treated mosquito net every night. U.S. Agency for International Development U.S. Mission Compound-South Wing, Plot 1577 Ggaba Road PO Box 7856 Kampala Uganda