i September 2015 Mass Distribution of Long Lasting Insecticidal Nets for Universal Coverage in Uganda Evaluation Report i Contents Acknowledgments .................................................................................................................................................................... iv Acronyms .............................................................................................................................................................................. v Executive Summary ............................................................................................................................................................. vii Recommendations ............................................................................................................................................................... ix 1.0. Program Background .......................................................................................................................................................... 1 2.0. Evaluation Purpose ............................................................................................................................................................. 2 3.0. Evaluation Methods and Limitations .................................................................................................................................. 3 3.1. Methods ......................................................................................................................................................................... 3 3.2. Limitations ..................................................................................................................................................................... 3 3.3. Evaluation Team............................................................................................................................................................. 4 4.0. Relevance and Implementation of the Universal Campaign ............................................................................................... 4 4.1. Relevance of the Universal Campaign in the Ugandan Context. .................................................................................... 4 4.2. Design of the universal campaign .................................................................................................................................. 6 4.2.1. Campaign Coordination Structures ............................................................................................................................. 7 4.2.2. Procurement of LLINs .................................................................................................................................................. 9 4.2.3. Quality Assurance of LLINs ........................................................................................................................................10 4.2.4. Training and Supervision ...........................................................................................................................................10 4.2.5. Social Mobilization and Behavioral Change Communication ....................................................................................13 4.2.6. Household Registration and LLIN Allocation .............................................................................................................15 4.2.7. Storage and Distribution to sub-counties .................................................................................................................16 4.2.8. Distribution of LLIN to Household .............................................................................................................................17 4.2.9. Information for decision-making...............................................................................................................................18 5.0. Effectiveness of the Universal Campaign in Attaining its Objective of National Coverage ...............................................21 6.0. Efficiency of the Universal Campaign. ...............................................................................................................................24 6.1. Financing of the LLIN campaign. ..................................................................................................................................24 6.2. Timelines for campaign implementation .....................................................................................................................26 6.3. Efficiency in the process adopted in implementing the UC .........................................................................................28 7.0. Universal Campaign Impacts .............................................................................................................................................29 7.1. LLIN Ownership and Use ..............................................................................................................................................29 7.2. Unintended Consequences of the Universal Campaign ...............................................................................................31 8.0. Sustainability of the Universal Campaign in Uganda. .......................................................................................................31 8.1. Sustainability of Impact ................................................................................................................................................31 8.2. Managerial Sustainability .............................................................................................................................................32 8.3. Financial Sustainability .................................................................................................................................................32 9.0. Conclusions and Recommendations. ................................................................................................................................33 9.1. Conclusions ..................................................................................................................................................................33 9.2. Recommendations .......................................................................................................................................................34 ii Appendix 1. Evaluation Scope of Work ....................................................................................................................................36 Appendix 2. Evaluation questionnaires ...................................................................................................................................47 Annex 3. Coordinating Structures. ...........................................................................................................................................51 Appendix 4. Procurement process ...........................................................................................................................................53 Appendix 5. Importation, storage and distribution of LLINs to sub-counties .......................................................................... 55 Appendix 6. Training and supervision process .........................................................................................................................58 Appendix 7. Registration of households process .....................................................................................................................60 Appendix 8. LLIN distribution to households process (Waves 2-8a) ........................................................................................62 Appendix 9. Social mobilization process ..................................................................................................................................68 Appendix 10. Information for decision making process ..........................................................................................................71 Appendix 11. Universal campaign chronology quantification ................................................................................................. 75 Appendix 12: Financial Mechanism for the UC ........................................................................................................................77 Appendix 13. List of Interviewees ............................................................................................................................................78 iii List of Tables Table 1Training Plans for Stakeholders at Various Levels ....................................................................11 Table 2. Records of LLIN delivery, from NCC logistics subcommittee. .................................................17 Table 3. Donor financial contribution to the LLIN campaign. ...............................................................25 Table 4. Timelines for campaign implementation ................................................................................28 Table 5. Forecast Need for LLINs (and Funding) to Support UC in 2017* ............................................33 Table 6. LLIN warehousing and distribution based on delivery notes ..................................................56 Table 7. Number of personnel trained. ................................................................................................59 Table 8. Distribution of LLINs and coverage statistics ..........................................................................63 Table 9. Budget for distribution of nets to households ........................................................................67 List of Figures Figure 1: UC Implementation Coordination Structure ............................................................................ 7 Figure 2. Personnel trainings & sensitization, target vs. actual, waves 4-8 in 74 districts ...................12 Figure 3. Flow of data from collection by the VHTs to processing sites at the district and the MoH ..20 Figure 4. Summary of LLIN Distribution by Wave .................................................................................22 Figure 5. Population coverage by wave ................................................................................................23 Figure 6. UC project milestones ............................................................................................................27 Figure 7. Reported use of ITNs in four pilot UC districts. .....................................................................30 iv Acknowledgments The authors would like to thank the Ministry of Health (MoH), members of district universal campaign taskforces, and all those who contributed to this evaluation. We appreciate the support received from the President’s Malaria Initiative’s (PMI), Department for International Development (DFID), Stop Malaria Project (SMP), and the Malaria Consortium (MC). While there was diversity of views the evaluation team ultimate consensus on success of the universal coverage campaign. The findings and recommendations reported in the report are not views of the authors and should not be attributed to the USAID Monitoring Evaluation and Learning program, the President’s Malaria Initiative or Department for International Development. v Acronyms ACT Artemisinin-based Combination Therapy AMP Alliance for Malaria Prevention ANC Antenatal Care ASM Advocacy and Social Mobilization ASMSC Advocacy and Social Mobilization Sub-Committee BCC Behavioral Change Communication CDC Centers for Disease Control and Prevention CHAI Clinton Health Access Initiative CDO Community Development Officer CSO Civil Society Organization DFID Department for International Development DGHS Director General of Health Services DHIS 2 District Health Information System 2 DHO District Health Officer DHT District health team DISO District Internal Security Officer DHS Demographic and Health Survey DTF District Task force EPI Expanded Program for Immunization FGD Focus group discussion GF Global Fund GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GOU Government of Uganda HBMF Home-Based Management of Fever HMIS Health Management Information System IPC Interpersonal Communication IPTp Intermittent Preventive Treatment in pregnancy IRS Indoor Residual Spraying ISO Internal Security Organization ITN Insecticide Treated Net JHUCCP Johns Hopkins University Center for Communication Programs JSI John Snow Incorporated K+N Kunhe + Nagel KII Key informant interview LC Local council LLIN Long Lasting Insecticidal Nets LSC Logistics Sub-Committee M&E Monitoring and Evaluation MC Malaria Consortium MFP Malaria focal person MHSDMU Medicines and Health Services Delivery Monitoring Unit MIS Malaria Indicator Survey MOH Ministry of Health MSP Malaria Strategic Plan MOU Memorandum of Understanding NCC National Coordinating Committee for the UC NDA National Drug Authority NGO Non-Governmental Organization vi NMCP National Malaria Control Program OECD Organization for Economic Cooperation and Development OSC Operations Sub-Committee PFP Private for Profit PMI President’s Malaria Initiative PNFP Private Not-for-profit PPS Public procurement service PSM Procurement and Supply Management RDC Resident District Commissioner RBM Roll Back Malaria RDTs Rapid Diagnostic Tests SMP Stop Malaria Program UBOS Uganda Bureau of Statistics UC Universal Campaign UMIS Uganda Malaria Indicator Survey UMRSP Uganda Malaria Reduction Strategic Plan UNBS Uganda National Bureau of Standards UNICEF United Nations Children’s Fund UPDF Uganda People’s Defense Force USD United States Dollars USAID United States Agency for International Development VHT Village Health Team VPP Voluntary Pooled Procurement WHO World Health Organization WHOPES WHO Pesticide Evaluation Scheme vii Executive Summary In May 2013, the Uganda National Malaria Control Program (NMCP) and partners embarked on an ambitious campaign to provide every single household in the country with one LLIN for every two residents. The universal campaign (UC) was funded by the Government of Uganda (GoU) donors that include Global Fund for AIDS, Tuberculosis and Malaria (GFATM), the President’s Malaria Initiative (PMI) the UKAID Department for International Development (DFID), and World Vision. The use of long lasting insecticidal nets (LLINs) is part of a broader Ugandan strategy which is to reduce malaria-related deaths by 70% by the end of 2015. The proportion of households with at least one LLIN increased from 47% in 2009 to 59% in 2011 The UC was government led and implemented in partnership with PMI, DFID, GFATM, World Vision, district and sub-county task forces, and Village Health Teams (VHTs). More than 40 million recipients were registered, and over 22 million LLINs were distributed by August 2014. The process evaluation is the first part of a two phase effort to assess the process and outcome of the LLIN UC. Evaluation Purpose and Scope: The purpose of the evaluation was to determine the performance and impact of the UC, provide accountability for the use of resources, and to document lessons for future comprehensive campaigns in Uganda and worldwide. The evaluation sought to answer, the following questions.  Relevance: Was the design and timing of the UC based on strong evidence? Was it appropriate and did it meet the expectations of the users?  Effectiveness: To what extent did the UC attain its objective of national coverage defined as one net for every two people in Uganda? Was this objective achieved on time? To what extent was LLIN use increased by the campaign and how credible is the evidence? What lessons can be drawn from the process?  Efficiency: Was the most efficient process adopted in implementing the UC? Were most activities implemented in a cost-efficient manner? Were there any, and if so what, alternative means could have been pursued by the program to accomplish the same outcomes at lower cost?  Impact: To what extent was LLIN ownership increased by the campaign? Will the program be able to detect any effects as a result of its activities and, to date; has there been any measurable decline in malaria morbidity and mortality following the distributions that can be attributed to the program? Are there any important unintended consequences either positive or negative of the UC?  Sustainability: Is funding sufficient to carry out the UC every three years as planned? How likely is it that the GoU will be able to conduct a UC every three years and what additional inputs (e.g. funding, infrastructure, staffing, etc.) would be needed? To what extent can the LLIN distribution Infrastructure be maintained? What are the opportunity costs of doing so? Evaluation Methods: The evaluation employed a mix of qualitative and quantitative methods including secondary document review, site and home visits, and primary data collection viii through key informant interviews (KIIs) and focus group discussions (FGDs) at central, district, sub-county and community levels. Site visits were conducted in seven districts (Jinja, Mukono, Mubende, Kabarole, Mbarara, Gulu and Nebbi) to gain an understanding of how the UC operated in different contexts. Findings Relevance: The UC in Uganda was relevant and its implementation was timely to address the huge malaria burden and mitigate the socio economic effects of the disease. The design of the UC was highly appropriate in both concept and implementation, and was based on international guidance. At the time of the evaluation, LLIN ownership and use was low and the old nets needed replacement. It was envisaged that the UC would achieve community benefit by increasing LLIN coverage to at least 85%. LLIN recipients expressed appreciation and were anticipating health benefits from the campaign. Effectiveness: A total of 22,267,777LLINs were distributed by the end of the eighth wave, achieving estimated population coverage of 98% assuming 1.8 persons per net. The campaign was generally successful in achieving the UC as defined apart from Kampala and Wakiso districts where universal coverage was not achieved due to shortage of nets following under￾estimation of need. Efficiency: Planning for the UC was thorough and professional, drawing heavily on both Ugandan and international experience. The MoH played the leadership role through the National Coordination Committee (NCC), significantly assisted by international partners and other stakeholders. A lead implementing agency, the Malaria Consortium (MC), provided extensive operational support. Donor contributions to the UC totaled $105,040,000, about $80 million was spent on procurement of nets and $25 million on distribution and advocacy support. GoU in-kind contribution included provision of goods, services and human resource. UC activities were generally implemented timely apart from slight from wave 5 of the distribution due to procurement delays. The pilot distribution of LLINs and subsequent implementation in waves allowed optimal use of resources and adoption of lessons learnt from the distribution of previous waves. Effects: The health outcomes of this program in terms of preventive behavior and a reduction in malaria related morbidity will be measured through the 2014 Malaria Indicator Survey (MIS) and the Phase 2 UC evaluation. While we conclude that the campaign was highly effective in achieving UC coverage objectives, it is only when coupled with correct and consistent use of LLINs at the household level that this translates into effects (or a reduction in malaria related morbidity and mortality ix Sustainability: Health benefits of the UC can be maintained only if recipients maintain their nets and use them correctly and consistently. Coverage of LLINs needs to be maintained through routine distribution. The MoH has built capacity (National Coordinating Committee (NCC), the district task force, the sub county task force and the Village Health Teams (VHTs) that can sustain future UCs, however support from lead implementing agencies is still relevant as capacity is strengthened within the MoH systems. Recommendations MoH has the policy framework for future UC and can lead its implementation however financial support from donors is still needed. Lead implementing agencies and malaria partners should continue to provide critical support for future UC as capacity is built within government systems. There should be an inbuilt mechanism for flexibility and adaptation in conducting future UC such as pilot distribution, regional waves spread over a period and task forces that can make decisions to allow implementation data to guide the next distribution process. Future campaigns should have a well built-in process for data collection, transmission, management, availability, completeness, accuracy and timely use. We recommend that future UC funding be linked to evidence of improved use of management information. All UC distribution channels should be supported by proven Inter Personal Communication activities focused at the local level to inform households, and interpersonal communication should be emphasized to promote proper and consistent use of LLINs for impact. The MoH should work hand in hand with Uganda Bureau of Statistics (UBOS) in future UC population and household projection and enumeration to avoid population discrepancies. 1 1.0. Program Background According to the 2011 Uganda Demographic Health Survey (DHS), the proportion of households with at least one long lasting insecticide treated net (LLIN) increased to 59% in 2011, from 47% in 2009.1 Considering the fact that the effective lifespan of LLINs is around 3 years, Uganda needed to replace the LLINs distributed in early 2010 to pregnant women and children by the beginning of 2013. Reducing high levels of morbidity and mortality associated with malaria in Uganda requires sustained delivery of proven prevention and treatment strategies to scale. The three primary interventions recommended by the World Health Organization (WHO) Global Malaria Programme to tackle malaria are: a) distribution of insecticide-treated nets (ITNs), more specifically LLIN, to achieve universal coverage; b) indoor residual spraying (IRS) of houses to reduce and eliminate malaria transmission; and, c) diagnosis of malaria cases and treatment with effective medicines. Uganda’s inability in the past to take these proven anti-malarial interventions to scale is the primary reason why malaria prevalence and malaria related deaths has remained so high. The National Malaria Control Program (NMCP) of the Government of Uganda (GoU) and its partners launched a major initiative to achieve universal coverage (UC) of LLINs in Uganda in May 2013. Endorsed by the World Health Organization (WHO), the Universal Coverage is part of a broader Ugandan strategy 2 to reduce malaria-related deaths by 70% by the end of 2015. As per WHO recommendations, LLINs would be distributed at no cost to all Ugandan households through a mass-distribution campaign and to all pregnant women and infants during ante-natal and immunization visits. To undertake this initiative, Uganda needed at least 20 million LLINs. The Global Fund for AIDS, TB and Malaria (GFATM) provided 15 million of the LLINs required, Department for International Development (DFID), the US President’s Malaria Initiative (PMI)3 and World Vision supported the procurement and distribution of the remaining 5.5 million LLINs. The mass distribution campaign was government led and implemented in partnership with PMI, DFID, GFATM, World Vision, district and sub-county task forces, and Village Health Teams. This was the first time that Uganda attempted Universal Coverage of LLINs, and the first time any country has attempted to roll out LLINs at such a scale, therefore understanding the implementation process and impact of the program and learning lessons from it for future programs is essential. 1 Uganda Demographic Health Survey - 2011 available www.ubos.org/onlinefiles/uploads/ubos/UDHS/UDHS2011.pdf 2 Uganda Malaria reduction strategy 2014 - 2020 3 The President’s Malaria Initiative (PMI) is a US Government partnership between us and the Centers for Disease Control and Prevention (CDC). 2 The evaluation commenced in June 2014 with development of methodology plans and document reviews. Existing guidelines were reviewed and tools were developed. The evaluation team conducted site visits throughout July including visits to Jinja, Mukono, Mubende, Kabarole, Mbarara, Gulu and Nebbi districts. LLIN storage and distribution sites and records of distribution were reviewed. 2.0. Evaluation Purpose The purpose of the evaluation was to determine the performance and impact of the UC, provide accountability for the use of resources, and document lessons for future comprehensive campaigns in Uganda and worldwide. The evaluation was implemented in two phases; phase I which is the a process evaluation sought to 1) Understand the extent to which the universal LLIN distribution campaign was implemented according to plan, and extract lessons relevant to implementation of future campaigns, broken down across the different components of the procurement and distribution plan. 2) Evaluate the management of the program, and determine whether the stakeholders funding and implementing it have designed and implemented it in such a way as to enable us draw lessons or learn from the experience and use the program to improve the level of data for decision making around malaria control and treatment in Uganda in general. 3) Review the design and instruments for the 2014 Malaria Indicator Survey (MIS) to determine if there are any feasible changes that would strengthen the ability of the survey to draw conclusions about the impact of the UC and its different components (as well as the objectives of Phase II below), with changes limited to those that do not threaten the comparability of the 2009 and 2014 MIS data. Phase II will aim at 1) using the findings of the 2014 MIS to articulate the effects of the UC on malaria indicators, identifying needs for additional analyses as necessary, and 2) identifying and reviewing reliable data on LLIN ownership, use, and effects and reaffirm the understanding of the outcomes and effects of the UC beyond the MIS. The overall evaluation sought to answer the following questions:  Relevance: Was the design and timing of the UC based on strong evidence? Was it appropriate and did it meet the expectations of the users?  Effectiveness: To what extent did the UC attain its objective of national coverage, defined as one net for every two people in Uganda? Was this objective achieved on time? To what extent was LLIN use increased by the campaign and how credible is the evidence? What lessons can be drawn from the process?  Efficiency: Was the most efficient process adopted in implementing the UC? Were most activities implemented in a cost-efficient manner Were there any, and if so what, 3 alternative means could have been pursued by the program to accomplish the same outcomes at lower cost?  Impact: To what extent was LLIN ownership increased by the campaign? Will the program be able to detect any impacts as a result of its activities and, to date, has there been any measurable decline in malaria morbidity and mortality following the distributions that can be attributed to the program? Are there any important unintended consequences—either positive or negative of the UC?  Sustainability: Is funding sufficient to carry out the UC every three years as planned? How likely is it that the GoU will be able to conduct a UC every three years and what additional inputs (e.g. funding, infrastructure, staffing, etc.) would be needed? To what extent can the LLIN distribution Infrastructure be maintained? What are the opportunity costs of doing so? 3.0. Evaluation Methods and Limitations 3.1. Methods The evaluation employed a mix of qualitative and quantitative methods including: secondary document review, site and home visits, and primary data collection through key informant interviews (KIIs) and focus group discussions (FGDs) that were conducted at central, district, sub-county and community levels. The evaluation team conducted site visits in seven districts (Jinja, Mukono, Mubende, Kabarole, Mbarara, Gulu and Nebbi) to gain an understanding of how the UC operated in different contexts and at different levels. The sampling of districts and sub-counties was purposive and this was based on malaria transmission intensity, accessibility, geographical representation, and stakeholder advice. Relevant documents pertinent to the UC were reviewed, among which included the UC work plan, the UC budget and minutes, reports of the NCC and sub-committees, Malaria Consortium (MC) and USAID Uganda’s Stop Malaria Program (SMP). At the national level, FGDs were conducted with the National Coordination Committee (NCC) members, Medicines and Health Services Delivery Monitoring Unit (MHSDMU) members, the National Drug Authority (NDA), DFID, WHO, MC, and with Stop Malaria Program. At the district level, FGDs were conducted with the District and Sub County Task Force members and community members from different sub counties in the selected districts. Key Informant Interviews were also conducted with GFATM, ABT Associates, PMI staff, district (Police commander, Army Commander, internal Security Officer, DHT members, and VHT members in a sample of villages. In addition, the evaluators further observed LLIN distribution activities in districts where the distribution of nets was ongoing. 3.2. Limitations The evaluation was conducted when collation of data and various reports by the MoH and implementing partners was ongoing, as a result some required information was not available or provided in a timely manner. Secondly, at the time of the evaluation, the final cleaned LLIN 4 distribution data was not available from the MoH, however available data from malaria control partners was collated and used in the report. The evaluation team did not verify the available data or carry out data quality assessment hence the quality of the data questionable. Some individuals were not available for interviews due to tight work schedules and competing priorities. 3.3. Evaluation Team The evaluation team was composed of five consultants namely Thomas Park, Wayne Stinson, Caroline Asiimwe, Adoke Yeka, and Jennifer Peters. The team included international malaria program experts and senior Ugandan malaria specialist. The team was led by Wayne Stinson, a senior evaluation expert with over 15 years of experience evaluating health programs and with knowledge and experience in malaria programs. In addition to the team were two senior officials from NMCP who included Dr, Humphrey Wanzira and Dr. Henry Katamba who were nominated to join the evaluation team. They coordinated the field activities and also participated in data collection. 4.0. Relevance and Implementation of the Universal Campaign 4.1. Relevance of the Universal Campaign in the Ugandan Context. Malaria is an endemic in 95% of Uganda, with approximately 90% of the population (an estimated 32 million people) at risk. Malaria is Uganda’s leading cause of morbidity and mortality and the primary contributor to poverty and low productivity, with 20% of household expenditure spent on malaria treatment.4 Due to the disease’s economic impact, malaria control is a cornerstone of the National Development Plan 2010/11 – 2014/15, with several malaria￾based indicators for monitoring progress. Malaria control is also a critical component of the Uganda Ministry of Health, Health Sector Strategic Plan III, 2010/11-2014/155 , the Uganda Minimum Health Care Package and the Uganda malaria reduction strategic plan 2014-2020. 6 LLINs have played an important role in the remarkable success in reducing malaria burden over the past decade7 . They are a core prevention tool, and widely used by people at risk of malaria. The WHO recommends universal coverage of LLINs for all people at risk of malaria to achieve community benefits. A Cochrane Review in 20098 showed that correct and consistent use of LLINs not only protected those sleeping under them but also had a protective “herd-effect” on others sleeping nearby when coverage rates reached 60% or higher. In order 4 Uganda National Malaria Strategic Plan 2010/11 - 2014/15 5 Uganda Ministry of Health, Health Sector Strategic Plan III, 2010/11-2014/15 6 The Uganda malaria reduction strategic plan 2014-2020 7 World Health Organization. World Malaria Report 2012. Geneva, 2012 8 http://summaries.cochrane.org/CD000363/INFECTN_insecticide-treated-nets-can-reduce-deaths-in-children-by-one-fifth￾and-episodes-of-malaria-by-half 5 to achieve and maintain universal coverage, countries should apply a combination of mass free distributions and continuous distributions through multiple channels, in particular antenatal and immunization services. The Uganda National Malaria Strategic Plan 2010/11 - 2014/15 and the malaria reduction strategic plan 2014-2020 spell out plans to achieve and sustain protection of at least 85% of the population at risk of malaria through recommended malaria prevention measures including use of LLINs. Under this strategic plan period, LLINs will be deployed to achieve and sustain universal coverage through mass distribution campaigns repeated every three years and continuous LLIN distribution through antenatal clinic (ANC), expanded program for immunization (EPI) and schools in the whole country. The approach by the MoH to undertake a UC as it routinely distributes LLINs to the most vulnerable populations (pregnant women and children) is in line with most recent thinking. Using both approaches has been found to be cost-effective in several studies in Kenya, Zambia and parts of West Africa. The UC in Uganda was therefore timely and relevant in concept and was based on international guidance. LLIN ownership and use was low and old nets needed replacement. The UC would achieve community benefit by increasing LLIN coverage to at least 85%. The UC was therefore an appropriate intervention implemented at the right time in Uganda. The UC was relevant from users’ perspective as well. Public health specialists acknowledged the benefit of malaria prevention as did well-informed users. Other users however expressed additional benefits such as protection from nuisance bites. Some users reported that the UC showed government positive response to the health needs of the population and boosted their support for the government. A minority, though, expressed concerns about the material and color of the nets and sharing nets with other household members as highlighted by some user comments “Many residents would have preferred a white net made of soft polyester to a blue net made of stiffer polyethylene” There is great enthusiasm about the LLIN distribution, however there is concern about sharing one net by two people since most people especially in urban areas don’t share beds. “Many people do not want to share nets with other household members and yet they were provided with one net for two people”. The first concern arose primarily in districts where people sew both soft polyester and the stiffer polyethylene nets, although it was unclear whether the objection was to color or material. Possible user preferences did were not considered in procurement. An analysis of the The mass campaign is timely and welcome and will boost efforts to control malaria. Director ABT Associates 6 qualitative findings indicated this might not have been considered majorly because of the need to control costs and adherence to GF procurement procedures (no differentiation between polyethylene and polyester). Spot checks found newly-distributed LLINs hanging in all homes visited, with predominantly positive comments about the program and the nets. The complaints that we did hear were minor but worth noting for future distributions and included:  Concerns that LLINs might be flammable (in fact, nets will initially melt and might later burn).  Hooks distributed with the LLINs are not long enough to hold in mud walls.  There was dissatisfaction that instructions were not attached to the nets or provided.  Some people in urban areas, who used double-sized (4x6, 5x6, 6x6) beds complained that nets were too small to fit on their large beds. In conclusion, the UC largely met user expectations from both a public health perspective and according to individual users, although for different reasons. 4.2. Design of the universal campaign Implementation guidelines for the UC process were developed jointly by the MoH and stakeholders, including the President’s Malaria Initiative (PMI), DFID/UK Aid, The Global Fund for AIDS, Tuberculosis and Malaria (GFATM), the World Health Organization (WHO), World Vision, and UNICEF Country Offices9 . The guidelines had seven key elements namely 1. Campaign management structure 2. Financing and financial management of the LLIN campaign 3. Procurement, transportation and storage of LLINs 4. Household registration and LLIN allocation 5. Distribution of LLINs to beneficiaries 6. Training and supervision 7. Advocacy, social mobilization and behavior change communication The implementation plan and guidelines recognized the importance of the different stakeholders who were either directly or indirectly involved in the UC including the Office of the President, The Parliament of the Republic of Uganda, Health Development Partners, The Armed Forces, Global Fund Country Coordinating Mechanism, The Ministry of Health, and The District Local Governments. The design of the UC 2013-14 largely followed international protocols, as developed by RBM /Alliance for Malaria Prevention (AMP)10. AMP is a partnership focused on malaria prevention with LLINs, both through mass distribution 9 Ministry of Health Uganda, 2013; Mass Distribution of Long Lasting Insecticide Treated Nets to Achieve Universal Coverage in Uganda: Detailed Implementation Guidelines; Kampala, Uganda 10 RBM, Alliance for Malaria Prevention (http://allianceformalariaprevention.com/index.php) 7 campaigns and through continuous distribution channels, to achieve and maintain universal coverage targets; it focused on the development and sharing of best practices, lessons learned and recommendations for malaria prevention, and supported implementation of LLIN distribution activities with the objective of strengthening skill sets and capacity at country level. The implementation was further guided by lessons learnt from a pilot LLIN mass distribution campaign implemented by PMI in Eastern Uganda. The pilot provided vital lessons and operational experience that guided planning the scale up of the campaign. 4.2.1. Campaign Coordination Structures The Government of Uganda through the MoH provided overall leadership and technical guidance for the whole process of the UC. The ministry developed and rolled out a single national implementation plan, under the leadership of the NMCP. The donors used the USAID’s Stop Malaria Project (SMP) mechanism under John Hopkins University Center for Communication Programs (JHUCCP) that managed the financial aspects of the UC. SMP appointed Malaria Consortium (MC) as a technical sub-partner to managed the UC campaign funds from DFID/PMI. The GFATM appointed MC as a technical sub partner to manage UC funds from the GFATM. Having one experienced lead partner ensured better coordination of the UC. To effectively manage the campaign, different structures were established at different levels. Task forces were formed at national, district and sub-county levels that coordinated and monitored the campaign activities including registration, advocacy, BCC, security, distribution, selection of appropriate distribution points, and oversight of storage. The composition of each of the structures is clearly defined in the national implementation guidelines and summarized in Annex 2. A summary of the coordination structures is shown in Figure 1. Figure 1: UC Implementation Coordination Structure 8 The overall leadership for the UC was provided by the National Coordination Committee (NCC) chaired by the Director General of Health Services (DGHS). The NCC comprised of representatives from the MoH, other government ministries, UNICEF, WHO, World Vision, PMI /USAID, DFID, GFATM, Makerere College of Health Sciences (MakCHS), professional associations, local governments, civil society organizations, cultural institutions, the armed forces and the president’s office. The membership of the NCC was very broad, and representative of the spectrum of malaria stakeholders. NCC members were competent enough to make strategic decisions to guide the UC. Below the NCC were three different sub￾committees (Operations, Logistics, and Advocacy & social mobilization) whose decisions 9 would feed into the NCC for review and final approval. The relationships coordinated through the NCC were complex yet ultimately effective: decisions were made, responsibilities shared, and LLINs distributed. A review of a sample of the minutes of the NCC monthly meetings demonstrated commitment to a well-managed and coordinated consultative process, and adherence to the implementation design. Minutes documented full discussion of opportunities, recommendations and challenges. A number of Key Informants appreciated the NCC management, although some expressed reservations on the level of commitment to bridge gaps between lead UC implementers and the MoH. According to some respondents, the lead implementer (MC) sometimes appeared unwilling to apply NCC decisions and provide mutual and transparent accountability on financial resources. Despite these obstacles, our interpretation of the finding is that the NCC and other task forces acted well as collective bodies as defined in the Implementation Guidelines. Collaboration between the districts, sub-county, the NMCP structures and implementing partners generally worked well, decisions made were implemented in a timely manner. The campaign was implemented in a top-down model, with limited scope for decentralization of some decisions to the local level. International guidelines (Alliance for Malaria Prevention) strongly encourage decentralized participation and ownership, but the guidelines in Uganda did not provide for adaption at the field level. In the words of one key informant: 4.2.2. Procurement of LLINs The procurement process began with quantification of need, followed by commitment of donor funds. Funding came from GFATM, PMI, DFID and World Vision. GFATM-financed procurement was planned through the Voluntary Pooled Procurement (VPP) mechanism; PMI/DFID procurement occurred through USAID’s global commodity contractor (JSI/DELIVER). The VPP mechanism is a strategic initiative established by the Global Fund (GF) aimed at ensuring a cost-effective and cost-efficient procurement process focusing on efficient, timely and reliable procurement, stringent quality standards for procured products and attractive pricing for key health products. UC planning in 2012-13 assumed that all procurement would be complete by the end of 2013. JSI/DELIVER, the procuring agent of PMI/DFID, imported 7,399,840 LLINs between May 2012 and July 2014 while World Vision procured 506,600 LLINs. PSI, a VPP agent for the Global Fund, imported 15,004,636 LLINs from a number of sources including Sumimoto Chemicals, Cargo ex China, Vector Health International (Arusha), Vestergaard Frandsen (Vietnam), Disease Control Technologies (India) between February and August 2013. . Two problems complicated implementation. The first problem, potentially serious, was that estimates of LLIN needed increased from 19,642,778 to 22,589,194 (rise of 15%), based on household registration data; secondly procurement was delayed in late 2013 because of global National planners neglected our views and sent us supervisors who were rigid. They did not give us room for flexibility as some of the guidelines were inapplicable in our context. Key Informant 10 production delays. The numbers of household members registered were much higher than initial estimates based on projected national population. Donor funding requirements increased as a result, but were somewhat offset by a one-third reduction in cost per net, credited to VPP. The second procurement-related problem was that global production shortages delayed importation and distribution by approximately four months in 2014. While the factors leading to funding, procurement and importation delays – but also the unexpected cost reduction – are unlikely to occur exactly this way again, it is clear that a successful UC in 2016-17 will require both internal and external factors to be closely managed and monitored. Uganda and its donor partners coped well with shifting needs and will have to be similarly flexible for future campaigns. 4.2.3. Quality Assurance of LLINs LLINs that met the WHO Pesticide Evaluation Scheme (WHOPES) Level II specification were procured and imported by JSI/Deliver the procuring agent of PMI/DFID, the World Vision and PSI, a VPP agent for the Global Fund. LLINs were imported in bales of 40 with individual nets un-wrapped as requested by MoH. All procured LLINs underwent pre-shipment testing according to international guidelines. The National Drugs Authority (NDA) carried out quality tests of a sample of LLINs to ensure that imports met minimum quality standards including tensile strength, bursting strength and other physical properties of the fabric, however testing was not done for all batches of LLINs because the acquisition of testing machines funded by GFATM was not completed by the time of the UC. The possibility of importing poor quality LLINs is however very remote since pre shipment testing followed international guidelines. 4.2.4. Training and Supervision Rollout of the UC involved a significant amount of orientation, sensitization and training. The central team (NMCP and Lead Agency) was trained to carry out training and supervision at district and lower level. The MoH and district staffs were trained in the use of standard tools for micro planning at the village level. All persons involved in or responsible for the transportation, storage and distribution of the LLINs were trained on the use of supply chain management tracking tools. The District leadership was sensitized on the LLINs distribution campaign. Sub County (Sub County) trainers/supervisors were trained to equip them with appropriate knowledge and skills to train and supervise the Village Health Team (VHTs) and Local Councils (LCs) on the LLIN campaign. National sensitization was mainly an advocacy strategy targeting Members of Parliament, Ministers, religious leaders, and other key stakeholders. The MoH and MC trained cluster supervisors, district supervisors, district coordinators and sub-county supervisors for 3 days. Regional sensitization targeted district leaders such as Resident District Commissioners (RDCs), District Health Officers (DHOs), Malaria Focal Persons (MFPs), Local Council Five (LCV) chairpersons and secretaries for Health. Each regional meeting involved 5 to 18 districts. The main purpose was to introduce the campaign to the district leadership so that they plan, implement and monitor all the UC activities in their respective districts. 11 The UC also involved ongoing supervision from central to the local level. The targeted number of personnel to be trained is shown in Table 1. Table 1Training Plans for Stakeholders at Various Levels Staffing Level Cadre Training target Central ToTs MOH Cluster Supervisor, 3-4 districts MOH District Supervisors MOH Sub-county Supervisors MC District Coordinators 1 per 3-4 districts 1 per district 1 per sub-county 1 per district Regional Police, UPDF, Commanders CAOs RDCs LC IV DHOs 1 each per region 1 per region 1 per region 1 per region 1 per region District District Leaders District Taskforce Teams 30 per district 17 per taskforce Sub-County Taskforce Chiefs OC Police GISO Parish Counselors Parish Chief LC III Chairperson Health Assistant 1 per sub-county 1 per sub-county 1 per sub-county 2 per sub-county 1 per sub-county 1 per sub-county 1 per sub-county Village VHT LC I 2 per village 1 per village The evaluation team was unable to establish if the intended number of trainees was trained because the final district training records and reports were not available at the time of the evaluation. The team however summarized the number of persons trained and sensitized at district level in the final five waves of the campaign using the MC Weekly Reports (figure 2). 12 Figure 2. Trainings & Sensitization across Waves 4-8 in 74 districts VHT training focused on household registration and completion of the registration forms procedures, the distribution exercise, and key LLIN BCC messages for the community. During the training and sensitization, the VHTs and the LC1s identified central distribution points for LLINs. Summary reports from MC indicated that district set-up/start-up training and sensitization activities and these were generally held on time although there were a few exceptions when training delayed at the district level. The MC report for Eastern Uganda (Sept 2012) recommended that the MoH to consider spreading VHT training over a two or three day period at parish level and that sessions participants should not exceed 50. Activities in each district were initiated by MoH Clusters, each consisting of the Cluster Supervisor, the district supervisor, and a variable number of sub-county supervisors. After participating in central training of trainers, teams spent an average of three weeks in each district, sensitizing leaders, creating district taskforces, and initiating sub-county rollout processes (including VHT training and household registration). Each District Taskforce sensitized sub-county leaders and LC1 chairpersons, created taskforces at that level, and trained VHTs. The Cluster Supervisor generally moved to a new district after several days, leaving the District Supervisor in charge but returning to validate registration and oversee distribution. The entire process, from sensitization to net distribution, was expected to take 21 days in each district, and reports indicate that it generally did. Given that the rollout of UC campaign was relatively successful, one can infer that the training achieved its objectives. District level performance in carrying out training and sensitization activities was noteworthy. Sensitization of district leaders took place in all regions and the 13 various UC implementers were trained. District rollout generally followed the UC central plan and schedule. 4.2.5. Social Mobilization and Behavioral Change Communication The goal of the National Communication Plan for UC, dated July 2013, was to contribute to increased ownership, care and correct use of LLINs, hence contribute to the overall reduction of morbidity and mortality due to malaria in Uganda. The plan targeted three different audiences: LLIN recipients, influential community leaders, and health sector partners and political leaders. At central and district levels, advocacy meetings were held with various stakeholders, including Parliament and district leaders respectively. Radio talk shows and messaging were done locally to alert communities of the upcoming campaign activities and to promote net use and other malaria control and prevention behaviors. At the community level there were demonstrations on the use of LLINs at the distribution points, interpersonal communication, and distribution of Information Education and Communication (IEC) materials and the promotion of LLIN use among primary school children To reach the primary audience, behavioral change communication (BCC) efforts included a mix of mass media including: radio, TV, posters, leaflets, phone messaging, hotlines and outdoor advertising. Interpersonal communications and mobilization activities included community and household visits by VHTs and volunteers to mobilize people to take part in the campaign and provide basic education and information about malaria. Secondary and tertiary audiences were reached through a smaller mix of sensitization and interpersonal meetings, and mass media efforts (such as print materials and newspaper and other reports and circulars as well as phone and TV campaigns). International guidelines recommend that communication should largely emphasize interpersonal channels (verbal and non-verbal messages in face-to-face communication) like speech, vision and body language. The UC plan as developed emphasized ceremonial events and use of mass media (radio, TV and newspapers). Communication at points of distribution on the UC was insufficient. Users needed information about location of collection points and what to do with the nets once received, and yet it appears that interpersonal communication (IPC) at LLIN distribution points was particularly weak. There was a particular concern about vertical implementation of BCC activities. Key informants encouraged greater use of local media as well as more decentralized decision￾making about appropriate communication channels. A decentralized BCC ensured an integrated approach to community mobilization through Sub County and community leaders, religious and opinion leaders, and schools at a reduced cost as long as clear and consistent messages are well packaged for the target audience. Social mobilization was one of the weakest areas of the UC. While measuring population awareness is beyond the scope of this evaluation, a common opinion arising from a number of 14 people interviewed points to limited BCC and IPC activities as reflected in a Focus Group Discussion at Mugusu HC III in Kabarole district. During in-depth discussions with individuals at the MoH, discussants felt that the use of mass media based mobilization was not adequately planned and implemented. They cited several challenges such as underestimating the need for BCC in a mass LLIN campaign; lack of communication and training for LLIN distributors; inconsistent and untimely messaging; lack of a balance between mass mobilization and IPC; and failure to engage NGOs, political and district leaders in BCC implementation. A unique opinion portraying lack of in-depth planning, was failure to consider BCC messages on the usefulness of LLINs in IRS districts. In all the seven districts the team visited, many of the challenges and fears communicated at national level were felt. Two community educators working with Mugusu sub-county, Kabarole district said; ‘People came asking for free bed nets, but we were not aware of such a program, whether it was true, when it would be and where it would take place. We told them there is no such program in our sub county. The next day, on a Saturday, we received abrupt calls from Kampala to come and distribute bed nets. This was very embarrassing. ’ The leadership from Pece Division, Gulu were in agreement and stated that, ‘We heard radio messages but we felt that they were irrelevant to the actual BCC need in our division. They should have let us design our own music, drama, with local opinion leaders, women and people living with HIV/AIDS groups. A common opinion arising from 26 men and women the team interacted with at Mugusu HC III in Kabarole district, points to limited BCC and IPC in this sub-county; ‘We received the bed nets, but we were not taught how often to wash them, use them and sleep in them. We don’t know how long these bed nets take before ageing and how to keep bed nets from developing holes and where to put the old ones’. ‘We received the bed nets, but we were not taught how often to wash them, use them and sleep in them. We don’t know how long these bed nets take before ageing and how to keep bed nets from developing holes and where to put the old ones’. ‘IRS districts did not have relevant and sufficient BCC messaging. LLINs working as a barrier to nuisance mosquitoes should have been a good message. I think the problem was failure to engage community-based organizations. It was also a wrong idea to expect SMP and Malaria Consortium to understand the entire country’s communication needs.’ Key Informant 15 We heard numerous suggestions that decentralized taskforces should have had greater authority to set schedules and particularly to modify BCC approaches within budget parameters. These modifications should be more feasible for the next round of universal coverage. 4.2.6. Household Registration and LLIN Allocation Registration was a massive exercise involving approximately 100,000 VHTs in 112 districts. Over a 2-3 day period in each district, two trained VHTs visited households, compiling information on the total number of residents, the number of pregnant women, and the number of children under five. LC1 chairpersons reviewed and signed registration forms; while supervisors confirmed demographic statistics while also conducting a small number of direct spot checks through home revisits. By the end of the campaign, VHTs had registered 41,034,354 residents11 , about 15% more than had been expected based on projections of the 2002 census and 18% more than UBOS population counted in August 2014. While we can speculate on the reasons for this difference, we can only point out that a difference as large as this has serious implications for planning and budgeting. 12 According to the LLIN implementation guide, it was the role of the sub county, district and cluster supervisors to cross check HH registration data with the sub county and district personnel, before LLIN allocation. Based on interviews held with some of the supervisors, verification of HH registration was not well implemented. LLIN allocation was based on one net for every two household residents, with quantities rounded up for households with odd numbers of people. Reportedly some supervisors capped the number of nets at four for large households. No provision was made for people who could not easily share a net. Allocations were written on individual registration forms, and then aggregated at several levels before being sent to the logistics coordinator. (BCC announcements that “everyone needs to register to get a net” evidently caused some to believe that every individual would receive a net.) As in other aspects of the UC, Kampala-Wakiso were managed differently for both registration and allocation. Registration stretched over 5-6 days, and a shortage of nets unavoidably limited allocation to a single net per household. According to AMP, some countries have attempted to count sleeping spaces rather than household size. Others, with large prior distributions, have deducted usable nets already in the household. Both adjustments have been problematic, however, and may be more appropriate for mop up campaigns than universal coverage. 11 According to tracking summary data provided by PMI. (We were unable to obtain an official report.) 12 Possible explanations derive from the differing incentives of census takers and those involved with LLIN registration. Census takers may not always take the difficult actions needed to locate every single household, whereas a few LLIN registrars may have exaggerated registration to justify additional nets. The “true” population figure undoubtedly lies somewhere between the two estimates, but simply splitting the difference would be inappropriate since each “unnecessary” net costs about $5.00 with distribution. 16 4.2.7. Storage and Distribution to sub-counties Storage and handling was the responsibility of Stop Malaria and MC who subcontracted to Kunhe + Nagel (K&N) and Spedag. The GF’s procurement agent, PSI, through its subcontractors, was responsible for transporting the nets from the manufacturer to the sub county stores. DFID/PMI’s procurement agent (JSI/Deliver), on the other hand, transferred nets to the USAID Stop Malaria Project (SMP) in Kampala, which in turn warehoused and eventually transported them to sub-county distribution points. The sub-county chief under oversight of the sub-county task force was responsible for managing the sub-county distribution points. Uniformed personnel provided security during transport and kept security at some distribution points. Upon allocation of LLINs, quantification data were sent to logistics teams at MC and SMP, for further allocation of LLIN bales to appropriate vehicles and districts. In general, weekly campaign reports reviewed by the Evaluation team noted the need for more time for allocation and transport of LLINs from the central level to districts. As a result of experiences in waves 2 – 4, later waves allocated longer time periods and used this time for training and other preparations so that once LLINs arrived, they could proceed in rolling out the remaining LLIN distribution activities. Other transport challenges noted included: delayed release of funds to and mobilization of delivery trucks; poor road conditions due to rainfall, and remote terrain. The UC also faced issues on the correct allocation of LLINs to districts and sub-counties, and furthermore with transport of the correct allocation to the correct areas. (In a number of instances, the transporter delivered the wrong number of nets; one truck couldn't find its destination and returned to Kampala.) However, in almost all cases, the districts were able to resolve these excesses/shortages in their own district or with assistance from neighboring districts, through the oversight of the NMCP and implementing partners, namely the MC. Despite these relatively minor issues, unexpected delays/constraints in transport did not affect the successful and timely rollout of any of the campaign waves. Warehouses used for storage maintained stock cards showing goods received and goods loaded to trucks for distribution. Each sub-county also had a “goods received” logbook. No losses were reported at the warehouse level, however there were discrepancies in delivery notes during some waves of distribution as shown in table 2. 17 Table 2. Records of LLIN Deliveries Waves Quantities delivered as per delivery note Approx. Bales Discrepancy in delivery notes Wave 1 Data not Accessed Data not Accessed Data not Accessed Wave 1 Wave 2 2,085,040 52,126 Unknown Wave 3 3,740,240 93,506 3,718 Wave 4 2,893,760 72,344 32 Wave 5 K+N 2,265,200 56,630 64 Wave 5 Spedag 552,899 13,822.48 Unknown Wave 6 Data not Accessed Data not Accessed - Wave 7 2,685,960 67,149 18 Wave 8 167,837 4,195.93 83 Wakiso 575,209 14,380.23 0 Total LLIN bales unaccounted for (excluding wave 2, partly 5) 3,915 Source: NCC Logistics Sub Committee The evaluation team established that approximately fifty bales of nets were stolen at the sub￾county level, but these were recovered and the culprit jailed. There were a few other instances of small numbers of nets reported missing or bales received with fewer nets than intended, however, none of these instances had significant consequence to the roll out of the campaign. Other challenges included shortfalls/mixes in the allocation of nets sent to districts (switching of allocations among districts, or within districts to sub-counties); despite this shortages/mix ups, most districts managed to cover shortfalls from excesses elsewhere within their own or with neighboring districts, and without any significant delay/hassle. Challenges in transport and other issues hampering the campaign were also addressed quickly and for most part locally, without causing delay. 4.2.8. Distribution of LLIN to Household During the course of the campaign, more than 120,000 VHTs distributed more than 20 million nets to approximately 41 million individuals. Most distribution points were situated on well￾known public gathering places, for easy access by the communities although some villages had no nearby distribution points to access. LLIN distribution took varying periods of time at each distribution point, depending on prevailing weather conditions and the time of LLIN delivery from the sub counties to the distribution points. Further delays were encountered at some places when sub counties did not receive LLINs on scheduled dates. Another tier of distribution not previously planned for emerged. The VHTs and local leaders delivered LLINs to households, whose members were registered but failed to access the 18 distribution points for a number of reasons. There was no transport arrangement for this unplanned but important activity. The VHTs walked to these homes, used bicycles, motorcycles, or mobilized other transport means with sub-county leaders. Some had to cross rivers to reach some households while others used boats to access islands. This initially unplanned for distribution mode was a massive achievement by any standard, though BCC activities could not be carried out during such distribution as it was only planned to take place at the main distribution points. The community members assembled to receive LLINs at their respective distribution points on the appointed days/dates. On the day of distribution, community members were however not adequately sensitized on proper net use and care before receiving their nets. Most VHTs were overwhelmed by the number of people who turned up and concentrated on tallying and distributing nets and paid little attention to sensitization. The evaluation team found only a few hold/ups/constraints to implementation in some of the waves/districts. Delays in transport of nets from central level to districts and sub-counties and in a few instances, delays in funding to districts, caused minor delays in selected areas; however, the districts managed to overcome these without significantly altering timelines for completion of their campaign. What is important to note is that none of the waves or distribution in districts was postponed due to operational issues.13District level performance in carrying out UC responsibilities was noteworthy: particularly the task forces and units below, and in their initiative and improvisation shown in overcoming problems. The support roles played by NMCP cluster groups through implementing partners showed effective collaboration and supervision. District rollout generally followed the UC central plan and schedule. 4.2.9. Information for decision-making The UC was planned to be data-driven to a great extent. Population projections based on UBOS Census from 2002 were used to estimate the needs for universal coverage. The allocation of nets and distribution was planned to be guided by household registration data and real-time reports on commodity movements, from warehouses to sub-counties, to distribution points and homes. Registration data was to move upwards for aggregation in the Resource Center to guide every aspect of decision making. Figure 3 summarizes the flow of data from collection by the VHTs to processing sites at district and the MoH. Information for decision-making started with household registration data, as described above, followed by real-time reports on commodity movements, from warehouses to sub-counties, to distribution points and homes. Registration data collected at the district level was submitted to the MoH Resource Center for aggregation to guide allocation of LLINs to various distribution points. There were delays in transmitting the registration data from the districts to the resource center, often leading to delays in allocating LLINS to the districts. After the distribution 13 Note though that completion of Wave 8 (Kampala-Wakiso) was delayed from March to August 2014because of a shortage of LLINs. 19 exercise, the final distribution data was transmitted to the Resource Centre for compilation, however, when the evaluation team visited the Center in July 2014, data remained in perhaps one hundred boxes because funds had not been released for data entry and the backlog was cleared by December 2014. 20 Figure 3. Flow of data from collection by the VHTs to processing sites at the district and the MoH Source; Ministry of Health Uganda, 2013; Mass Distribution of Long Lasting Insecticide Treated Nets to Achieve Universal Coverage in Uganda: Detailed Implementation Guidelines; Kampala, Uganda The overall progress of district implementation was tracked in several formal and informal ways and periodically reported to the NCC. The chair of the Monitoring and Evaluation Operations sub-committee also oversaw district supervision clusters and thus had first-hand knowledge of progress and problems. In addition, the Malaria Consortium prepared weekly reports, summarizing progress to date. However, important information was sometimes not 21 timely or well-presented, reducing its effectiveness as a management tool. Objectives for use of data in decision-making were admirable but sometimes beyond reach because of inadequate resources. Overall, the evaluation team concludes that the UC Implementation Plan was well designed, carefully developed, clear and sufficiently detailed to make it accessible and usable by all stakeholders and partners. The campaign proceeded for the most part according to plan; however, the team believes the UC implementation plan could be strengthened in a few areas, notably:  Inclusion of more time between waves to allow for the processing of results, analysis of implementation and timeline required  More flexibility and delegation of authority for local adaptation of plans at district and sub￾county levels;  Additional detail about the flow of data and reports back to districts after the campaign is complete, including follow up reports and evaluation activities to take place after the campaign. These activities are deemed critical both for this campaign and to inform future ones. 5.0. Effectiveness of the Universal Campaign in Attaining its Objective of National Coverage The objective of the UC was to attain universal coverage (defined as one net for every two persons). For procurement purposes and per international guidelines, the calculation was adjusted when quantifying at the population level to an overall ratio of one LLIN for every 1.8 persons since half of households have an odd number of members. Distribution coverage reports (figure 4) indicate that Uganda successfully distributed 22,267,777 nets country￾wide14 and generally in the correct quantities to the places intended, on time and consistently, until LLIN supplies were exhausted and the final three waves delayed as result. The last-minute shortage of nets was “resolved” when PMI reallocated 1.2 million nets initially intended for routine ANC distribution and NMCP altered the distribution formula for Kampala city and urban Wakiso district. (Instead of one net for two persons, Kampala-Wakiso families in urban areas were given one net per household regardless of size, with one additional for pregnant women or children under 5.)15 15 The need for these adjustments was evidently not foreseen, apparently because inventory records had not been updated. 22 Figure 4. Summary of LLIN Distribution by Wave A total of 22,267,777 LLINs were distributed by the end of the eighth wave (pilot plus waves 1-8), achieving estimated population coverage of 98% to 114% using population estimates from the household registration and UBOS forecast figures respectively and assuming 1.8 persons per net (figures 4 and 5). It is clear from these figures that the campaign was successful in distributing more than enough LLINs to achieve UC, as defined, in all waves except the pilot and Kampala-Wakiso. Based on household registration data, all waves achieved a coverage above 100% apart from the pilot, Wakiso and Kampala districts which implies that at most every 2 registered persons received an LLIN. 23 Figure 5. Population Coverage by Wave Distribution coverage reports (table 4) indicate that Uganda successfully distributed 22,267,777 nets country-wide16 and generally in the correct quantities to the intended places, consistently and on time. There were delays in the last three waves mainly because the supplies were finished and there were procurement delays. Despite the achievement, there are number of questions that remain un answered regarding estimated population size, forecast LLIN needs, allocation and the actual numbers of nets distributed. While it is too early to accurately assess and evaluate these questions in Phase I, the evaluation team noted the issues and recommend their inclusion in Phase II evaluation. 16 Based on tracking summary data provided by PMI 0% 20% 40% 60% 80% 100% 120% 140% 160% Pilot Wave 1 Wave2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7 Wave 8a Wave 8b National % by registration data % by projected population 24 6.0. Efficiency of the Universal Campaign. 6.1. Financing of the LLIN campaign. The UC was financed by GFATM, PMI/DFID, World Vision International, and very substantially through contributions by the Government of Uganda. The total cost of the campaign is not known, especially because GoU direct and in-kind expenditures haven't been quantified; however, donor contributions totaled $105,040,000. Of this, approximately 76% ($80 million) was spent on nets and 24% ($25 million) on distribution and advocacy support. Government contribution included payment of the uniformed forces who provided security, provision of goods, services and human resource. Government provided a variety of goods including vehicles, stationery, office equipment (computers and photocopiers), office space for personnel, storage space for the LLINs at various sub counties, utilities including water and electricity among others. Government personnel at the national and district level and the armed forces were deployed to support implementation of the UC. The District Local Government was at the center of coordinating the LLINs distribution process at Sub-county, Parish, Village and household levels. The armed forces provided security for the LLINs at national and sub national level. Between 2009 and 2012, the GFATM allocated 51% of its overall Uganda portfolio to malaria, totaling $121,180,480 however not all these funds were spent for the UC. Funds from Round 7 Phase II ($51,194,127) supported procurement of 15.5 million LLINs (nearly 75% of the total need) for the UC. In addition, $ 2,043,705 has been allocated under Round 10 Phase I and II to procure approximately 6.7 million LLINs to be distributed through ANCs, EPI, schools and commercial markets in 2015 – 16 to assist the NMCP in maintaining universal coverage in the years between its mass campaigns. DFID made a commitment in 2010 to significantly increase support for health and malaria control. In 2012-13, it provided GBP 17 million through USAID to procure and distribute 5 million LLINs, filling a critical gap in the forecast LLINs needed to achieve universal coverage. DFID intends to continue to support additional LLIN and preventive activities in the coming five years. In 2013-14, DFID increased its budget by GBP 2,900,000 to a total of 19,900,000, and committed a further GBP 500,000 for routine distribution (primarily through ANCs) in financial year (FY) 2014/15. This increase is to support costs for warehousing & distribution, training and M&E for LLIN distribution. PMI procured 1,200,000 LLINs using FY12 USAID funds, of which 200,000 were distributed routinely through ANC clinics in the central and eastern regions, and more than 650,000 distributed via the mass campaign in four districts in the eastern region of the country. USAID matched DFID critical gap contributions with $33 million of which $6,790,000 was programmed for procurement and distribution of one million LLINs through ANC clinics to 25 sustain high net ownership following the UC in 2013-14. To further support and increase the impact of the LLIN mass distribution, PMI provides funds for BCC campaigns comprised of both mass media and community mobilization strategies, which are designed to increase malaria knowledge and promote correct and consistent use of LLINs. Planning for and coordinating the timing and allocation of the considerable level of funds required for a campaign of this magnitude is a complicated logistical exercise. The collaboration between different donors, agencies and implementing partners with the MoH was driven by cooperative agreements. DFID/USAID and GFATM signed a MoU with the GoU on Dec 11, 2012 to provide joint support for the LLIN UC 2013–14. All the funds were disbursed before the actual implementation time (October 2013 – August 2014). USAID/DFID response to financing the UC was timely and complementary to the GFATM contributions. The donors had good collaboration with the MoH, quick decisions were made based on what was going on and there was flexibility to accommodate unplanned expenses due to under estimations as shown previously. A summary of the donor financial contribution to the UC is shown in table 3. Table 3. Donor Financial Contributions to the LLIN campaign. Donor Service delivery area Committed (2013-14) Actual Disbursed (2012-14) Sub recipient USD GBP USD GBP GFATM Round 7 Ph II (15.5m LLIN VPP) 66,515,210 62,229,429 MC GFATM Round 7 Ph II 5,800,000 4,800,000 JHU/MC USAID /DFID LLIN VPP 33,785,000 11,500,000 33,785,000 19,900,000 PSI/JSI DELIVER/MC Warehouse & Distribution 3,000,000 SMP/MC Training 2,000,000 SMP/MC Micro planning & Social Mobilization 1,250,000 SMP/MC M&E 2,150,000 SMP/MC PMI/ DFID 6,000,000 6,000,000 MC USAID /PMI 3,450,000 3,450,000 MC PMI 1,200,000 1,200,000 SMP/MC World Vision LLIN 2,900,000 2,900,000 World Vision Source: [1] MoH, The GF (2011), Status Report on Global Fund Grants in Uganda. [2] MoUs (USAID/DFID & MC; MoH/MC), [3] RBM Gap Analysis – MoH Report This successful collaboration can provide a good lesson for global development agencies and country programs. The UC facilitated the establishment of a complex yet workable funding mechanism for channeling donor funds with a shorter turnaround time. This demonstrated a strong MoH and non-MoH partnership in Uganda. We recommend financial data quality audits to accurately ascertain the cost and expenditure during the UC and proper 26 documentation of the UC finance management processes and protocols, for future use/reference. 6.2. Timelines for campaign implementation The UC was first planned and budgeted for in Uganda’s Global Fund Round 7 plan in 2009. In 2012 the NMCP assisted by PMI initiated the pilot UC in four districts (figure 6). The rest of the UC was scheduled to run from 30th June 2013 to 31st March 2014, but was later extended until August 2014 for distribution in Kampala and Wakiso districts. 27 Figure 6. Universal Campaign Project Milestones For operational purposes, Uganda’s 112 districts were divided into clusters called “waves,” each comprising 15-16 districts. The UC Implementation Guidelines provided for a 24-day timeline to implement activities in each district. Each wave was handled separately to increase the feasibility of achieving total coverage. During the preparatory meetings, timelines were set to implement activities in each wave (Table 4); however, these timelines could not be adhered to after December 2013 because of procurement delays coupled with delays in release of nets from the National Drug Authority (NDA). Thus, waves 6-8 were delayed by some months, requiring an extension of the termination date from March to August 2014. There were occasional delays in transport of LLINs s from the central level to districts and to sub￾counties and in a few instances, delays in funding to districts caused minor delays in selected areas; however, the districts managed to overcome these problems without significantly altering timelines for completion of their campaign. What is important to note is that none of the planned distribution waves in the districts was delayed due to field implementation issues. From the UC weekly reports, we believe the campaign successes stem from careful planning, as well as testing of these plans in the pilot phase. The first five waves were all implemented successfully and within the timeframe allotted. District set-up/start-up activities were on time or even ahead of schedule. Training/sensitization, household registration, net allocation and distribution seem to have worked well, even as waves and the number of districts increased. Despite some few shortages/mix ups in LLIN allocations, the NCC and the district coordinators, in coordination with the district task forces, managed to cover shortfalls from excesses elsewhere, without apparent delays. Local solutions were found for most constraints, such as shortages in forms, irregularities in LLIN supplies, etc. Close supervision assisted in identifying and rectifying problems/issues early. Project Milestones 2009 Project first planned and budgeted for in Uganda’s Global Fund Round 7 plan. 2010 National Malaria Control Program distributed 7,200,000 nets to children and pregnant women. Household net ownership increased to 60% (up from 16% in 2006). 2012 National Malaria Control Program, assisted by PMI initiated pilot in four districts. Late 2012 – July 2014 Uganda imported LLINs, trained workers, registered recipients, and eventually distributed more than one net for every two people to the remotest corners of the country. 28 Table 4. Timelines for Campaign Implementation Wave Timeline planned Actual timeline Start End Completed Pilot: 4 districts 28 Aug. 2012 17 Sept. 2012 17 Sept. 2012 Wave 1: 2 districts May 2013 30 June 2013 30 June 2013 Wave 2: 16 districts October 2013 7 Oct 2013 9 Oct 2013 Wave 3: 16 districts 22 Sept 2013 2 Nov 2013 2 Nov 2013 Wave 4: 18 districts 6 Oct 2013 2 Dec 2013 1 Dec 2013 Wave 5: 15 districts 10 Nov 2013 6 Jan 2014 5 Jan 2014 Wave 6: 17 districts 3 Dec 2013 3 Feb 2014 1 June 2014 Wave 7: 17 districts 12 Jan 2014 3 Mar 2014 8 June 2014 Wave 8: 7 districts 9 Feb 2014 31 Mar 2014 August 2014 The UC proposed timelines were generally adhered to up to the fifth wave. Waves six – eight delayed by some months because of the procurement delays. The evaluation team recommends that for as long as Uganda continues to distribute nets through the UC , this should be done using the regional approach that focuses on ensuring that the waves are spread over a year rather than through a 3 month national campaign (as some have proposed). The evaluation has established that the procurement, importation and quality control of 23m nets cannot easily be compressed into a three-month window as this may be characterized by internal mobilization, procurement and distribution delays. 6.3. Efficiency in the process adopted in implementing the UC The pilot distribution of LLINs gave valuable lessons that were implemented in future waves, secondly the implementation of the campaign in waves allowed optimal use of resources and adoption of lessons learnt from the distribution of previous waves. The process evaluation did not carry out a detailed cost-benefit analysis, however a number of respondents did identify several areas where costs might be reduced to ensure effectiveness. They cited the BCC, which relied heavily on expensive mass media rather than interpersonal communication. There was limited time in terms of days that was spent when training the VHTs on how to pass malaria messages to the communities. It was also noted that innovation was not encouraged in use of traditional village-based communication channels. Some of this might have been difficult because of UC’s restricted time schedule, but benefits would have extended long beyond the campaign itself and would have facilitated sustainability and replication. The evaluation team recommends considering other alternative distribution channels in Uganda, as was in fact done during the abortive 2010 campaign. These might include 29 churches, mosques, women’s groups, schools, and so forth; however, mobilizing them would take significant effort and might not be appropriate on a national scale. Where workable, they would probably improve outreach, but perhaps at a higher cost. A third way in which efficiency might be improved, would be by delegating management responsibility and ownership to district and sub-county health teams rather than having all decisions made at the national level. In addition to potentially making more locally￾appropriate decisions, empowered local teams would reduce the human resource requirement for central supervision. Local capacity takes time to develop, however, and may be difficult to create overnight in a campaign environment. The evaluation team recommends the planning and implementation of the UC activities to begin earlier and should be extended to all areas of malaria control. District local governments operate integrated systems approach to management and implementation. This allows for other sectors to contribute to the UC, as evidenced by the Local government’s contribution to logistical needs of the campaign. The cost of remunerating a district and community leader is significantly lower than that of an ‘expert’ from the central level, who will additionally pay for services of guides and interpreters at the local level. 7.0. Universal Campaign Impacts 7.1. LLIN Ownership and Use There is preliminary evidence that showed marked increase in LLIN use in areas where the LLIN mass distribution campaign was implemented. Findings of the 2014 Lot Quality Assurance Surveys (LQAS) showed a general decline or stagnation in LLIN use among the districts where the UC was implemented in comparison with the 2012 results. LLIN use among children under 5 years ranged from 20% in Namayingo in East Central region to 96.5% for Busia in the East. In the four pilot UC districts, there was however a significant increase in ITN use among children 0-59 months in Busia (from 62.5% to 96.5%), Bugiri (from 35.8% to 80.5%), Kaliro (from 42.2% to 83.2%) and Mayuge (from 20% to 70%) (Figure 7). It is envisaged that the improvement of net use in these districts could be attributed to the mass distribution exercise since they benefitted from the UC during the pilot distribution phase. 30 Figure 7. Reported use of ITNs in four pilot UC districts. While the process evaluation was not able to establish the outcomes of the UC, the MIS 2014 should be able to document this in terms of net ownership and use and the measurement of changes in malaria prevalence. As a result of the UC it is anticipated that the findings of the 2014 MIS will document an increase in LLIN coverage and use, as well as declines in malaria prevalence in children under five years. Notably, it’s important to note that any reduction in malaria prevalence would not be quantifiably attributable to LLIN use since there are other factors, such as ACT and RDT use and indoor residual spraying, which might equally contribute to the reduction in the malaria prevalence. (In fact malaria transmission is highly cyclical and may even increase.) The evaluation team recommends the following • MIS should assess the universality of LLIN coverage by asking specific questions about access and use by all household members. (Current questions ask about each net but not about each resident.) • MIS should ask about recent (two weeks) incidence of fever for all household members, even though fevers have many causes and no comparison group or period is available, as an approximate way to judge if all demographic groups have benefited. • If possible, the waves of LLIN distribution should correspond to the regional blocks for the MIS sampling to identify any significant geographic differences in UC rollout which might influence planning for future campaigns. . 0 10 20 30 40 50 60 70 80 90 100 Busia Bugiri Kaliro Mayuge District 2012 2013 National target 31 7.2. Unintended Consequences of the Universal Campaign Among the unintended consequences of the UC include the strengthening of the district management teams. It is anticipated that the capacity built is sufficient to enable the district team handle a net distribution campaign and this was affirmed with the findings from the Key informant interviews. Among the negative unintended consequences of the UC the diversion of nets from the Antenatal care centers (ANC) to the targeted distribution centers. As a result, there was a setback in the allocation of nets at the ANC for an indefinite period because nets intended for continuous channels were diverted to be distributed during the UC however this diversion is now rectified. It was also noted that the free distribution reduced the potential private sector LLIN markets, at least in rural areas. The UC on the other hand, it might have stimulated the demand for specialized nets among those with adequate income. 8.0. Sustainability of the Universal Campaign in Uganda. Sustainability of the UC does NOT primarily depend on the financial aspects it includes other aspects such as the managerial and user dimensions. 8.1. Sustainability of Impact Among the health benefits the UC shall bring to Uganda can be maintained only if recipients maintained their nets and used them correctly and consistently. Notably, weakness of BCC noted during the UC could impact on correct and consistent use of LLINs. This weakness however can be addressed through effective interpersonal communication. International specialists increasingly believe that effective universal coverage for LLINs requires both mass campaigns and routine distribution, and that one without the other will fall short of universality17 . The approach by the MoH to undertake a UC and then routinely distribute LLINs to the most vulnerable populations (pregnant women and children) is in line with most recent thinking. Using both approaches has been found to be cost-effective in several studies in Kenya, Zambia and parts of West Africa18 . The NMCP estimated that 3 million LLINs would be required annually to maintain the achievements gained through its recent universal coverage campaign. PMI has already procured 3.9m nets for continuous distribution, of which 1.6m had arrived in-country by December 2014. Routine net distribution is planned through ANC clinics, primary schools, and immunization programs (EPI). The first and third will benefit from relatively high ANC and 17 Kilian A, Wijayanandana N, Ssekitoleeko J. Review of delivery strategies for insecticide treated mosquito nets-are we ready for the next phase of malaria control efforts? TropIKA.net Journal 2010; 1(1). 18 Chuma J. Okunga V, Molyneux C. Barriers to prompt and effective malaria treatment among the poorest population in Kenya. Malaria Journal 2010, 9: 144. 32 primary school attendance (the latter because of Uganda’s Universal Primary Education policy). Further opportunities may arise through private retail sales, social marketing and community organizations: community leaders and VHTs, for example, can identify coverage gaps and replacement needs, and provide vouchers to obtain free nets at health facilities. Although not yet fully in place, these strategies would and should suffice to maintain high coverage between mass distributions. 8.2. Managerial Sustainability Any future UC is likely to come with some challenges, however skills and motivation are likely to be stronger because of the 2013-14 experience and expertise accumulated/attained in the UC. The lessons learned as documented in this report will improve implementation of the next campaign. The MoH developed capacity in the NCC and its subcommittees, the district task force, the sub county task force, the VHTs and in the armed forces. The country now has a pool of trained and experience personnel who can be called upon to implement any future UC provided they remain district or government of Uganda staff including the VHTs. There was political support for the UC from the executive arm of the government, the parliament and the armed forces; it is important that this support is sustained for future campaigns. To a large extent, the role of the Lead implementing agencies was significant and should be maintained for future campaigns. The role, however should be reduced with time up to a time when MoH capacity is built to take on similar roles and responsibilities. The Efforts to reinforce continuous distribution channels, as described above, will also help to make UC less of a “stretch” next time and should be undertaken with an eye to building long-term capacity. The National Medical Store (NMS) and the Joint Medical Store (JMS) should play a role in procurement, storage and transportation of LLINs for both routine and future UC as part of capacity building process. Periodic LQAS surveys to monitor LLIN ownership and use should be conducted. Net durability studies should be conducted to monitor the continued efficacy to guide implementation strategies. The MoH needs to ensure that all LLIN distribution channels are supported by proven behavior change communication (BCC) activities focused at the local level to inform households of correct and consistent use of LLINs. The communities should be engaged to ensure that they know how to use, maintain and repair nets and this should be the focus of BCC messages. 8.3. Financial Sustainability A rough estimate of the funding required to undertake another mass distribution campaign in 2017 is between $118, 900,000 and $136,700,000 (including LLIN procurement and operational costs) without government in kind contribution. Table 5 shows the forecast population growth, LLIN needs and funding required for a UC in 2017, as projected for Uganda’s GFR10 proposal.19 While there are economies of scale in UC, the overall investment 19 These projections are illustrative and require updating based on UBOS2014 data. 33 of time, resources and money required to mobilize subsequent UC will be challenging to sustain over time, given the presence of competing health priorities. Challenges will be compounded by rapid population growth. Table 5. Forecast Need for LLINs (and Funding) to Support UC in 2017* Population by Year (Current and Forecast) Population Growth Rate Per UBOS Estimates Per Household Registration 2013 population: 35,357,000 40,660,550 2014 population: 3.40% 36,559,138 42,043,009 2015 population: 3.40% 37,802,149 43,472,471 2016 population: 3.40% 39,087,422 44,950,535 2017 population: 3.40% 40,416,394 46,478,853 Forecast LLIN need, 2017: 23,774,349 27,340,502 Forecast LLIN Budget, 2017: $97,474,833 $112,096,058 Other costs $21,396,914 $24,606,451 Forecast Campaign Budget, 2017: $118,871,747 $136,702,509 *Source GF round 10 proposal Barring unforeseen circumstances, it is anticipated that the willpower to achieve that is in place; however, requirements are significant and cannot be met solely through Ugandan resources. Donor commitment to malaria control in Africa and elsewhere remains strong; It is also envisaged that the Government of Uganda will continue to demonstrate political and managerial commitment. The GF10 proposal is based on a median estimate of 25,925,281 LLINs for the next UC. It is important to note that UBOS projections are based on the 2002 census, and may be updated after getting final results of the 2014 census. Finally, as MIS 2014 results are established and health economic benefits are documented, donors will meet with GOU officials as the next UC round is being organized to discuss GOU investments to a shared program. 9.0. Conclusions and Recommendations. 9.1. Conclusions The big picture conclusion is that the UC was implemented as planned, indeed the objective of providing at least one net for every 2 persons was achieved except in pilot, Kampala and Wakiso districts. The UC was relevant in Uganda to address the huge malaria burden. Malaria is endemic in 95% of Uganda, with approximately 90% of the population (an estimated 32 million people) at risk. Uganda hopes to achieve and sustain protection of at least 85% of the population at risk 34 by a combination of mass free distributions and continuous distributions of LLINs through multiple channels. Universal coverage of LLINs for all people at risk of malaria is relevant since it achieved community benefits and also met the expectations of the users as a malaria prevention. The UC was found to be cost-effective in several studies. The MoH worked in partnership with a lead agent to implement the UC, the lead agent had the capacity to manage the UC hence bridged the capacity gap in Uganda. A total of 22,267,777 LLINs nets were distributed by the end of the eighth wave (pilot plus waves 1-8), achieving estimated population coverage of 98% to 114% using population estimates from the household registration and UBOS forecast figures respectively and assuming 1.8 persons per net. Findings of the campaign indicate that the campaign was successful in as more than enough LLINs were distributed through the UC. This was achieved as defined, in all waves except the pilot and Kampala-Wakiso. Donor contributions to the UC totaled $105,040,000. 76% was spent on the procurement of nets and $25 million on distribution and advocacy support. Government in-kind contribution included provision of goods, services and human resource may not be accounted for. There was a successful collaboration between different donors, agencies and implementing partners and MoH and this was guided by agreements that were signed by all the partners. The pilot distribution of LLINs gave valuable lessons that might be implemented in future UC using the wave approach. Secondly the implementation of the campaign in waves allowed optimal use of resources and adoption of lessons learnt from the distribution of previous waves. There is evidence that there was increased use of the LLINs in the in the areas where the LLIN mass distribution campaign happened during the earlier waves as noted in the 2014 LQAS findings. The MIS of 2014 shall however be the best means of documenting the outcome of the UC, in terms of net ownership and use, and for measuring changes in malaria prevalence. A negative attibutte of the campaign is the diversion of the LLINs from the ANC to other UC distribution sites. This however is currently rectified. The free distribution of the LLINs reduced the potential private sector LLIN markets in rural areas. Health benefits the UC shall bring to Uganda can be maintained only if recipients maintain their nets and use them correctly and consistently. Effective universal coverage for LLINs requires both mass campaigns and routine distribution, and one without the other will fall short of universality. The MoH has built capacity in a number of structures and personnel including the NCC, the district task force, the sub county task force and the VHTs that can conduct future UCs. 9.2. Recommendations 35 1. Government should continue to lead the implementation of future UC with financial support from donors. The policy supporting another UC is in place; however there is need for an inbuilt accountability system. 2. Future campaigns should retain and strengthen inbuilt mechanisms for flexibility and adaptation such as conducting a pilot distribution, distribution through waves, and task forces that can make decisions. Regional waves should spread over a period that allows for the processing of results and analysis of implementation data to guide the next distribution process. There should be more flexibility and delegation of authority for local adaptation of plans at district and sub-county levels; 3. Ensure that all UC distribution channels are supported by proven IPC activities focused at the local level to inform households of the time and place of net distributions. BCC should equally focus on interpersonal communication to promote proper and consistent use of LLINs for impact. 4. Lead implementing agencies and malaria partners should continue to provide critical support for future UC as capacity is built within government systems. 5. Future campaigns should have a well built in process for data collection, transmission, management, availability, completeness, accuracy and timely use. 6. The roles of the implementing partners and the NMCP should change in tandem with increased authorities at the district level. Such roles should place more emphasis on guiding, mentoring, assisting, rather than implementing directly. The implementation guidelines should be revised to delegate such authorities and to allow for if not encourage flexibility and improvisation at the district level as needed to account for local realities. 7. MoH should consult/work hand in hand with UBOS in future UC population and household enumeration to avoid population discrepancies. 36 Appendix 1. Evaluation Scope of Work 1. Background Malaria remains Uganda’s number one killer, a serious public health challenge that has dealt severe socio-economic effects throughout Uganda. The British Government’s Department for International Development (DFID) and the United States Agency for International Development (USAID) signed a Memorandum of Understanding (MOU) on Dec 11, 2012 to provide joint support for the 2013–14 campaign (May 2013-expected May 2014) to achieve universal coverage of long lasting insecticide-treated nets (LLINs) to every household in Uganda to reduce the burden of malaria, henceforth referrred to as the Universal Campaign (UC). Evaluating the effectiveness of anti-malaria programs has become of considerable interest to host country governments, donors and implementing partners. The DFID-USAID MOU notes the need for an evaluation of the UC, and funds have been budgeted. This SOW is submitted to DFID in compliance with the terms of the MOU and strives to meet the requirements of both USAID’s and DFID’s Evaluation Policies. The Government of Uganda, The President’s Malaria Initiative (PMI), DFID, USAID and the Global Fund (GFATM) are financing an unprecedented campaign between October 2013 and March 2014 to distribute LLINs to every household in Uganda. This is the first time that Uganda has attempted universal coverage of LLINs, and the first time any country has attempted to roll out LLINs at such a scale, therefore understanding the impact of the program and learning lessons from it for future programs is essential. A Malaria Indicator Survey will run in summer 2014, and will document any shifts in outcomes and outputs since the last MIS in 2009. Lot Quality Assurance Sampling surveys (LQAS) is also in place in 61 key districts collecting data on uptake of nets and malaria prevalence rates. The UC involves the following activities that are implemented by USAID/DFID (through USAID’s Stop Malaria Project (SMP)), GFATM (through Malaria Consortium (MC)), and World Vision; coordinated by the National Coordination Committee, which is chaired by the Ministry of Health (MOH). 1. Procurement of bednets, storage, and positioning them at distribution-points in sub￾counties 2. Registration of households by Village Health Team workers, followed by allocation of bednets 3. Distribution of bednets to registered households 4. A data center to enter distribution data All the above activities are overseen by the District Health Teams and Ministry of Health’s National Malaria Control Program, with financial support and technical assistance from the campaign donors’ implementing partners (SMP and MC). 37 As stated in the DFID business case, Insecticide treated nets (ITN) are an effective tool for preventing the transmission of malariai . This is particularly the case since regular re-treatment of nets with insecticide has become unnecessary with the introduction of LLINsii . Recent guidelines from WHO on LLINs have shifted from a focus on vulnerable populations to a broader objective of universal coverage, defined as the use of ITN/LLINs by all household members regardless of age or genderiii . There has been considerable progress towards increasing access to LLINs across Uganda. The first targeted mass distribution of LLINs was carried out by the NMCP in 2010 to target all pregnant women and children with support from GFATM Round 7 Phase 1 Malaria grant, USAID and others. With support from GFATM, over 7 million LLINs were procured and successfully distributed to pregnant women and children in Central, Western, Northern and some parts of Eastern Uganda. USAID complemented this effort by supporting routine distribution of LLINs to pregnant women and children during ante-natal care and immunization visits in parts of Eastern Uganda not covered by the targeted mass-campaign. As a result of these efforts, considerable improvements in LLIN coverage have been achieved in Uganda. According to the 2011 Uganda Demographic Health Survey, the proportion of households with at least one LLIN has increased to 59% in 2011, from 47% in 2009. However, considering the fact that the effective lifespan of LLINs is around 3 years, Uganda needs to replace the LLINs distributed in early 2010 to pregnant women and children by the beginning of 2013. Expected Impact The impact of this support will be reduced malaria morbidity and mortality in Uganda. Based on the findings of a Cochrane review of the effectiveness of ITNs, this support will avert at least 30,000 child deaths over the next three years. The reduction in the burden of malaria will in turn yield considerable socio-economic benefits, for example increased educational performance, and reduced absenteeism from work and school, and savings in healthcare spending. Expected Outcomes The outcome of this support will be increased use of LLINs in Uganda. Five million LLINs will be distributed free through a mass distribution campaign and through routine distribution systems. As a result, approximately 1.8 million children under five years of age and 2.2 million people above five years of age will be effectively protected20 from malaria. Equity concerns are inherently addressed in this activity, as all households receive LLIN’s in this campaign, thereby benefitting households too poor to purchase LLINs The UC is taking place alongside other components of malaria prevention that include indoor residual spraying (IRS), Intermittent Preventative Treatment of Malaria in pregnant women 20 5 million LLINs will cover 7.5 million people (five million people through mass distribution of 2.5 million LLINs and 2.5 million pregnant women and children who will receive LLINs through routine channels), however based on current utilisation rates it is estimated that at least four million of these people will effectively use their LLINs . 38 (IPTp), and routine Ante Natal Care (ANC) bed net distribution implemented by USAID’s IRS-II project and SMP respectively. To date, SMP has delivered a total of 459,701 LLINs via ANC, IPTp coverage increased from 39 percent in 2008 to 55 percent in 2012, largely due to SMP technical support. 2. Purpose, Objectives and Questions of the Evaluation The purpose of this evaluation is two-pronged: to determine the performance and impact of the LLIN campaign to provide accountability for the use of resources, and to document lessons for future comprehensive campaigns in Uganda and worldwide. The evaluation will be conducted in two phases with Phase I focusing on a Process Evaluation in 2014 of the campaign implementation and Phase II focusing on an Impact Evaluation in 2015 on the results/impact of the campaign largely based on the findings of the MIS (to be initiated in June 2014 with a final report in March 2015) and follow-up in-country interviews and related data reviews. The objectives of each phase are as follows: Phase I  Understand the extent to which the universal LLIN distribution campaign has been implemented according to plan, and extract lessons relevant to implementation of future campaigns, broken down across the different components of the procurement and distribution plan.  Evaluate the management of the program, and whether the stakeholders funding and implementing it have designed and implemented it in such a way as to enable us draw lessons on learn from the experience and use the program to improve the level of data for decision making around malaria control and treatment in Uganda in general.  Review the design and instruments for the 2014 MIS to determine if there are any feasible changes that would strengthen the ability of the survey to draw conclusions about the impact of the UC and its different components (as well as the objectives of Phase II below), with changes limited to those that that do not threaten the comparability of the 2009 and 2014 MIS data. Phase II (Note: these objectives are subject to change and strengthening with the findings of Phase I)  Use the findings of the 2014 MIS to articulate the impact of the UC on malaria indicators, identifying needs for additional analyses as necessary  Identify and review all other reliable data on LLIN ownership, use, and impact, to see if it reaffirms or amplifies understanding of the outcomes and impact of the UC beyond the MIS. The overall evaluation will seek to answer, inter alia, the following questions, based on the standard OECD DAC criteria: 39  Relevance: Was the design and timing of the UC based on strong evidence? Was it appropriate and did it meet the expectations of the users?  Effectiveness: To what extent did the UC attain its objective of national coverage, defined as one net for every two people in Uganda? Was this objective achieved on time? To what extent was LLIN use increased by the campaign and how credible is the evidence? What lessons can be drawn from the process?  Efficiency: Was the most efficient process adopted in implementing the UC? Were most activities implemented in a cost-efficient manner Were there any, and if so what, alternative means could have been pursued by the program to accomplish the same outcomes at lower cost?  Impact: To what extent was LLIN ownership increased by the campaign? Will the program be able to detect any impacts as a result of its activities and, to date, has there been any measurable decline in malaria morbidity and mortality following the distributions that can be attributed to the program? Are there any important unintended consequences—either positive or negative—of the UC?  Sustainability: Is funding sufficient to carry out the UC every three years as planned? How likely is it that the GoU will be able to conduct a UC every three years and what additional inputs (e.g. funding, infrastructure, staffing, etc.) would be needed? To what extent can the LLIN distribution Infrastructure be maintained? What are the opportunity costs of doing so? Although there may be some overlap across the two phases, Phase I is expected to answer questions of relevance, effectiveness, efficiency and operational aspects of sustainability, while Phase II is expected to answer the question of impact (or the potential for impact) and the lasting effects of that impact. Evaluation Users DFID and USAID will be the primary recipients of the evaluation products. Other users will be the Government of Uganda and the Global Fund (GFATM) Scope, Approach and Methodology The Phase 1 Process Evaluation will focus on the performance of the Universal Campaign. It will be initiated during the final roll-out of LLINs in March 2014 to enable the evaluators to observe the distribution process first hand, while also drawing on administrative data from the previous waves of the roll-out. It is expected that Phase I will inform the design of the 2014 MIS to make it more useful in delineating the impact of the UC, and, if possible, its different components. The major components of the UC, the questions that the assessment will seek to answer, and the methodologies to be used are listed Table 7 at the end of this section. All documents pertinent to the UC will be reviewed, including  The UC work plan 40  The UC budget  Stop Malaria Program (SMP) reports relating to the UC  SMP (and sub-partners) quarterly and annual reports  Minutes, reports of and coordinated actions taken by Uganda’s National Coordination Committee (NCC) and sub-committees  Data related to UC monitoring, and LLIN coverage and usage Key Informant Interviews may include:  DFID Uganda health team  PMI/Uganda team  GOU staff, including national level (MOH, NMCP), district, and sub-county–level  SMP staff, including prime recipient and sub-recipients  ANC clinic staff  VHTs  beneficiaries  Global Fund Focal Coordination Office (FCO) and sub-recipient staff in both Phase I and II. The evaluation will include site visits to gain an understanding of how the UC operated in different contexts and at different levels. The evaluators will also observe distribution activities. Phase 2 Impact Evaluation will be carried out following the release of the MIS report in March 2015. The MIS should provide outcome and impact data on the coverage and use of the LLINs. Hence, Phase II will be a summative approach, drawing together the data and evidence from the Phase I process evaluation and the MIS into a final evaluation report covering all of the evaluation questions outlined above. Findings from Phase I may also influence the objectives and approach of Phase II. A detailed plan for Phase II will be developed following the completion of Phase I. It is likely that the key activities will be:  An in-depth review of the final MIS report  Interviews with the researchers and analysts for clarification of the data and findings  Interviews with people involved in various aspects of the UC to understand the reasons behind the changes or non-changes revealed in the data.  A review of other data on LLIN ownership, use, and impact, to see if other data expands understanding of the outcomes and impact of the UC beyond the MIS.  A presentation to USAID and DFID staff with an overview of the proposed final report of the Assessment Team and a draft report for review and comments by AID and DFID key staff for the final report.  Completion of the final report once the team returns to the U.S. 41 Evaluation Outputs 1. Periodic oral progress reports will be made to the QED Chief of Party 2. Detailed evaluation design and work plan for Phase 1: Based on the SOW, the evaluation team should prepare and submit a detailed evaluation design document and work plan specifying the data collections and analysis methodology and tools including proposals for sampling where relevant, plan for the management of the evaluation, and timing, as well as identify risks and challenges the evaluation may potentially face. 3. Draft evaluation report for Phase I that complies with the USAID and DFID evaluation standards (ca. 35 pages) by the end of April 2014 4. Revised final draft report for Phase I by mid-May 2014 5. Draft evaluation report for Phase II: Summative Evaluation Report by June 2015 (assuming the MIS report is available by March 2015) 6. Final report in line with USAID and DFID evaluation standards (ca. 50 pages, June 2015 7. Oral presentation: Power Point Presentation. The oral presentation should, at a minimum, cover the major findings, conclusions, recommendations, and key lessons. The evaluation team will liaise with the Mission to agree on the dates, audience, venue and other logistical arrangements for this briefing. The presentation will be held before departure, two weeks before the deliverable is due for each phase to enable feedback to be incorporated. 8. The contractor will share any data/information that is formally collected. No surveys or primary data collection are anticipated. Performance Period The evaluation is scheduled to begin in late March 2014 with field visits during actual net distribution and be completed by June 2015. The Phase 1 report will be completed in early May 2014 and the Phase 2 report in early June 2015. Calendars for the Phase 1 and 2 evaluations are at the end of this document. Governance and Responsibilities The Team Leader will be responsible for the organization of the team, scheduling of team activities, and the achievement of the deliverables. S/he will work under the direct supervision of the QED COP. Operational Assumptions and Potential Risks To be carried out successfully, the Evaluation Team assumes it will have:  Access to all key documents, budgets, and data related to the UC in a timely manner.  Support of the NMCP and other UC-related organizations in facilitating meetings with national, regional, and district level personnel who have participated in the UC 42 If at any time these risks appear to pose significant challenges to the success of the evaluation, the evaluation team will immediately coordinate with their immediate supervisors, who will inform USAID and DFID as appropriate. Staffing Requirements  Evaluation team will be composed of four consultants: one team leader, one malaria program expert, one senior Ugandan malaria specialist, and a field assistant.  The team leader will be a senior evaluation expert with over 15 years of experience evaluating and/or implementing health programs and with knowledge of and experience in malaria programs, preferably including LLIN programs. The team leader will have played substantive roles in more than five other evaluations and played the team leader role in a minimum three related evaluations.  The malaria program expert will have ten years of extensive malaria program experience in Sub-Saharan Africa and demonstrated skills in monitoring and evaluation of malaria programs, including publications in refereed journals. S/he will have played a substantive role in a minimum of three evaluations of health programs.  The Ugandan malaria specialist will have significant experience with Uganda malaria prevention programs, including LLINs, and have an in-depth understanding of the public health infrastructure at the national, regional, and district levels that supports the UC. S/he should know most of the key figures active in malaria prevention and be able to guide the Team through the process of identifying key interviewees, selecting sites for field visits, and arranging any government clearances needed for meeting with regional and district level officials. 43 LLIN campaign components, questions, & methodologies/data The team will review all the key components of the LLIN campaign to assess how well they functioned and to identify lessons learned from the experience. The team will attempt to find answers to the following questions regarding each component. Component Relevance, efficiency, sustainability Questions Methodologies/data 1. Initial Design  What were the key themes/orientation of the UC joint workplan?  What were the key assumptions underlying the joint workplan?  Who did the plan? Collaborating organizations?  Fits within NMCP plan?  Are the activities and outputs of the program consistent with the overall goal and the attainment of its objectives?  Total cost and sources of funding?  National, regional, district, village components?  M&E plan?  Sustainability thinking? What comes after this UC effort that will ensure the population remains covered? How will damaged LLINs be replaced? Will new LLINs be provided for newborns?  Were there any significant changes in the national plan because of new information or field experience? Document review: --UC Plan --UC budget --UC Design documents, contract agreements, modifications, etc.. Semi-structured interviews with 2-4 plan creators 2. Implementers  What were the respective roles of the major stakeholder and implementing organizations?  Was this the right mix of participants?  How well did the partners work together?  Who provided leadership of the UC? Consistent leadership and follow-up?  What was the structure of campaign management?  What was done to ensure consistent and quality leadership at each level? Document review: --UC Plan --Reports from Uganda National Coordination Committee (NCC) and meeting minutes --Malaria Consortium reports Semi-structured interviews with implementers: 10-20 key personnel from procurement through distribution, in various levels of system, to get different perspectives 3. Allocated Resources  Who funded each component?  Flow of funds to each level adequate? Timely?  Staffing from what organizations? Volunteer & paid? Document review: --UC Plan --UC budget Interviews: Include as part of semi￾structured interviews with appropriate implementers 4. Procurement of LLINs  Process of deciding numbers, types, and size of LLINs  Bidding process? Document review: --UC Plan --Reports from Uganda National Coordination Committee (NCC) 44 Component Relevance, efficiency, sustainability Questions Methodologies/data  Were recipient preferences taken into account?  Were the nets suitable for the target sleeping areas? Environment? --Quantitative campaign records --Post-distribution debriefing reports --Malaria Consortium weekly reports Include questions as part of semi￾structured interviews with implementers as appropriate 5. LLIN Purchase, Storage, and Distribution  What was the flow of LLINs from manufacturer to distribution point?  Safe storage with adequate security?  Timely delivery along the chain?  Any bottlenecks? Stockouts? Plan to deal with stock outs and ensure smooth flow of LLINs?  How were the recipients informed of what to do? Proof of eligibility required?  Any leakage and/or theft? If so, from which points in the supply chain?  How well did the various channels perform: ANC? EPI? Ugandan Military/Police?  Were the pilots using schools and commercial sector carried out?  How were they monitored & evaluated?  What was the reporting system from local (e.g. Parish, district, etc.) to national levels?  What were the initial results in each district in reaching its distribution target?  What were the major factors that facilitated reaching targets?  What were the major factors that hindered reaching targets?  Were there more efficient alternatives for reaching the objective? Document review of records and reports: --UC Plan --UC budget --Stop Malaria Program reports --Reports from Uganda National Coordination Committee (NCC) --Quantitative campaign records --Post-distribution debriefing reports --Malaria Consortium weekly reports Semi-structured interviews with implementers: 10-20 key personnel from procurement through distribution, in various levels of system 6. Recipients  Who was to receive them? Who made that decision?  How was eligibility for free LLIN determined? How did UC account for the varying ability of recipients to access distribution centers? Was there any special targeting of the highest risk groups (hard to reach/poorest)?  Was the campaign successful in reaching them?  Were the recipients pleased with the process?  Were they generally happy with the LLIN provided?  To what extent are recipients using the LLINs and what tools does UC have in place to detect LLIN usage?  Has there been any evidence of leakage/resale? Document review: --UC Plan --Stop Malaria Program reports --Reports from Uganda National Coordination Committee (NCC) --Quantitative campaign records --Post-distribution debriefing reports --Malaria Consortium weekly reports Data review: --LQAS data --SMP performance monitoring data Semi-structured interviews with implementers: 10-20 key personnel from procurement through 45 Component Relevance, efficiency, sustainability Questions Methodologies/data distribution, in various levels of system Group discussion with beneficiaries Market walks to see whether LLINs are being resold locally; informal discussions with vendors 7. BCC  Was there a BCC plan? Who created it?  Funding adequate? Number of channels used?  Start and end dates of BCC?  Balance of mobilization and mass media?  Monitoring of communication efforts?  Is there evidence of its effectiveness? Document review of records and reports: --Stop Malaria Program reports --Reports from Uganda National Coordination Committee (NCC) --Quantitative campaign records Interviews with implementers: include questions in semi-structured interviews with implementers as appropriate 8. Staff Preparation & Training  Who were the staff at various levels?  How were they briefed or trained?  To what extent did the campaign take them from their usual work? Document review of records and reports: --UC Plan --Stop Malaria Program reports --Reports from Uganda National Coordination Committee (NCC) --Quantitative campaign records Semi-structured interviews with implementers: 10-20 key personnel from procurement through distribution, in various levels of system 9. M&E process  Who created, implemented the M&E plan? What was its structure? Data gathering tools? Verification of reliability of data?  Rapid turnaround of data to allow for responding to problems?  Who did what? From where and to whom did data flow?  What problems were identified and how were they addressed?  Were adequate data collected to keep track of and provide feedback for implementation?  Is an adequate study in place to assess outcomes and impact? --Data already in place, e.g. HMIS --Data set up to track UC, such as LQAS data --National tracking: DHS, MIS 10. Field Implementation Did the structure for LLIN distribution function well? Were there any problems hindering LLIN distribution? Document review of records and reports: --UC Plan --UC budget 46 Component Relevance, efficiency, sustainability Questions Methodologies/data  At what points in the distribution chain did the most serious problems occur?  Was there a plan to reach people who were missed during the main distribution?  How well the different distribution did channels function? --Stop Malaria Program reports --Reports from Uganda National Coordination Committee (NCC) --Quantitative campaign records --Post-distribution debriefing reports --Malaria Consortium weekly reports Semi-structured interviews with implementers: 10-20 key personnel from procurement through distribution, in various levels of system Visits to ANC clinics to review process and interview staff Visits to schools to review process and interview staff and recipients Group discussions with beneficiaries 11. Debriefs of staff Were these carried out in all districts? Were they done with a formal structure that enabled the collection of insights and reactions from the participants? What were the most common comments from the field staff? What were the lessons learned from the Debriefs? Document review of debriefs Interviews with implementers who participated 12. Sustainability What is seen as the logical next step in LLIN coverage? Is there sufficient funding to repeat every 3 years as planned? What will it take to maintain the distribution Infrastructure? What are the opportunity costs and unintended consequences? Component Effectiveness, impact Questions Methodologies/data 1. Effectiveness: National coverage Did the campaign achieve national coverage, defined as one LLIN for every two people? To what extent are people using (sleeping under) LLIN? MIS data: compare 2009 and 2014 Other data that might be available 2. Impact: health outcomes What evidence exists that links the UC with health outcomes, particularly malaria morbidity and mortality? MIS Forthcoming evaluation of impact of malaria interventions on mortality 47 Appendix 2. Evaluation questionnaires LLINs universal campaign process evaluation (July 2014), Focus Group Discussion questionnaire A. Key Informant/In depth interviews District level questions Key Informants -District Health Officers / Resident District Commissioners/ LC 5 Evaluation Objectives 1. Participation in the UC Planning and implementation 2. ASM /BCC efforts for LLIN distribution and use 3. Efficiency and effectiveness of the UC 4. Recommendations for future UC activities Questions District level Planning  Please share with us how the UC ran in your district  Who were the collaborating organizations/departments in the UC planning effort?  Did the UC plan fit into the district health plan? Were you satisfied with the UC planning and management process?  In your opinion, did the UC plan have essential district and community components?  Where the M&E /implementation plan was availed; o In your opinion, was the UC was implemented according to plan? o If not, in what aspects do you think the actual UC implementation deviated from plan? o Do you think the deviation from work plan was beneficial to the UC?  How best should the next UC plan be approached in your district? /What do you suggest as the logical ‘Next Steps’ in LLIN UC?  Do you think you have sufficient resources to implement the next UC? Debriefs /Sensitization of district leaders about UC 1. Tell us about the district sensitization. How helpful was the sensitization to your management team? 2. Were these sensitization meetings conducted in a way that enabled collection of insights and reaction of participants? 3. What were the most common comments from the participants? 48 4. Based on what you learnt in the sensitization, do you think the implementation of the UC went as communicated? 5. How best do you think sensitization sessions can be improved in the next LLIN UC? District and sub district level planning and coordination of Advocacy, Communication and Social Mobilization (ASCM) activities for the LLINs UC Objective: Evaluation of effectiveness, efficiency and sustainability of ASCM approaches at the frontline 1. Tell us about the ASCM /BCC activities which your taskforce organized for the UC. 2. On what basis were these activities conducted (i.e. availability and utilization of national guidelines/standards) 3. Do you think there are opportunities / innovations realized in ASCM approach implemented in your district (i.e. efficient, value for money ideas)? 4. Do you think there was good balance between mobilization and social media campaigns for UC? 5. What plans do you have to monitor communication efforts? Is there evidence of the effectiveness of the plan you have? 6. What lesson would you recommend for us to retain and advocate for, in next UC? Training and support Supervision Objective: Evaluate efficiency, stakeholder engagement, innovations and record lessons learned for improved action 1. Regarding staff preparation and training, who were involved in your district? 2. How were they prepared /trained for UC? 3. To what extent could this training have affected /added value to their usual work? 4. Are there any new ways staff preparation /training was done? Was it effective? 5. What recommendations would you make for the next UC? Monitoring and Evaluation Objective: District participation in data quality improvement and data utilization for planning and implementation 1. Who created /participated in the creation and/or implementation of the M&E plan for the UC? 2. In your opinion, how rapid or effective was the turnaround of data to allow you to respond to problems in UC implementation? 3. Who did what in the UC implementation monitoring? From where and to whom did the data flow? 4. What problems were identified in the monitoring process and how were they solved? 5. Do you think adequate data was collected to keep track of, and provide feedback for better implementation? 49 B. Focus Group Discussion Guiding questions. COMMUNITY - Recipient of the LLINs Objective: Understanding Knowledge Attitude and Perception of the LLINs use and UC 1. Engagement questions a. We would like to know if bed net distribution took place in your community b. How do you feel about the bed nets you received? Exploratory questions (VHTs, Opinion leaders) c. Tell more about the registration and distribution of the bed nets. How did it go with you? d. In your opinion, do you feel the people appreciated LLINs UC? e. Could there have been households that never received LLINs? Why is that? f. I am wondering, are there any gender or equity concerns raised in your village over UC g. For equity, how best should the next UC plan be approached in your village? h. Please comment on the level of security and transparency for the most at risk people, at the LLINs distribution point? i. What lessons do you think are worth taking note of, from the UC implementation? j. What would you like to see changed in next UC? Exploratory questions (Community members…found at health center /organized by VHT) 1. Why do you think LLINs were worth distributing to families? 2. If you were to use a LLIN what do you feel is your major motivation? 3. How can your VHT or community leader support you to use the bed net effectively? 4. Do you have any problems with the LLINs you received? Exit question 1. Is there anything you would like us to say about why you would use or not use a LLIN on a regular/daily basis? Approach: Focus group discussions with engagement (minimum of 8), exploratory and exit questions. Exploratory questions will be supported with the following; “Can you talk about that more?” “Help me understand what you mean” “Can you give an example?” 50 51 Annex 3. Coordinating Structures. No. Coordination structure Roles 1 Health Development Partners  Providing technical support for gap analysis.  Financial support in procurement of commodities and for software activities  Supporting activities of the NMCP through participation in the RBM partnership meetings as defined by Government 2 National Coordinating Committee  Ensure engagement of national and district authorities and stakeholders through open and transparent communication;  Support districts with coordination between the MoH and the District Task Force  Consolidate district budgets from districts into a national budget and identify any gaps requiring national or international advocacy;  Centralize information and monitor LLIN procurement and arrival dates;  Coordinate and monitor activities of sub-committees against campaign timelines;  Support training of actors at all levels;  Track performance of districts and provide overall guidance on planning and implementation;  Resolve bottlenecks at central and, where necessary and possible, district level;  Advocate at all levels for engagement and support (international, national and lower levels);Supervise and monitor campaign implementation;  Provide the district coordination task force with budget ceilings and guidance on accounting and financial reporting 3 M&E/Operations Sub￾committee  Develop and review the implementation guidelines, including adaptation based on lessons learned as each wave of distribution takes place;  Ensure regular meetings are organized with minutes/action points circulated and validated;  Undertake macro quantification of LLIN and personnel needs for each district;  Develop training materials in collaboration with the other sub-committees;  Plan for training according to timelines of the activity chronogram;  Participate in training (or supervision of training) of all personnel involved in the implementation of activities, including supervisors for supervision visits;  Identify and resolve bottlenecks in implementation of activities;  Supervise and monitor activities, notably data collection/collation and by supervisors;  Write final distribution report, based on distribution point data, district reports and supervisors’ reports, including lessons learned for future improvements. 4 Logistics Sub-Committee (LSC)  Ensure regular meetings are organized, chaired by the NMCP logistician, and minutes and action points are circulated and validated by all members of the LSC;  Train MoH and district logistics focal points regarding use of appropriate tools for supply chain management;  iii) Monitor progress of activities against projected timelines;  Identify and resolve in-country supply chain bottlenecks;  v) Plan/implement storage and transport plans from Sub/County to distribution points;  Oversee finalization of logistics plans and budgets with MoH and districts;  vii) Collate all documents for the movement of LLINs through the supply chain and ensure proper filing;  Review and confirm effectiveness of supply chain management through Commodity Management Analysis;  Review LLINs allocation lists and post-distribution data;  Identify indicators for logistic activities to be used for monitoring and evaluation;  Write final logistics report, based on district distribution reports, including lessons learned for subsequent LLIN distribution campaigns.  Carry out post-distribution commodity audit to provide evidence of performance 52 No. Coordination structure Roles 5 Advocacy and social Mobilization Sub￾committee (ASMSC)  Design the campaign advocacy and communication plan for international, national and in-country partners and stakeholders. This includes developing talking points for the NCC to provide to MPs, district officials and other advocates.  Develop the advocacy and communication activity timeline and monitor implementation to ensure there are no delays;  Ensure regular meetings are organized and minutes and action points circulated and validated;  Engage partners and stakeholders at all levels;  Develop tools and supports for mass and interpersonal communication and ensure that these are pre-tested, reproduced and delivered to implementing districts on time;  Support development of advocacy and social mobilization elements of training materials that will be used at all levels for all actors;  Participate in training of actors at all levels;  Prepare documents and messages for any negative communication or rumors arising;  Produce and disseminate mass media spots (radio, television) according to advocacy and communication timeline;  Ensure planning and implementation of communication activities focused on increasing utilization;  Carry out orientation, supportive supervision and monitoring of activities;  Identify indicators for advocacy and communication activities to be used for monitoring and evaluation;  Support the process of literature review to improve the overall LLINs campaign;  Evaluate the validity and appropriateness of different key messages;  Write final communications report, based on district reports, including lessons learned for subsequent LLIN campaigns. 6 District task force  Ensure engagement of national and district authorities and stakeholders through open and transparent communication;  Translate the national micro-plan into district specific micro-plans taking into consideration the local factors like hard to reach and difficult to access areas (focus on logistics, transportation, monitoring and documentation);  Develop district budgets and identify any gaps requiring local advocacy;  Monitor LLINs receipt, storage, and transportation to sub-county level and subsequently to the distribution points;  Coordinate training of sub-county supervisors and VHTs;  Support registration at household level;  Guide the process of selecting the actual LLINs distribution points;  Resolve bottlenecks at district, sub-county and designated distribution points;  Oversee receipt of the LLINs.  Coordinate actual distribution of the LLINs;  Participate in social behavior change communication for LLINs;  Support documentation of best practices and compilation of the district specific process reports 7 The Sub County Task force  Training of the registration and distribution teams;  Receiving the hard copies of the completed household registers;  Reviewing the submitted registers for completeness and accuracy;  Submitting the registers to the DISO for safe storage before transportation to the center for management.  Receiving the LLINs in the Sub County stores;  Receiving complaints and concerns and resolving them accordingly;  Liaising with the DTF/lead agency as need may arise; The Uganda People’s Defense Forces and Uganda Police  Support security of the LLINs at the Sub-county stores, during transportation of the LLINs to and from the SC stores to the distribution points and at the distribution points.  Contribute vehicles for the transportation of LLINs to the distribution points. 53 No. Coordination structure Roles  Contribute personnel for distribution of LLINs to the beneficiaries at designated points. Appendix 4. Procurement process Process as planned: The procurement process began with quantification of need, followed by commitment of donor funds. Initial funding came from GFR7/Phase 2 ($51,194,127), but was later supplemented by PMI, DfID and World Vision. GF-financed procurement was planned through the Voluntary Pooled Procurement (VPP) mechanism; PMI/DfID procurement was to occur through USAID’s global commodity contractor (DELIVER). UC planning in 2012-13 assumed that all procurement would be complete by the end of 2013. Process as implemented: Two major changes occurred during implementation: estimates of need increased from 19,642,778 to 22,267,777 (15%), based on household registration data; and procurement was delayed in late 2013 because of global production delays. Commitment of donor funds: GFR7 committed $51,194,127 for LLIN procurement for the objective of universal coverage. Funds were transferred in two phases to the principal recipient, the Ministry of Finance; however, a two year funding delay required additional funding to replace the 7,295,850 Phase 1 nets which were now losing effectiveness. The need for new financing caused a significant gap, eventually covered by World Vision (506,600 nets, self￾financed) and DFID/PMI (6,199,600 nets). (The final Global Fund contribution was 15,004,636 nets.) PMI funds were diverted from allocations for continuous LLIN distribution; DFID funds were managed through PMI. Quantification of need: The number of nets required for UC turned out to be 15% higher than originally estimated, as described in the household registration section. In addition, GFR7 (Phase 1) nets intended for UC in 2010 were approaching the end of effective life by the time Phase 2 funds were released; thus, all had to be replaced. At the start of the second UC effort in late 2012, planners estimated that 19,642,778 nets would be required. This estimate of need continued to rise as natural population growth and household registration above UBOS projections raised estimations. Procurement documents: Both PMI and GF require competitive procurement, based on specifications which more than one potential supplier can satisfy. Net size, shape and color can normally be specified, but material (polyester or polyethylene) cannot. In other countries, LLIN users sometimes have strong preferences about color, shape, size and material; however, we are not aware of any consultation with future Ugandan users. The Global Fund confirmed and approved all bidding documents before they were sent to potential suppliers. Selection of suppliers: Using the VPP and the USAID DELIVER procurement mechanism, Uganda eventually procured about 22 million LLINs from suppliers in Vietnam, Tanzania, 54 China, Japan and India. Between 2009 and 2012, VPP had reduced the cost per net by approximately one-third; enabling the NMCP to purchase 15,004,636 LLINs with funds originally budgeted for 10,371,134. Ministry of Finance transfer of funds: This step, which may seem trivial, has significantly delayed other African programs. We found no evidence of delays in Uganda. Budget and expenditures: The team did not get information on procurement budget and expenditure. Its however important to note that cost savings achieved through VPP are unlikely to be repeated; hence the even greater importance of accurate forecasting and budgeting for the next round. Outputs: About 22 million nets were procured. While the factors leading to funding, procurement and importation delays, but also the unexpected cost reduction are unlikely to occur exactly this way again, the lesson is salutary: repetition of a UC campaign in 2016-17 will require a fortuitous confluence of both internal and external factors which must be closely managed and monitored. Uganda and its donor partners coped well with shifting needs and will have to be similarly flexible for future campaigns. 55 Appendix 5. Importation, storage and distribution of LLINs to sub-counties Process as planned: According to the LLIN UC implementation guidelines, Uganda would procure WHO pre-qualified LLINs through various funding sources. Kunhe + Nagel, the distribution agent of nets procured by the Global Fund would deliver the LLINs to sub county stores of each district. JSI/Deliver the distribution agent of nets procured by DFID/PMI would be responsible for transporting the nets up to central warehousing in Kampala. MC/SMP was responsible for delivering the LLINs to Sub County stores in some districts in northern Uganda. The sub county chief, under oversight of the sub-county task force would be responsible for managing distribution at the sub-county stores. Process as implemented: Storage and processing. JSI/Deliver; the procuring agent of PMI/DFID, imported 7,399,840 LLINs between May 2012 and July 2014. PSI, a VPP agent for the Global Fund, imported 15,004,636 LLINs from a number of sources including Sumimoto Chemicals, Cargo ex China, Vector Health International (Arusha), Vestergaard Frandsen (Vietnam), Disease Control Technologies (India) between February and August 2013. This massive quantity of LLINs might have overwhelmed a weaker supply management system; however, Uganda’s appears to have coped well, with staggered procurements and wave-based distribution. Most nets moved quickly from the port of entry to sub-counties and then to users. The procurement and distribution process is outlined below.  Receipt and customs clearance: The evaluation team did not receive importation data from the LLIN transporters. It was not clear who was the custodian of information on procurement and clearance. The team however saw copies of LLIN waybills, and goods receipt notes originating from sub-counties, at the Resource Centre.  Details of records kept, taxes paid, and timeliness of delivery: In line with the memorandum of understanding (MoU) between the donors and Ministry of Health, financial management was delegated to JHUCCP to MC under the SMP mechanism. The evaluation team did not review taxation records to answer this question as they were not availed to them by the time of the evaluation.  Bulk storage: The table below shows details of LLINs distributed by Spedag, malaria Consortium, Kuhne + Nagel (on behalf of PSI, the GF VPP agent) and by SMP/Malaria Consortium, on behalf of JSI-Deliver, the DFID-PMI procurement agent. Kuhne + Nagel received and distributed 10,984,240 LLINs, while Spedag working with MC received and distributed 4,015,734 LLINs financed by Global Fund. This evaluation could not map out the same process for DFID/PMIs LLINs per wave, as data was not readily available from Malaria Consortium and Spedag. The storage and distribution of LLINs is summarized in table 6. 56 Table 6. LLIN warehousing and distribution based on delivery notes UC Wave Storage, handling and distribution agent LLINs distributed GF PMI/DFID Wave 2 Kuhne+Nagel 2,085,040 Wave 3 Kuhne+Nagel 3,740,240 Wave 4 Kuhne+Nagel 2,893,760 Wave 5 Kuhne+Nagel 2,265,200 Wave 5 Spedag 552,899 Wave 6 Spedag Data not ready Wave 7 Spedag 2,685,960 Wave 8a Spedag 167,837 33,829 Wave 8b 575,209 1,291,791  Quality inspection: Ugandan law requires the National Drug Authority (NDA) to inspect all imported health-related products prior to use. This was a necessary step for GF-financed nets, of which 120 pieces/batch of 50,000 LLINs, totaling 36,000, were sampled for batch testing. The proposed batch testing was duplicative for PMI, however, because the DELIVER project inspects all commodities before they are shipped, based on internationally accepted standards. At the beginning of 2014 (midway in the campaign), a reported three million nets were in the country but not yet inspected because of equipment difficulties. These were eventually released (uninspected) in May 2014 on the premise that previous supplies from the same companies had met quality standards; however, the program promised to recall nets if quality defects were found later.  Transport to sub-counties: The evaluation team witnessed the process of LLIN delivery and handover at four sub-county stores in Arua district (wave 8). Trucks were monitored closely by the District internal Security officer (DISO) with the help of the MHSDMU teams. Following delivery, bales were counted to confirm that the number of LLIN bales allocated to each distribution point had been delivered. Behind this effort, was a detailed transportation and delivery plan prepared by MoH and its implementing agencies. LLIN shortfalls: According to data received from the MoH, a few LLIN bales were not received at the predestined sub counties. About 34 bales (wave 3), 32 bales (wave 4), 64 (wave 5 K+N), 83 (wave 8a) were reported as shortfalls. 57 Budget and expenditures: According to SMP, LLIN warehousing and transport had cost USD736, 747, as of October 2014. It is difficult for the evaluation team to ascertain the total cost, given that Malaria Consortium was still in the process of organizing its financial data by the time of the evaluation. Outputs: About 22 million LLINs were distributed to 112 districts and 8.5m households were registered. Recommendations:  Although the process seems to have been seamless, it would be interesting to know the process and cost of capitalizing the LLIN transportation plan, in order to way value for money output  The bidding, selection of transporters (Spedag, Kuhnel + Nagel) and MoUs involved, should be of interest, to provide lessons on NGO-Private for profit company partnerships, for future PFP plans or otherwise. 58 Appendix 6. Training and supervision process Process as planned: National/regional level training targeted central trainers/supervisors clustered for three areas of training and sensitization (30), registration (30) and supervision of LLINs distribution (30). The aim of this training was to prepare the central team (NMCP and Lead Agency) to be able to carry out training and supervision at district level and below. At the district level, training was planned for district executive, departmental heads, DHO, MFP, DHT, SC , opinion leaders, sub-county Community Development Officers (CDO), Health Assistant and major district religious leaders. Training aimed at sensitizing the district leadership on the LLIN distribution campaign in order to solicit their support and cooperation in the forthcoming activity as well as in the ongoing promotion of LLIN use. At the Sub￾County/Parish Level, training was planned for parish representatives (LC2 and parish chiefs), LC1 and women’s representative and CMDs / VHTs. Training was additionally planned for all persons involved in or responsible for the transportation; storage and distribution of the LLINs to familiarize them with the supply chain management tracking tools and for MoH and district staff on the use of standard tools for micro planning at the village level. Process as implemented: With oversight from the cluster supervisor, district supervisors initiated training and sensitization activities. A cluster of 15-16 districts comprised a wave. A district supervisor was entrusted with several districts in a cluster, depending on personal capabilities and experience. A district coordinator, who handled matters related to logistics and administration, supported each district supervisor. Training usually started on day 4 (earlier days being dedicated to sensitizing leaders, micro planning, electing taskforce members and mobilizing trainees). The first trainees included sub-county leaders, CDOs and Health Assistants. District supervisors, district health officers and malaria focal persons trained this cadre. Training sessions focused on broad knowledge of UC, household registration, distribution, ASM, data management and reporting. The trained team mobilized first line implementers for training on actual implementation on the 5th day. The last week would be dedicated to training VHTs and LCI, registration of households and allocation. District level performance in carrying out their responsibilities per the Implementation Plan was noteworthy: particularly the task forces and units below, and in their initiative and improvisation shown in overcoming problems. The support roles played by NMCP cluster groups through implementing partners showed effective collaboration and supervision, particularly in the sensitization phase of the UC. District rollout generally followed the UC central plan and schedule, although we heard numerous suggestions that decentralized Taskforces should have had greater authority to set schedules and particularly to modify BCC approaches within budget parameters. These modifications should be more feasible for the next round of universal coverage, especially if experienced personnel remain in place. 59 According to documented processes and KIIs with district and sub-county supervisors, as well as a district coordinator, this component of the UC was implemented more or less, as planned. Duration of training varied from wave to wave, depending on communication received from Malaria Consortium, but implementers noted that districts receiving PMI-CDC purchased LLINs, implemented according to days stipulated in the Training guide of the MoH, whereas GF supported districts spent 2-3 days less in training. Generally, an average of 12 -13 days were spent in each district. The numbers trained were eventually enormous, as shown by the Malaria Consortium analysis from Waves 4-8 (table 7). Table 7. Number of personnel trained. Category of Persons Trained/Sensitized Target Number trained Number Percentage District Leaders Sensitized 2,120 1,891 89% District Taskforce Trained 1,184 1,054 89% Sub-County Trainers/Supervisors Trained 1,818 1,823 100% Sub-County Taskforce Trained 2,639 2,478 94% Sub-county Leaders Sensitized 21,906 20,058 92% LC1 Chairpersons Sensitized 37,882 35,818 95% VHTs Trained 75,011 69,691 93% TOTALS 142,560 132,813 93% Average per District 1,926 1,795 93% 60 Appendix 7. Registration of households process Process as planned: According to the Detailed Implementation Guidelines, “each and every household will be registered. The Parish Chiefs will supervise the VHTs during registration and distribution. LC1 chairperson will verify the registration forms. The registration exercise will be implemented by trained VHTs and community resource persons where applicable.” Registration was supposed to be completed within 2-3 days per district, under the overall guidance of the sub-county supervisors. Each household was to be allocated one net for every two persons, with an extra net for households with an odd number of residents. Process as implemented: Registration was conducted by two VHTs in each village in collaboration with the LCs and supervised by sub-county supervisors. VHTs were provided with standard household registration forms that they used to register households and the final data submitted to LCs for verification. Upon verification of the household registration data, the VHTs submitted the forms to sub-county supervisors for LLIN allocation per household and village. The allocation was done manually using the UC allocation formula of one net per every two persons in a household. The final allocation data was entered into an excel template designed by the MOH M&E team which was able to check for errors in data such as average number of persons per household and expected children below five years of age. The district supervisors aggregated the village data into district summaries and submitted to the M&E focal person at the MOH to review the data before final allocation was done for each district. MOH sent this final data to the transporters to plan for movement of nets from the central warehouse in Kampala to sub-county stores. Nets were then transported from the sub-county stores to the distribution points. A detail of the registration exercise is outlined below.  Listing of households: Trained VHTs (generally two per village) went door to door, completing a separate form (in duplicate) for each household. Records showed names and demographic status (under 5, pregnant, other). No effort was made to count sleeping spaces nor to check the presence or quality of any existing nets (a challenging task, although one done in some other programs). VHTs often had to return to find persons away from home during the first visit. Key informants reported that two days was often not enough, mainly because of the difficulty of finding everyone; however, it appears that facilitation funds were only available for two days. It is certain that some households missed or refused registration; however, we have no way to estimate the numbers.  Validation: Registration was validated in two ways: by local leaders checking household and individual names based on personal familiarity, and by supervisors checking for demographic reasonableness. (Pregnant women should constitute roughly 4-5% of the population and children under 5 approximately 20%) Key informants reported that some registration data was adjusted at this stage, but the number affected has not been quantified. 61  LLIN allocation: Households with an even number of residents were allocated half as many nets (two persons per net); those with an odd number were given an additional net for the odd-numbered resident. In urban Wakiso and Kampala district, households were allocated less than the planned number due to a shortage of nets as discussed elsewhere.  Submission of registration data to the Resource Center: One copy of each registration form was retained locally to guide subsequent net distribution, while the other copy was taken to the Resource Center in Kampala. In July, the evaluation team found most registration forms in a large room and still in boxes because funds had not been released for data entry; however, by December we were told that forms had all been processed.  Micro-quantification: Registration forms were used to calculate the number of net bales (40 nets per bale) which were to be transported to each of the distribution points throughout the country. Previous calculations, used for budgeting and procurement, had been based on projections from the UBOS Census of 2002; however, household registration consistently yielded results that were 15 to 20% higher than UBOS projections, eventually necessitating procurement of 1.9million more nets than had been planned (at a cost of approximately USD8m). (We cannot assess this major discrepancy, only to note that the 2014 Census conducted after UC measured an even lower growth rate of 3.03 %.) Budget and expenditures for household registration and allocation: according to SMP, this activity cost USD$ 829,334 by October 2014. This excludes figures from MC (if any), since detailed financial reports were not accessible. Outputs: Preliminary figures indicate that 41,034,354 individuals were registered. Recommendations: The discrepancy between UBOS projections and registration data raised procurement costs by USD8m and resulted in diversion of PMI nets from continuous distribution channels to UC. World markets and donor cooperation overcame what might have been a major failure, but this leniency cannot be expected in 2017. 1. Ways should be explored to use lower cost methods to update household registration, perhaps using the national identity card registration system that is already in place. 2. We cannot comment on the accuracy of either UBOS or registration numbers, only to suggest that estimates be validated well before the next campaign to avoid last minute adjustments. Census and registration data should be rigorously compared at a micro level. Any proposed new methods of registration should also be validated. 3. NMCP should consider counting sleeping spaces, as is done in some other countries, since these numbers may be more useful than household size for allocating LLINs. 62 Appendix 8. LLIN distribution to households process (Waves 2-8a) Process as planned: Distribution was scheduled to take place at the village level manned by two uniformed officers (one UPDF, one Police) and one VHT member. The VHT member would oversee the tallying of the registers and verification of the beneficiaries while the uniformed officers will be responsible for security. The village ITNs distributors will be supervised by the sub-county task force. The LLINs would be delivered to each distribution point by 7:00am by the GISO supported by uniformed officers. Two methodologies were planned; fixed point distribution for peri-urban and rural settings and the door to door methodology in the urban settings if applicable. The internal health education system would be used to register data and distribute the nets. Together with partners, the MoH would carry out a post-distribution evaluation to ascertain the output and outcomes of this exercise’ Process as implemented: Most distribution points were situated on well-known public gathering places, for easy access by the communities although some villages had no nearby distribution points to access. LLIN distribution took varying periods of time at each distribution point, depending on prevailing weather conditions and the time of LLIN delivery from the sub counties to the distribution points. Further delays were encountered at some places when sub counties did not receive LLINs on scheduled dates. Another tier of distribution not previously planned for emerged. The VHTs and local leaders had to deliver LLINs to households, whose members were registered but failed to access the distribution points for a number of reasons. There was no transport arrangement for this unplanned but important activity. The VHTs walked to these homes, used bicycles, motorcycles, or mobilized other transport means with sub-county leaders. Some had to cross rivers to reach some households while others used boats to access islands. One official from the MHSDMU unit that monitors health matters in the office of the president echoed some of the challenge in the following statement. ‘There was no money to facilitate RDCs in their monitoring activities and for VHTs to transport bed nets to some islands. The official transporters carried nets and dumped bales at the landing sites, saying that their contract was to transport nets up to the landing site.’ The VHTs had to mobilize their own means to transport nets to various distribution points. The distribution exercise was scheduled to take place in one to two days per district. Distribution points were selected during the VHT training sessions and communicated to the users during household registration. According to the weekly reports from MC, there were a few complaints about long distances from recipient households to the distribution points. When this happened, VHTs and LC leaders sometimes created micro distribution points that were not there in the original plan. In a few districts and sub-counties, distribution was not possible in two days and had to be 63 extended for a third or fourth day to ensure that full coverage was achieved. Where this occurred, the program quickly increased its communication efforts to reduce anxiety and avoid missing households. Supervisory staff together with local leaders physically moved to the communities to communicate changes in distribution days. This was further backed by radio announcements and talk shows on local stations. The community members assembled to receive LLINs at their respective distribution points on the appointed days/dates. On the day of distribution, community members were however not adequately sensitized on proper net use and care before receiving their nets. Most VHTs were overwhelmed by the number of people who turned up and concentrated on tallying and distributing nets and paid little attention to sensitization. This issue feature prominently in a Focus group Discussion with a number of VHTs and LCI members in Ongako sub county, Gulu District who reported the following “Although we received training, we were not prepared for what would happen on the day of distribution. It was like a voting day; people were watching to ensure transparency, we spent a lot of time crosschecking names and faces, and keeping a watchful eye on the opened bales. Not all distribution points had security personnel. We had to plan quickly how to deliver nets to those who registered and had left earlier to do their chores, or had not turned up, yet an allowance of 5000 shillings was too small to facilitate all this.’ The community members acknowledge receipt of the LLINs by signing or thumb printing against their names in the village lists. In some sub-counties, the distribution took two days, because nets arrived late, and security personnel were not present in some centers. The distribution was supervised and overseen by a number of officials at different levels. After distributing LLINs, registration data was crosschecked for completeness by the village leadership and the VHTs and submitted to the sub county-supervisor. The sub-county supervisors ensured that all serving VHTs were paid their allowances and all documents fully completed before departure. A summary of the nets distributed is shown in table 8 Table 8. Distribution of LLINs and coverage statistics Region District Population Projection Registered population Popl variance (%) LLIN Projected need Actual LLIN need Actual LLIN Sent to District LLINs Distributed by District Projected Coverage Actual coverage Sept 12 Eastern Mayuge 477,700 563,515 18% 268371 316581 241,393 225,750 84% 64% Eastern Kaliro 216,500 225,650 4% 121629 126770 120,650 120,650 111% 86% Eastern Bugiri 447,200 440,060 -2% 251236 247225 175,903 175,900 79% 64% Eastern Serere 309,600 336,824 9% 173933 189227 129,590 129,590 84% 62% May 13 Eastern Soroti 339,300 355,986 5% 190618 199992 213,877 213,877 101% 96% Eastern Busia 306,000 373,284 22% 171910 209710 208,139 208,139 109% 89% Sept 13 Eastern Sironko 245,700 331,297 35% 138034 186122 181,730 180,402 117% 87% Eastern Kaberamaido 207,700 243,014 17% 116685 136525 139,000 136,570 105% 90% Eastern Bukedea 194,400 211,606 9% 109213 118880 129,200 129,182 106% 98% 64 Region District Population Projection Registered population Popl variance (%) LLIN Projected need Actual LLIN need Actual LLIN Sent to District LLINs Distributed by District Projected Coverage Actual coverage Eastern Bukwo 76,300 97,919 28% 42865 55011 56,391 54,600 114% 89% Eastern Bulambuli 128,600 195,480 52% 72247 109820 126,360 110,428 137% 90% Eastern Kapchorwa 119,300 107,818 -10% 67022 60572 67,738 65,888 88% 98% Oct 13 Eastern Katakwi 184,000 186,148 1% 103371 104578 115,080 113,000 98% 97% Eastern Kumi 267,000 299,078 12% 150000 168021 169,640 159,669 96% 85% Eastern Kween 107,728 117,782 9% 60521 66170 64,457 64,630 96% 88% Eastern Manafwa 380,000 469,532 24% 213483 263782 271,040 266,406 112% 91% Eastern Nakapiripirit 171,100 187,501 10% 96124 105338 125,320 101,259 95% 86% Eastern Ngora 164,400 170,689 4% 92360 95893 96,991 96,081 94% 90% Eastern Pallisa 357,400 458,478 28% 200787 257572 265,178 260,092 116% 91% Eastern Amuria 441,200 316,956 -28% 247865 178065 180,567 181,355 66% 92% Eastern Bududa 187,600 181,886 -3% 105393 102183 102,040 99,494 85% 88% Eastern Amudat 120,157 118,533 -1% 67504 66592 63,360 59,925 80% 81% Nov 2013 Eastern Mbale 453,900 618,539 36% 255000 347494 346,520 345,878 122% 89% Eastern Budaka 183,700 246,237 34% 103202 138335 139,554 139,553 122% 91% Eastern Kibuku 188,000 261,012 39% 105618 146636 156,634 135,895 116% 83% Eastern Namutumba 224,800 298,680 33% 126292 167798 166,680 166,111 118% 89% Eastern Tororo 500,300 601,955 20% 281067 338177 328,570 339,907 109% 90% Eastern Namayingo 243,700 230,394 -5% 136910 129435 142,693 145,773 96% 101% Eastern Luuka 269,800 296,664 10% 151573 166665 166,760 166,914 99% 90% Eastern Iganga 517,000 616,089 19% 290449 346117 348,619 345,120 107% 90% Eastern Buyende 273,900 358,163 31% 153876 201215 198,280 197,480 115% 88% Eastern Jinja 514,300 462,639 -10% 288933 259910 301,080 301,107 94% 104% Eastern Kamuli 517,400 554,156 7% 290674 311324 306,720 306,715 95% 89% Eastern Buikwe 441,100 491,068 11% 247809 275881 283,035 276,370 100% 90% Eastern Buvuma 56,800 131,055 131% 31910 73626 74,152 75,117 212% 92% Eastern Mukono 565,700 624,816 10% 317809 351020 374,917 368,309 104% 94% Eastern Kayunga 365,700 439,123 20% 205449 246698 241,837 243,187 106% 89% Eastern Butaleja 228,800 293,695 28% 128539 164997 163,398 163,920 115% 89% Dec - 2013 Central Rakai 493,000 581,238 18% 276966 326538 334,288 325,200 106% 90% Central Lwengo 269,900 314,157 16% 151629 176493 175,842 176,786 105% 90% Central Kiboga 172,100 161,940 -6% 96685 90978 98,520 96,685 90% 96% Central Kyankwanzi 190,800 254,747 34% 107191 143116 142,920 142,400 119% 89% Central Mityana 316,500 357,860 13% 177809 201045 204,000 203,921 103% 91% Central Luwero 451,500 516,352 14% 253652 290085 289,894 289,894 103% 90% Central Mpigi 218,300 269,200 23% 122640 151236 151,840 150,816 111% 90% Central Sembabule 223,900 276,978 24% 125787 155606 153,234 156,200 112% 90% Central Nakasongola 159,800 204,142 28% 89775 114687 114,560 113,832 114% 89% Central Gomba 155,400 173,307 12% 87303 97363 96,607 96,720 100% 89% Central Lyantonde 86,326 114,857 33% 48498 64526 66,320 64,085 119% 89% Central Mubende 633,400 763,736 21% 355843 429065 420,171 421,714 107% 88% Central Kalungu 178,800 198,085 11% 100449 111284 112,120 111,977 100% 90% Central Masaka 254,100 281,596 11% 142753 158200 176,680 172,315 109% 98% Central Nakaseke 200,058 228,392 14% 112392 128310 128,028 126,004 101% 88% Central Butambala 100,900 114,203 13% 56685 64159 63,840 63,700 101% 89% Central Kalangala 70,800 67,867 -4% 39775 38128 39,986 39,866 90% 94% Central Bukomansimbi 155,400 163,856 5% 87303 92054 91,801 91,801 95% 90% Jan 2014 West Rubirizi 126,900 155,927 23% 71292 87599 86,530 86,570 109% 89% West Bushenyi 256,500 289,435 13% 144101 162604 161,464 161,622 101% 89% West Mitooma 200,500 236,965 18% 112640 133126 131,350 131,320 105% 89% West Sheema 224,400 254,405 13% 126067 142924 140,671 140,862 100% 89% West Mbarara 454,800 555,928 22% 255506 312319 312,801 312,804 110% 90% West Buhweju 103,200 134,505 30% 57978 75565 74,680 74,427 115% 89% West Isingiro 432,100 499,323 16% 242753 280519 277,494 288,074 107% 92% West Kabarole 421,700 556,106 32% 236910 312419 312,280 312,249 118% 90% West Bundibujyo 275,100 322,226 17% 154551 181026 181,520 181,367 105% 90% 65 Region District Population Projection Registered population Popl variance (%) LLIN Projected need Actual LLIN need Actual LLIN Sent to District LLINs Distributed by District Projected Coverage Actual coverage West Ntoroko 88,400 94,214 7% 49663 52929 52,400 52,338 95% 89% West Kyegegwa 165,800 307,756 86% 93146 172897 164,841 164,841 159% 86% West Kyenjojo 397,700 475,901 20% 223427 267360 268,720 265,445 107% 89% West Kamwenge 339,500 467,503 38% 190730 262642 264,047 263,988 124% 90% West Kiruhura 311,300 396,188 27% 174888 222578 220,608 219,387 113% 89% West Ibanda 261,900 287,064 10% 147135 161272 160,585 160,489 98% 89% Jun 2014 West Buliisa 82,800 150,986 82% 46517 84824 83,795 83,768 162% 89% West Kanungu 257,200 287,482 12% 144494 161507 160,160 160,677 100% 89% West Rukungiri 326,000 408,028 25% 183146 229229 227,920 228,320 112% 90% West Kabale 502,100 615,444 23% 282079 345755 346,280 345,809 110% 90% West Kisoro 257,800 307,739 19% 144831 172887 171,560 171,208 106% 89% West Ntungamo 491,200 586,707 19% 275955 329611 327,080 326,084 106% 89% West Kasese 774,800 901,890 16% 435281 506680 493,744 493,500 102% 88% West Hoima 575,100 699,295 22% 323090 392862 387,240 386,298 107% 88% West Masindi 371,600 304,290 -18% 208764 170949 179,080 178,813 77% 94% West Kiryandongo 334,500 294,119 -12% 187921 165235 163,107 163,107 78% 89% West Kibaale 717,500 862,832 20% 403090 484737 474,960 473,463 106% 88% North Apac 360,500 443,637 23% 202528 249234 248,960 248,823 110% 90% North Oyam 391,900 471,916 20% 220169 265121 264,040 262,858 107% 89% North Kole 239,600 271,566 13% 134607 152565 150,200 150,951 101% 89% North Amolatar 130,900 179,945 37% 73539 101093 111,600 91,518 112% 81% North Lira 416,100 540,272 30% 233764 303524 299,360 298,817 115% 88% North Dokolo 189,700 223,153 18% 106573 125367 121,800 123,943 105% 89% North East Kaabong 422,300 396,465 -6% 237247 222733 217,120 216,803 82% 87% North East Kotido 248,900 256,280 3% 139831 143978 138,160 137,652 88% 86% North East Moroto 143,800 131,842 -8% 80787 74069 73,627 73,627 82% 89% North East Napak 209,100 233,805 12% 117472 131351 128,908 128,908 99% 88% North East Abim 57,200 185,144 224% 32135 104013 102,120 102,042 285% 88% North Otuke 88,800 127,051 43% 49888 71377 73,040 72,356 130% 91% North Kitgum 257,600 251,640 -2% 144719 141371 146,000 145,319 90% 92% North Gulu 407,500 564,939 39% 228933 317381 315,400 313,801 123% 89% North Alebtong 233,400 206,876 -11% 131124 116222 170,200 154,190 106% 119% North Agago 314,700 281,531 -11% 176798 158163 154,680 154,410 79% 88% North Nwoya 314,700 161,934 -49% 176798 90974 89,719 89,589 46% 89% North Lamwo 178,100 167,293 -6% 100056 93985 95,800 95,461 86% 91% North Pader 243,200 220,830 -9% 136629 124062 123,640 123,588 81% 90% North Amuru 183,600 335,013 82% 103146 188210 187,360 185,760 162% 89% West Nile Nebbi 355,100 535,850 51% 199494 301039 306,560 306,373 138% 91% West Nile Zombo 225,300 269,009 19% 126573 151129 150,760 150,679 107% 90% West Nile Maracha 205,600 229,840 12% 115506 129124 133,625 133,598 104% 93% West Nile Koboko 251,800 245,150 -3% 141461 137725 133,625 133,588 85% 87% West Nile Yumbe 589,500 379,465 -36% 331180 213183 206,205 209,882 57% 88% West Nile Moyo 444,700 161,041 -64% 249831 90472 92,160 92,062 33% 91% West Nile Adjumani 399,700 171,808 -57% 224551 96521 114,840 114,840 46% 107% West Nile Arua 801,400 989,423 23% 450225 555856 548,600 548,505 110% 89% Aug 2014 Mulago - Kampala 51,245 28789 28,880 28,668 90% Central Kampala 1,788,600 2,085,833 17% 1004831 1171816 774,847 775,059 69% 59% Central Wakiso Urban 1,429,500 849,334 -41% 803090 477154 334,043 340,723 38% 64% Wakiso Rural 1,407,132 790524 729,075 722,395 82% Source: Malaria Consortium and MoH, implementation plans, registration summaries and procurement data. The uniformed forces ensured security at the storage points, during transportation and at the distribution points. The Government made separate arrangements to facilitate the uniformed forces, although there were reports that some of the facilitation delayed and allowances were paid after the UC. This evaluation team could not confirm these reports, as the data was in the custody of the Ministry of Finance. 66 The budget for distribution of nets to households is summarized on table 9 67 Table 9. Budget for distribution of nets to households Objectives / Service Delivery Areas Budget ($) Expenditure ($) 1. GF Budget to MC Distribution of Nets to Beneficiaries 176,924 209,155 Distribution mop up to Households 88,462 78,929 Retrieval of Distribution registers. 47,937 40,734 313,323 328,818 2. PMI/DFID budget to SMP Distribution of nets to beneficiaries, distribution mop up to Households and retrieval of distribution registers. 661,903 736,747 Total 975,226 1,065,565 68 Appendix 9. Social mobilization process Process as planned: The National Communication Plan for UC, dated July 2013, was developed by the NMCP in collaboration with PMI, MHSDMU, Stop Malaria, the Malaria Consortium, and other NGOs. The goal of the Communications Plan was to “increase ownership and correct use of LLINs, hence contribute to the overall reduction of morbidity and mortality due to malaria in Uganda”. IEC/BCC and advocacy activities would be implemented at all levels to maximize participation in campaign activities and use of LLINs once they are received and hung. At central level, advocacy meetings will be held with various stakeholders, including Parliament, to ensure their commitment and participation in the campaign and for malaria control and prevention, in general. Similar activities will be done at district level. Radio talk shows and messaging will be done locally to alert communities of the upcoming campaign activities and to promote net use and other malaria control and prevention behaviors. At national level, a launch for the mass LLINs distribution campaign was planned led by the executive arm of Government supported by Top Management of the Ministry of Health. Related activities will include, press release on the mass LLINs campaign, press conference, messages on TV and radio stations that have national coverage and messages on TV and radio stations that have local coverage and appeals. District level activities will include district level mass media campaigns, demonstrations on the use of LLINs at the distribution points, interpersonal communication, distribution of Information Education and Communication (IEC) materials and the promotion of LLIN use among primary school children Process as implemented: This plan was not fully implemented, in part because of disagreements among the implementing partners as well as lack of funds. It appears that interpersonal communication (IPC) at LLIN distribution points was particularly weak. At the national level, the campaign involved mainly radio/ TV talk shows, radio spots, newspaper pullouts, press briefs by high ranking personnel, National/regional Advocacy meetings, printing of all IEC materials (T-shirts, Banners and Posters) and radio announcements. The campaign was launched in Soroti by his Excellency Yoweri Kaguta Museveni, President of the Republic of Uganda. Masses were further mobilized through use of two brass bands (UPDF and civilian) at the UC launch accompanied by famous local musicians. District level activities were mainly radio talk shows with district leaders as guests, radio announcements, banners and district level sensitization meetings. At the Sub-county level there were sensitization meetings, road shows, display of banners and posters. At the village level there was sensitization of LCs, house to house mobilization by the VHTs, distribution of posters, and use of local radio announcements 69 A mix of advocacy, community mobilization and BCC activities (including interpersonal and educational activities as well as mass media) were employed. Key messages had two primary objectives: to mobilize people to register, collect, use and care for the UC LLINs, and to improve knowledge about malaria transmission, prevention and management. Messages were directed at promoting access to and regular use of LLINs, rather than raising the level of awareness of malaria causes. Messages attempted to correct common beliefs that malaria was low risk and could be treated by drugs. These beliefs were supported by resigned attitudes that malaria has always been with us and is part of our reality. IPC channels also provided practical information on net hanging, proper care and maintenance. To reach the primary audience, BCC efforts included a mix of mass media including: radio, TV, posters, leaflets, phone messaging, hotlines and outdoor advertising. Interpersonal communications and mobilization activities included community and household visits by VHTs and volunteers to mobilize people to take part in the campaign and provide basic education and information about malaria. Secondary and tertiary audiences were reached through a smaller mix of sensitization and interpersonal meetings, and mass media efforts (such as print materials and newspaper and other reports and circulars as well as phone and TV campaigns). The implementation of the BCC activities was patchy. The MoH representative on the operations subcommittee echoed this sentiment, ‘The original plan for the UC was mass media and not IPC. In my opinion, what affected this component of the UC was to take the entire BCC program from MoH to SMP. From that point, BCC lost coordination and management efforts by the MoH.’ A focused group discussion with about 10 members of the MHSDMU echoed the same sentiment: ‘Malaria Consortium and Stop Malaria took the lead on BCC activities instead of the MoH, leading to lack of oversight from MoH and the patchy implementation of BCC. We recommend in-country public systems and mechanisms to be fully utilized.’ During in-depth discussions with individuals at the national MoH, discussants felt that the use of mass media based mobilization was not adequately planned and implemented. They cited several challenges such as underestimating the need for BCC in a mass LLIN campaign; lack of communication and training for LLIN distributors; inconsistent and untimely messaging; lack of a balance between mass mobilization and IPC; and failure to engage NGOs, political and district leaders in BCC implementation. A unique opinion portraying lack of in-depth planning, was failure to consider BCC messages on the usefulness of LLINs in IRS districts, A key informant had this to say ‘IRS districts did not have relevant and sufficient BCC messaging. LLINs working as a barrier to nuisance mosquitoes should have been a good message. I think the problem was failure to 70 engage community-based organizations. It was also a wrong idea to expect SMP and Malaria Consortium to understand the entire country’s communication needs.’ In all the seven districts the team visited, many of the challenges and fears communicated at national level were felt. Two community educators working with Mugusu sub-county, Kabarole district said, ‘People came asking for free bed nets, but we were not aware of such a program, whether it was true, when it would be and where it would take place. We told them there is no such program in our sub county. The next day, on a Saturday, we received abrupt calls from Kampala to come and distribute bed nets. This was very embarrassing. ’ The leadership from Pece Division, Gulu were in agreement and stated that, ‘We heard radio messages but we felt that they were irrelevant to the actual BCC need in our division. They should have let us design our own music, drama, with local opinion leaders, women and people living with HIV/AIDS groups. A common opinion arising from 26 men and women the team interacted with at Mugusu HC III in Kabarole district, points to limited BCC and IPC in this sub-county; ‘We received the bed nets, but we were not taught how often to wash them, use them and sleep in them. We don’t know how long these bed nets take before ageing and how to keep bed nets from developing holes and where to put the old ones’. Funds committed for BCC activities were not fully spent on planned activities. About US $ 1,459,658 was committed for BCC activities including development and printing of IEC materials and tools. A total of US $ 1,149,113 was disbursed to MC for BCC in 43 districts, of which only US $ 752,760 was spent. 71 Appendix 10. Information for decision making process Process as planned: Monitoring and evaluation of the LLIN campaign was to be based on data from the District Health Information System 2 (DHIS2) and surveillance by the Medicines and Health Services Delivery Monitoring Unit (MHSDMU). The existing mTrac systems would be used to provide weekly HMIS surveillance reports. Regular monitoring reports were to be submitted to the NCC. Evaluation of the outcome and early impact of the campaign was planned in three parts: through a post-distribution evaluation; the Annual Health Sector Performance report (AHSPR) and the Malaria Indicator Survey (MIS) 2014. Five major reports were to be produced once the campaign was completed: an Overall National Mass Campaign Report; a Logistics Report; a BCC Report; an Operations/M&E Report; and a Financial Management Report. Process as implemented: The planning and implementation of the UC was based on data to a large degree, although the requisite information was not always available in a timely manner. Population projections based on UBOS Census from 2002 were used to estimate the needs for universal coverage. The allocation of nets and distribution was guided by household registration data, and real-time reports on commodity movements, from warehouses to sub￾counties, to distribution points and homes. Registration data was to move upwards for aggregation in the Resource Center; however, when the evaluation team visited near UC wrap￾up in July 2014, data compilation had been delayed because funds had not been released for data entry. The logistics sub-committee lacked procedures for transparent reporting on commodity movements. The overall progress of district implementation was tracked in several formal and informal ways and periodically reported to the NCC. The chair of the M&E/Operations sub-committee also oversaw district supervision clusters and thus had first-hand knowledge of progress and problems. In addition, the Malaria Consortium prepared weekly reports, summarizing progress to date. Information for local planning: Household information was key to the entire UC and was recorded on registration forms (one per household). These were completed in duplicate, with an original sent to the Resource Center for LLIN allocations and a copy retained locally to guide and record receipt of nets. Household data were verified village by village, then aggregated at the parish and sub-county levels. As noted in the section on household registration, the numbers generated were 15% higher than expected (and may still have missed some intended beneficiaries); however, we heard no complaints about the overall process. 72 Supervision tools: Several tools were developed to guide supervisors in overall quality assurance, among them a form for confirming that VHTs had actually visited households, another for quality assurance of training, and another for overseeing local distribution. These forms reflected efforts to use data for improving quality; the form for distribution, for example, asked how far recipients had walked and if they were shown how to hang nets. There was no overall form for assessing BCC. All supervision forms, while well intentioned and designed, were unfortunately underused during implementation. Activity reporting: As lead implementing agency, Malaria Consortium submitted weekly narrative activity reports describing field activities. These did not follow a consistent format and only occasionally provided summaries of activities to date. Both MC and the Resource Center reported regularly to the National Coordinating Committee. LLIN tracking: An essential but largely lacking monitoring system for this campaign should have been for commodity tracking, starting with development of procurement specifications, through shipment and importation, to NDA clearance, to delivery and distribution within each district. While implementers seemed to have had a general idea of procurement status, warehouse quantities, and field distribution, details were not reported in the transparent fashion required for efficient management. Weekly stock and flow reports posted on the internet, would have allowed for quicker resolution of problems. Resource Center: The MoH Resource Center in Kampala played a vital role in information management but was far too under-resourced to enter and interpret data in a timely fashion and to promote use of data for decision-making. When the evaluation team visited the Resource Center in July, more than a hundred boxes of district forms sat unprocessed in a large room, awaiting funds for data entry; these forms were eventually entered once funds were released. Routine M&E: The National Malaria Control Program’s M&E plan, developed in 2012, includes the following indicators relevant to LLIN ownership and use:  Proportion of households with at least one ITN  Proportion of households with at least two ITNs  Proportion of households reaching universal coverage with ITNs (one net/two people)  Proportion of children under 5 years old who slept under an ITN last night  Number of ITNs distributed, by target group  Number of ITNs sold in the commercial sector  Number of distributors trained in ITN distribution  Number of ITNs procured for free distribution  Number of ITN guideline books distributed  Number of ITNs distributed through routine keep-up distribution Data generated during universal coverage could in theory respond to indicators 5, 7 and 8; however, the HMIS database is considered too weak for analysis. 73 Malaria indicator survey (MIS): The MIS planned for late 2014 will provide household-level data to measure other indicators listed above; however, results will become available too late for this Phase 1 evaluation. Recommendations: 1. Commodity and logistics reporting should be made fully transparent and functional well before the next UC. 2. The team strongly endorses MIS 2014 as the best means of documenting the effect of universal coverage, although it will not be possible to quantify the effect precisely. We expect to see major increases in LLIN coverage and use, as well as declines in malaria prevalence for under-fives; however, the effect of UC on prevalence cannot be attributed precisely because of likely confounding factors (especially increased availability of ACTs, indoor residual spraying and improved case management). We do suggest minor changes in MIS tools and possibly in sampling frames, however, to increase the MIS’ utility for future campaigns: • MIS should assess the universality of LLIN coverage by asking specific questions about access and use by all household members. (Current questions enumerate and ask about each net but not about each resident.) • MIS should ask about recent (two weeks) incidence of fever for anyone in the household, even though results are soft (fevers have many causes and no comparison group or period is available), as an approximate way to judge if all demographic groups have benefited. • If possible, the sample should be stratified by the timing of LLIN distribution (early or late) to identify any significant geographic differences in UC rollout which might influence planning for future campaigns. 3. MIS by itself will not provide definitive evidence of the health impact of UC because of the confounding effects of greater ACT availability, case management training and stronger program implementation overall. We recommend that the Phase II UC evaluation include scope for investigating possible confounding factors. While HMIS data are considered weak, trend lines within specific districts may indicate if abrupt declines in malaria morbidity occurred within a few months after LLIN distribution. 74 75 Appendix 11. Universal campaign chronology quantification Date uncertain: GF Round 7 Phase 1 provided $51,422,148 for procurement of 7,295,850 nets ($40,885,477) through Voluntary Pooled Procurement (VPP) + $6,297,305 for distribution support through CSOs. Objective was UC by December 2010 but starting with targeted campaign for pregnant women and (non-pregnant) children. Phase 1 nets fell short by 657,372, leaving 7 eastern districts uncovered (Bududa, Soroti, Namutumba, Bugiri, Kaliro, Mayuge, and Manafwa). 2009: MIS estimated 47% of households owned at least one net. 2010: Phase I nets (7,295,850) distributed (with PSI support) to targeted recipients but omitted 33 sub-counties in seven districts (partly because estimates of target population developed in 2007 were no longer valid in 2010).  May: Central Region (2,367,800 nets)  July-August: Western (2,710,075 nets)  September: Northern (1,213,275 nets)  October to December: Eastern (1,004,075) Date uncertain: PMI offered to provide 650,000 nets to cover 33 sub-counties plus estimated population increases since 2007. 2011: UDHDS estimates household ownership at 74% 2011-12: Phase II nets delayed by GF issues. By 2012, nets distributed in 2010 are reaching maturity and cannot be considered for universal coverage. Phase II postponed to 2013. Population increases require further adjustment of UC quantification requirements. 2012: Estimate of UC requirement raised to 19.9m. PMI offers 650,000 nets to cover the growing gap between GF projections (originally developed in 2007) and updated projections. May 2012: 650,000 PMI/DfID nets arrive August 2012: 550,000 PMI/DfID nets arrive September 2012: Malaria Consortium distributes 651,890 nets in four eastern districts (Serere, Mayuge, Kaliro, Bugiri), for registered household population of 1,619,227. 217,610 of 329,558 registered HH received at least two nets. Falls short of UC. Registered population exceeded adjusted UBOS data (2002 population + 16%) by an average of 15.7% (range 3.1 to 34.2%). May 2013: Wave 1 (Busia and Soroti) distributed 422,016 nets 76 October 3-7, 2013: Wave 2 (16 Eastern districts: Sironko, Kaber, Bukedea, Bukwo, Bulambuli, Kapchorwa, Katakwi, Kumi, Kween, Manafwa, Nakapiripirit, Ngora, Pallisa, Amuria, Bududa, Amudat) distributed 2,078,981 nets. September 22 to November 4, 2013: Wave 3 (16 Eastern districts: Budaka, Kibuku, Namutumba, Tororo, Namayingo, LuukaIganga, Buyende, Jinja, Kamuli, Buikwe, Buikwe, Buvuma, Mukono, Kayunga, Butaleja) distributed 3,717,356 nets October 6 to December 2, 2013: Wave 4 (18 Central and Southwestern districts: Rakai, Lwengo, Kiboga, Kyankwanzi, Mityana, Luwero, Mpigi, Sembabule, Nakasongola, Gomba, Lyantonde, Mubende, Kalungu, Masaka, Nakaseke, Butambala, Kalangala, Bukomansimbi) distributed 2,843,916 nets November 10, 2013 to January 6, 2014: Wave 5 (15 Western districts: Rubirizi, Bushenyi, Mitooma, Sheema, Mbarara, BuhwejuIsingiro, Kabarole, Bundibujyo, Ntoroko, Kyegegwa, Kyenjojo, Kamwenge, KiruhuraIbanda) distributed 2,815,783 nets. Campaign passes halfway mark. December 3, 2013 to February 3, 2014: Wave 6 (17 Northern and western districts: Buliisa, Kanungu, RUKUNGIRI, Kabale, Kisoro, NTUNGAMO, Kasese, Hoima, Masindi, Kiryandongo, Kibaale, Apac, Oyam, Kole, Amolatar, Lira, Dokolo) distributed 4,187,957 nets January 12 to March 3, 2014: Wave 7 (Kaabong, Kotido, Moroto, Napak, Abim, Otuke, Kitgum, Gulu, Alebtong, Agago, Nwoya, Lamwo, Pader, Amuru, Nebbi, Zombo) distributed 2,450,558 nets January 2014: 2,050,000 PMI/DfID nets arrive February 9 to March 31, 2014: Wave 8a (6 West Nile districts: Maracha, Koboko, Yumbe, Moyo, Adjumani, Arua) distributed 1,232,475 nets March 2014: 1,450,000 PMI/DfID nets arrive April 2014: 1,500,000 PMI/DfID nets arrive May 2014: RBM estimates UC requirement for 19,642,778 nets based on UBOS adjusted population of 35,357,000 for 2013; 15% registration adjustment brings estimated need to 22,589,194 for 2013 and 23,393,300 for 2014. 2014: GF concept note estimates 25,925,281 UC requirement for 2017 July 2014: 600,000 PMI/DfID nets arrive August 2014: Kampala Wakiso distribution: 1,866,845 nets 77 Appendix 12: Financial Mechanism for the UC Planned Process: The financial management function for this exercise shall be undertaken by a lead agency or its representative guided by a memorandum of understanding (MoU). The MoU would ensure compliance with requirements of the Global Fund, DFID, PMI and any development partner supporting the UC. In addition, the MoU would bind all stakeholders to transparency, proper financial accounting and standard reporting at regular intervals to the NCC. Process as implemented: The MoH provided overall leadership and technical guidance for the whole process of the LLINs mass distribution campaign. Overall technical and administrative oversight was provided by the National Coordination Committee (NCC), supported by the Logistics Sub-Committee (LSC), Operations Sub-Committee (OSC) and the Advocacy and Social Mobilization Sub-Committee (ASMSC). There was a single national implementation plan, under the leadership of the National Malaria Control Program. John Hopkins university (JHU) was appointed as the lead agent to manage the financial aspects of the LLIN campaign. JHU appointed Malaria Consortium as a technical sub partner to manage the LLIN distribution campaign funds from the DFID/PMI partnership. Furthermore the MoH appointed Malaria Consortium to manage the funds from the Global Fund for the LLIN universal coverage campaign. Malaria Consortium managed the funds from USAID/DFID through the Stop Malaria Project mechanism. By the time of the evaluation closure the team had not verified the funding expenditure. Emerging lessons: The successful collaboration between different donors, agencies and implementing partners with the MoH is possible as was the case in Uganda during the UC. The successful collaboration was based on cooperative agreements. This can provide a good lesson for global development agencies and country programs. The UC facilitated the establishment of a complex yet workable funding mechanism for channeling donor funds with a shorter turnaround time. This demonstrated a strong MoH and non-MoH partnership in Uganda. Recommendation  We recommend financial data quality audits to accurately ascertain the cost and expenditure during the UC.  Proper documentation of the UC finance management processes and protocols, for future use/reference 78 Appendix 13. List of Interviewees Name Title Affiliation 1 Dr Ruth Aceng Director General of Health Services MoH 2 Dr. Wondimagegnehu Alemu Country representative WHO Uganda office 3 Dr. Charles Katureebe Country Advisor on Malaria WHO Uganda office 4 Dr. Magumba Godfrey Country Director Malaria Consortium 5 Dr. Katamba Henry LLINs UC M&E coordinator/Cluster supervisor NMCP 6 Mr. Rukaari Medard Coordinator of UC Logistics teams NMCP 7 Ms. Caty Fall LFA GF 8 Dr. Rwakimari JB Chief of Party Abt Associates IRS program 9 Ms. Chime Mukwakwa Chief of Party SMP 10 Dr. Gidudu Sam Technical Officer SMP 11 Ms. Kate Kikule Inspector and coordinator of GF matters NDA 12 Mr. Mwesigwa Dennis Inspector - Importation and Quality Assurance NDA 13 Dr Atwine Diana Director MHSDMU 14 Mr. Ayume Charles Medicines Monitoring officer MHSDMU 15 Ms. Judith Kayinga Communications Officer MHSDMU 16 Dr. BKKapella Malaria Technical Advisor CDC/PMI 17 Dr. Kassahun Belay Malaria Technical Advisor USAID/PMI 18 Mr. Ekanu Godfrey District Coordinator MC 19 Ms. Patience Karungi District Supervisor NMCP 20 Capt Eliab Kabagambe DISO Kabarole District 21 Ms. Regina Nakabugo VHT Mubende 22 Malaria Focal Person Mubende 23 Community members at Kalonga HC III Kalonga sub county Mubende 24 Mr. Aine Richard District Police Commander Kabarole 25 Sub county Chief Karambi Kabarole District 26 Mr. Kassami Ronald Community Development officer Mugusu Sub county, Kabarole 27 Community members at Mugusu HC III Mugusu sub county Kabarole District 28 Community members at Bwizibwera HC IV Bwizibwera sub county Mbarara District 29 Mr. Nkwasibwe Anthony MJAP Peer Educator Bwizibwera A, Mbarara 30 Ms. Nahabwe Teddy VHT Bwizibwera B, Mbarara 79 Name Title Affiliation 31 Mr. Tumukunnde Didas VHT Bwizibwera A, Mbarara 32 Dr Amooti Kaguna DHO Mbarara District 33 Mr. Kato Ahmed Health Inspector Kakoba Division Mbarara District DISO Mbarara District 34 Mr. Opio Mike Sub county Chief Ongako Gulu District 35 Parish Chief Ongako Gulu District 36 Ms. Aceng Mary VHT, Pece Division Gulu District 37 Ms. Akello Marggie Community Development Officer, Pece Division Gulu District 38 Mr. Musiime Danstun Internal Security Officer Gulu District 39 Community members in Jopakeno Village Gulu District 40 Community members at Ongako HC III Ongako sub county Gulu District 41 Dr. Jakor Oryema DHO Nebbi District 42 Malaria Focal Person Nebbi District 43 Nyaravuru sub county chief Nebbi District 44 Community members at Nyaravu HC III Nyaravu sub county Nebbi District 45 Dr. Elly Tumushabe DHO Mukono District 46 Ms. Crisitine Adyebo Malaria Focal Person Mukono District 47 Household heads two households, Mafuba village Mukono District 48 Household heads two Households Jinja town Jinja District 49 Mr. Bayenda Gilbert Malaria Focal Person Jinja District 50 Mr. Richard Gulume RDC Jinja District