October 2013 Submitted to: Zdenek Suda, Program Officer, USAID/Uganda By Robert Pond, Fred Matovu and Festus Kibuuka [insert here a project photo such as the photo of the Health Assistant conducting a malaria education session at a health facility] Report of the Evaluation of the USAID/Uganda Stop Malaria Project Report of the final evaluation of the USAID/Uganda Stop Malaria Project 0 ACKNOWLEDGEMENT This evaluation was supported by funding from the USAID Mission in Uganda. It benefited from technical advice and support by a number of stakeholders which the consultants highly appreciate. The consultants also appreciate the support and advice by staff of the Stop Malaria Project (SMP) implementing partners, the National Malaria Control Program (NMCP) and the Resource Center of the Ministry of Health who provided useful information towards the implementation of the evaluation and preparation of this report. The input from the USAID Uganda Mission Leadership, Daryl Martyris, the agreement officer’s representative for Stop Malaria Project, the Presidential Malaria Initiative (PMI) team members of USAID/UGANDA and the Stop Malaria Project (SMP) Final Evaluation Task Managers (Ms. Salome Sevume and Ms. Charmaine Matovu) is particularly appreciated. The evaluation team also appreciates the technical input and logistical support provided by the SMP staff, particularly the M&E expert and the Chief of Party. The cooperation and input from all respondents in the districts and facilities visited is highly commended. Cover photo: Uganda. A Midwife at an Antenatal Clinic that is supported by the USAID/Stop Malaria Project, explains to an expectant woman how to use a long lasting insecticide treated mosquito net to prevent malaria. Photographer: Unknown Report for the final evaluation of the USAID/Uganda Stop Malaria Project 1 Report of the Evaluation of the USAID/Uganda Stop Malaria Project October 2013 Submitted to: Zdenek Suda, Program Officer, USAID/Uganda By: Robert Pond Fred Matovu Festus Kibuuka The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 2 TABLE OF CONTENTS Executive Summary……………………………………………………………………………….…………i 1. Background to Stop Malaria Project ………………………………………………………………6 2. Scope of the Evaluation……………………………………………………………………………7 3. Evaluation Methodology ……………………………………………………………………….....7 4. Limitations……………………………………………………………………………………...…10 5. The extent that SMP improved delivery of key malaria interventions in the Districts of operation …………………………………………………………………………..10 5.1 Preventive……………………………………………………………………………......10 5.2 Curative ……………………………………………………………………….………...20 5.3 Systems Strengthening ……………………………………………………………..........28 6. The effectiveness of the project in building the capacities of NMCP and Districts….…………..29 6.1 Strengthen the M&E Capacity of NMCP ……………………………………….……….29 6.2 Strengthen capacity at district level for malaria M&E …………………………………...31 6.3 Strengthen District Supportive Supervision ……………………………………...............35 7. How the partnership between JHU MC, IDI CDFU, UHMG worked …………………..............36 8. Factors associated with major successes and performance weaknesses………………………...38 8.1 Key strengths ……………………………………………………………………...……..38 8.2 Major weaknesses…………………………………………………………………............38 Annex 1 Calendar for the Evaluation Annex 2: Persons Interviewed Annex 3: Documents Reviewed Annex 4 Location of the clusters Of The 2006 DHSs, The 2009 MIS And The 2011 DIS Annex 5 Statement of Work Annex 6 Assessment of the Report of the BCC Survey Annex 7 Field Data Collection Instruments Report for the final evaluation of the USAID/Uganda Stop Malaria Project 3 List of Abbreviations and Acronyms ANC Antenatal care ANC1 The reported number of women who came to a health facility in a month for their first ANC visit BCC Behaviour Change Communication CCP Johns Hopkins University Center for Communication Programs CDFU Communication for Development Foundation Uganda CMD Community medicine distributor DHIS2 District health information system, version 2 (software /data management system used to manage the routinely reported data of the MoH) DHO District health officer DHS Demographic and Health Survey DHT District health team DOTS Directly observed treatment strategy GFTAM Global fund for Tuberculosis, Aids and Malaria GoU Government of Uganda HA Health Assistant (an environmental health cadre of the MoH) HBMF Home-based management of fever HC Health centre (level II, level III or level IV) HF Health facility HFA Health Facility Assessment HMIS Health management information system HW Health worker ICCM Integrated Community Case Management IMCI Integrated management of childhood illnesses IDI Infections Diseases Institute (IDI) IEC Information, education and communication IPTp Intermittent presumptive treatment during pregnancy IPT1 The reported number of women given their first dose of IPTp in a month IPT2 The reported number of women given their second dose of IPTp in a month IPT2 uptake = reported IPT2 / reported ANC1 IPT2 coverage = % of women reporting during a household survey that, when last pregnant, they received 2 or more doses of sulfadoxine- pyrimethamine (SP) IR Intermediate result of a results framework ISS Integrated Supportive Supervision ITN Insecticide treated nets JHU Johns Hopkins University (CCP) LLIN Long-lasting insecticide treated nets LQAS Lot quality assurance sampling survey MC Malaria Consortium MCH Maternal and child health (also a division of the MoH) M&E Monitoring and evaluation MDD Music, Dance and Drama (an annual, nationwide school-based competition) MFP Malaria focal person MIS Malaria Indicator Survey MoH Ministry of Health NMCP National Malaria Control program OPD Out-patient department PMI U.S President’s Malaria Initiative PMP Performance Monitoring Plan QA Quality Assurance division of the MoH RC Resource Center of the MoH (which manages the HMIS) Report for the final evaluation of the USAID/Uganda Stop Malaria Project 4 SDA Safari day allowance (a per diem paid to government workers if they travel for field work – in addition to any transport allowance) SMP Stop Malaria Project SP Sulfadoxine-pyrimethamine (a.k.a. “Fansidar” – a brand name) Testing ratio = reported number of malaria tests (positive + negative) / reported number of malaria cases UHMG Uganda Health Marketing Group USAID United States Agency for International Development USG US government WHO World Health Organisation VHT Village Health Team Report of the final evaluation of the USAID/Uganda Stop Malaria Project i EXECUTIVE SUMMARY SMP was established as a flagship project to increase coverage and use of key interventions for prevention and treatment of malaria in Uganda. The project has been managed by a partnership of organizations with $20.9 million of funding to date from the United States President’s Malaria Initiative and the United State Agency for International Development. The project was designed to provide support to the National Malaria Control Programme as well as work in half of the districts of the country. After year 2 of the project, the original geographic focus of the project was reduced by about one third due to the extra burden of working in newly created districts. The evaluation team addressed four areas: service delivery (preventive, curative and systems strengthening), project performance successes and weaknesses , cost effectiveness and efficiency of the partnership between JHU, IDI, Malaria Consortium, CDFU, UHMG and capacity building of NMCP and districts. The following conclusions and recommendations are based upon evidence compiled from review of project documentation, interview of key informants, visits to a sample of districts and health facilities and secondary analysis of multiple datasets from household surveys, a health facility survey and routine health data of the Ministry of Health (MoH). SMP activities on ANC LLIN distribution and full implantation of ISS did not start until year 3. Therefore most of the evidence on these interventions presented in this report relate to activities for year 3-5. . Key findings and conclusions Service delivery (preventive, curative and systems strengthening) Preventive Support distribution of LLINs  SMP was a major source of support to the NMCP for LLIN distribution campaigns, including outside of the project districts;  SMP achieved high levels of coverage with ANC LLINs between the last quarter of Year 3 (2011) and the middle of Year 5 (2013). (Thereafter, ANC net distribution has declined as PMI has stopped supplying the project with ANC LLINs)  Data from DHS, MIS and LQAS surveys suggest that usage of LLINs by pregnant women has increased significantly in “SMP-supported districts” compared to “non-SMP supported districts”  Future distribution of ANC LLINs depends upon supply mechanisms outside of the control of SMP and the US government (USG) Strengthen intermittent presumptive treatment of malaria in pregnancy (IPTp)  SMP trained some ANC staff at 80% of health facilities. By year 5, however, this percentage was declining due to staff turnover and recruitment;  IPT2 uptake increased from 40% in year 3 to 50+% in year 4.  No further improvement for the last 24 months  For some SMP facilities, IPT2 uptake <30%  Non-SMP-supported districts have caught up Communicate to change behaviours  SMP spent $3.8 million on a series of media campaigns using more than 100,000 radio spots, more than 100 radio talk shows, 15 large billboards, 2300 signs and 210,000 flyers Report for the final evaluation of the USAID/Uganda Stop Malaria Project ii  The 2012 BCC household survey showed that persons exposed to SMP malaria messages were 6% more likely to have been tested before treatment. However, exposure to the messages was not associated with other key behaviours (‘i.e.’ early treatment; and using an LLIN)  CDFU spent $1.1 million to support MoH Health Assistants in 10 districts to conduct malaria education activities in schools, communities and health facilities. Focus group discussions showed that residents sampled from these 10 districts had more knowledge on LLINS, IPTp and early treatment than residents in control districts.  SMP’s work in schools led to malaria being adopted as the theme for this year’s Music Dance and Drama competition which aims to target 7,200,000 students nationwide. Curative Improve diagnosis and treatment  SMP has improved the accuracy of malaria microscopy  SMP trained some laboratory staff at 80% of health facilities  The “testing ratio” has increased from 0.35 to 0.70 in SMP-supported districts (vs. 0.55 in non￾SMP supported districts), but the majority of malaria diagnoses are still not lab confirmed  There is only anecdotal evidence of improved quality of malaria case management  The proportion of staff trained is declining due to staff turnover and recruitment  Appropriate drugs for pre-referral treatment of severe malaria are not yet supplied to many health facilities Support access to ACTs in the home and community  At the request of PMI and NMCP, SMP stopped working on this component in year 3. However, the MoH now seems to support community-based treatment of malaria.  A large proportion of febrile illness is still managed at home without care from a health facility. Systems strengthening Operationalizing Malaria related policies and guidelines SMP met project objectives for NMCP to complete the development of more than 20 important policies, guidelines and training materials; however the MoH needs additional support to implement many of the current policies and guidelines. The effectiveness and efficiency of the partnership between JHU, IDI, CDFU Malaria Consortium and UHMG  All evidence suggests that the SMP partners worked well together. Their capacities and roles were complementary and respected by each other.  The quarterly coordination meetings enhanced the partnership and provided a platform for joint planning and common understanding of the SMP interventions  SMP was implemented according to plan and successfully completed a large number of activities at national level (policies, guidelines) and in the 34 focus districts.  Progress at national level was constrained by the increasing limitations of the NMCP, particularly low staffing levels. Capacity building Strengthen the M&E capacity of NMCP  SMP seconded an M&E specialist to NMCP for 3 years. She was quite effective at training staff and increasing production of strategic information (e.g. MPR Report). However, by the time she left, there was no counterpart within NMCP to carry on the work.  However, since year 3, half of NMCP staff left without replacement and NMCP is now too weak to undertake M&E  SMP support for the MoH Resource Center (RC) has helped develop the national HMIS Report for the final evaluation of the USAID/Uganda Stop Malaria Project iii Strengthen capacity at district level for malaria M&E  SMP trained records assistants and facility in-charges at more than 80% of health facilities in data quality assessment. However, the available evidence shows that improvements in data quality have been limited.  SMP succeeded at developing the M&E capacity of district biostatisticians and the data use practices of health facilities.  SMP also developed some other aspects of district capacity. However, SMP planning was not well integrated with district planning and tended to by-pass the constraints of district capacity. Strengthen district supportive supervision  From years 3 to 5, SMP spent about $2 million on quarterly “Integrated Supportive Supervision” (ISS) of 50% to 70% of HC’s in SMP-supported districts.  ISS was however “vertical” (only malaria), depended on SMP for vehicles/SDA and not sustainable.  ISS built the capacity of individual district staff for malaria supervision and “mentoring”, but did not strengthen the districts’ own supervision processes.  There is insufficient evidence to show that ISS mentoring has improved health worker knowledge and practice. Recommendations Prevention  ANC LLIN distribution should be taken to scale nationwide. This will require building the capacity of other organizations including non-SMP supported districts and other relevant intermediaries in micro-planning and training for ANC LLIN distribution. Logistical support to deliver LLINs to health facilities and technical assistance for monitoring of LLIN distribution are also vital priority  MoH and partners should focus more on health facilities which have performed persistently poorly with key indicators such as IPT2 uptake, the testing ratio and inconsistency of data. Each of these indicators can be tracked on a monthly basis for each health facility in the country using the DHIS database. The district biostatisticians should play the key role in tracking such indicators and identifying low performing health facilities.  To document impact, suitable household surveys should be carried out at baseline and “end￾line” (following) BCC campaigns costing more than $1 million.  Even without better evidence of their effectiveness, the evaluation team is convinced that major, long-term BCC campaigns are an essential component of malaria control and should be included in future USG-supported projects.  School-based approaches to BCC appear promising and warrant further support and evaluation. Curative  MoH and partners should strengthen pre-service training of laboratory workers.  Every HC and every district should monitor and display their testing ratio (total malaria tests reported / total malaria cases reported; target > >100%).  There is a pressing need for updated, user friendly and widely distributed job aides.  Due to new recruitments and staff turnover, large scale trainings now need to be repeated.  Broaden case management training (and job aides) to cover management of other childhood illnesses (IMCI – integrated management of childhood illnesses).  The effectiveness of SMP’s “clinical audits” needs to be better documented including with use of well defined, fixed indicators that are tracked over multiple visits.  Projects supporting drug supply need to focus on the supply of artesunate and injectable quinine.  The US government (USG) should support any strategy for community-based treatment of malaria that is embraced by the MoH (e.g. Integrated Community Case Management). Report for the final evaluation of the USAID/Uganda Stop Malaria Project iv  Avoid massive overload of VHTs. Serious attention must be given to VHT motivation and refresher training.  A “flag bearer” or “country champion” is needed to advocate for raising the status of malaria control in Uganda. Finding and supporting such a flag bearer should be one type of BCC intervention. NMCP M&E capacity building  USG projects should support the strengthening of linkages between NMCP and other divisions of the MoH (Resource Center, MCH, and Quality Assurance). These other divisions will be able to implement malaria-related activities in a way that complements the role of the NMCP.  USG, GoU or other development partners should again fund the secondment of a seasoned M&E specialist to the NMCP as part of overall capacity-building support. This secondment should depend upon the staff achieving concrete deliverables. The appropriate deliverables should be identified on the basis of a needs assessment of the NMCP. District M&E capacity building  USG projects should integrate project activities, including supportive supervision, into district planning and budgeting.  For nationwide impact, USG should find ways to provide additional support to the Resource Center (RC) of the MoH for further development of the DHIS.  The DHIS software should be configured to reduce entry of inconsistent data. (e.g. ANC1 < IPT 1 & IPT2).  District biostatisticians should be trained to regularly download the disaggregated data and review it to identify health centers with inconsistent data. District Supportive Supervision capacity building  Supervision checklists should include questions to objectively assess health worker knowledge and practice.  Supervision checklists need to be broadened to provide for supervision of other health services in addition to malaria prevention, diagnosis and treatment (i.e. supervision of management of other causes of febrile illness). Major project performance successes and weaknesses Strengths: Each of the implementing partners had had a long experience in the areas they were focusing on (i.e. Malaria consortium for case management and LLIN distribution, IDI for training, and CDFU for BCC activities). The coordination framework (quarterly meetings) and mutual respect helped the partnership to make joint planning and have a platform for review of progress and common understanding of the project interventions. Weaknesses:  SMP was not in direct control of LLINs and drug supply. This limited progress on these activities (e.g. LLIN distribution did not start until Year 3).  SMP did not directly work within the district planning and budgeting framework (by-passing the district capacity limitations). This brings sustainability/continuity of SMP ISS into question. Report for the final evaluation of the USAID/Uganda Stop Malaria Project v  Use of outdated job aids that do not reflect the new malaria treatment guidelines plus the clinical guidelines that are quite bulky and not user-friendly.  Due to new recruitments and turnover, large scale trainings in IPTp and laboratory diagnosis of malaria now need to be repeated.  The positioning of the NMCP within the MoH organogram is low. The implication of this is a restricted decision-making space on policy, technical and resource allocation matters. It minimizes the mandate and authority of the NMCP to properly head and guide malaria policy and implementation activities.  Since year 3, half of NMCP staff have left without replacement. This understaffing has impacted on NMCP participation in SMP supported activities in general. Report of the final evaluation of the USAID/Uganda Stop Malaria Project 6 1. BACKGROUND ON THE STOP MALARIA PROJECT SMP aims to support the goals of the National Malaria Control Program (NMCP) of the Ministry of Health (MoH): specifically to achieve PMI/Uganda’s targets of 85% coverage with key interventions for the prevention and treatment of malaria. These interventions are presented in the Results Framework of the project (see Figure 1). SMP has been managed by the Johns Hopkins University Center for Communication Programs (CCP) and implemented in partnership with the Malaria Consortium (MC), Communication for Development Foundation Uganda (CDFU), the Infectious Diseases Institute (IDI), and Uganda Health Marketing Group (UHMG), with $20.9 million of funding to date from PMI. SMP was designed to work at the national level with the NMCP as well as to support 45 of the 88 districts that existed as of 2008. By year 3 of the project, the government of Uganda (GoU) had split many districts, the total number of districts had grown to 111 and USAID/PMI agreed that the project should focus on only 34 districts due to the extra burden of providing support to newly created districts. As shown in Figure 2, this involved a reduction of about one third in the geographic coverage of the project. Development Objective 3: Improved Health and Nutrition Status in Focus Areas and Population Groups Program Objectives 3.1.1: Reduce Malaria Mortality Malaria prevention improved IR1: IR2: Malaria diagnosis and treatment improved IR3: NMCP capacity strengthened IR 1.2: Coverage with IPTp increased ’ IR 2.1 Malaria diagnosis, treatment and referral at health facilities improved , Technical resources and skills of M & E sub￾unit improved IR 3.1 - IR 3.2 Collection, processing and use of data by districts and implementing partners improved IR 1.3: Coverage with LLINS increased IR 2.3 Community knowledge, attitudes and practices for management of febrile illnesses improved IR 2.2 Increased access to ACTS Malaria related policies and guidelines operationalised Figure 1: The SMP Results Framework Report for the final evaluation of the USAID/Uganda Stop Malaria Project 7 Figure 2: Geographic focus of SMP 2. THE SCOPE OF WORK FOR THIS EVALUATION The evaluation aimed to provide evidence to guide strategic targeting and investment for future USAID/Uganda malaria interventions by answering the following four evaluation questions: 1. To what extent has SMP improved delivery of key (global standard) malaria interventions in the districts of operation? (Key intervention areas cover preventive, curative and systems strengthening). 2. What are the factors associated to the major successes and performance weaknesses? 3. How well did the partnership between JHU, Malaria Consortium, IDI, CDFU, UHMG work in terms of cost effectiveness and process/implementation efficiency? 4. How effectively is the project building the capacities of NMCP and districts as laid out in the cooperative agreement and the monitoring and evaluation (M&E) framework, and with regard to improving capacity to properly manage malaria control in Uganda? 3. EVALUATION METHODOLOGY Composition of the evaluation team The evaluation team was composed of three consultants: a team leader, a malaria program expert and an organizational capacity expert. The timeline for the evaluation In-country work of the evaluation team extended from 2 September to 12 October, 2013. A detailed calendar for the evaluation is provided in Annex 1. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 8 Overview of evaluation methods The evaluation team  reviewed relevant documentation (list included as Annex 2);  interviewed key informants (list included as Annex 3);  visited a sample of 6 project supported districts and 3 non-project supported districts to interview staff of District Health Teams and a sample of health facilities;  Performed secondary analyses of relevant datasets from household surveys (DHS, MIS, and LQAS), a health facility assessment (“HFA”), routine health service data (“HMIS”) and integrated supportive supervision (“ISS”). Field visits to a sample of districts and health facilities In advance of the evaluation field visits, the data collection instruments were pre-tested in Mukono district then further refined. Over a two week period, the three consultants split up to travel separately and visit a total of 9 districts from the three regions (i.e. Central, Eastern and Mid-western) in which SMP operated. Five districts were selected in the Central Region, of which 4 districts were SMP-supported districts (i.e. Kayunga, Mityana, Mpigi and Masaka) and one district was a non-SMP supported district (i.e. Lyantonde) for purposes of comparison. In the Eastern Region an evaluator visited 2 districts (i.e. Kumi – SMP supported, and Pallisa – Non-SMP supported) while in the Mid-Western Region, an evaluator visited Hoima district (SMP supported) and Kyenjojo district (Non-SMP supported). Within each district, 3 health facilities were randomly selected from among the list of health facilities surveyed by the SMP during the 2011 Health Facility Assessment (2011 HFA). In this way, the findings of the evaluation team could be compared with the findings 2 years previously of the 2011 HFA for the same health facilities. The SMP-supported districts were selected to represent the regional distribution of SMP (i.e. more districts were visited in Central Region than in Eastern or Mid-Western Regions) and, for logistical purposes, the three non-SMP supported comparison districts were chosen from among those adjacent to the selected six SMP-supported districts. Data collection and analytic plan Further analysis of survey data and ISS data The datasets for the 2006 DHS, the 2009 MIS and the 2011 DHS included the latitude and longitude for each of the clusters surveyed. This permits the clusters within SMP and non-SMP-supported districts to be identified as illustrated by the map in Annex 4. The team also conducted further analyses of data from LQAS surveys in 2011, 2012 and 2013. This data is provided for each individual district and thus could be easily grouped to compare SMP and non SMP-supported districts. SMP’s data from quarterly Integrated Supportive Supervision (ISS – conducted only in SMP-supported districts) were analyzed to assess longitudinal trends in key indicators (e.g. % of health facilities with a stock out of key drugs, % of health facilities with trained staff). These data also permitted assessment of whether the trends varied between levels of health facilities (hospitals versus HCIV’s vs HCIIIs versus HCIIs). Analysis of HMIS data There were two reasons why it was important for the evaluation to analyze the data reported monthly by health facilities to the Health Management Information System (HMIS) of the Ministry of Health. That is:  to assess the Intermediate Results 3.1 and 3.2 – strengthening of M&E at NMCP and district levels  to measure key indicators for IPTp (i.e. the “IPTp uptake” = IPT2/ANC1) and diagnosis of malaria (i.e. the “testing ratio” = total malaria tests performed / total malaria cases reported) Report for the final evaluation of the USAID/Uganda Stop Malaria Project 9 HMIS data have the virtue of being available for all districts of Uganda and for a range of months. Hence, these data permitted comparison of a sample of 6 SMP-supported districts (the same as those visited by the evaluation team) and 6 non-SMP-supported districts (the 3 visited by the evaluation team plus 3 more districts randomly selected from the Eastern and Mid-Western Regions. However, HMIS data typically have significant problems with incompleteness and inaccuracy. Hence, the first task in analyzing these data is to assess their completeness and quality. To do this, the fully disaggregated data from September 2012 to August 2013 (i.e. the data for each individual month and for each and every individual health facility in the 12 selected districts) were downloaded from the District Health Information System (DHIS) database maintained by the Resource Centre of the MoH. With this data it was possible to identify months for which data were not reported and identify internal inconsistencies (e.g.. reported positive malaria tests > reported number of malaria tests performed; reported number of positive malaria tests > reported number of malaria cases; reported IPT2 > reported ANC1 visits; reported IPT1 > reported IPT2) 1 and other deficiencies in the data (e.g. data were reported for the number of malaria cases but no data were reported for the number of malaria tests). The second task in analysis of the HMIS data was to clean them to permit valid interpretation. For a valid estimate of the malaria test ratio (reported number of malaria tests / reported number of malaria cases), the data were first cleaned by removing from the numerator and the denominator the data for months when either tests or cases were not reported. For a valid estimate of IPT2 uptake (reported IPT2 / reported ANC1), the data were first cleaned by removing from the numerator and the denominator the data for months when these data were inconsistent (i.e. ANC1 < IPT2). To illustrate the importance of using the disaggregated data and of cleaning the data before analysis, consider what the evaluators discovered when they reviewed the data from Hoima District. In the month of April 2013, Kyehoro HCII reported only 56 ANC1 visits but 6,868 IPT2 administrations. This was very likely the result of a data entry mistake at district level. If these obviously invalid data are included, the IPT2 uptake for Hoima district for the last 12 months appears to be 65%. If, however, the invalid data are omitted, the IPT2 uptake for Hoima district for the last 12 months drops to 46%. Collection and analysis of data from field visits to districts and health facilities Questionnaires were developed for collection of data from districts and health facilities. For health facilities, five different questionnaires were used (instruments included as annex 7):  one for a facility audit (this is administered by posing questions to the facility in-charge and other staff and by inspecting the stocks, records and the laboratory);  one for interviewing clinical staff providing either ANC services;  one for interviewing clinical staff who manage febrile illnesses;  one for interviewing clients exiting from ANC; and  one for interviewing clients exiting from the OPD department/consultation room. 1 ANC1 >100%).  There is an urgent need for updated, user friendly and widely distributed job aides.  Due to new recruitments and turnover, large scale trainings now need to be repeated  Broaden case management training (and job aides) to cover management of other childhood illnesses (IMCI). Training materials as well as the Uganda Clinical Guidelines must be made more user- friendly if they are to be understood and used by nurses and nursing assistants.  The effectiveness of SMP’s “clinical audits” needs to be better documented including with use of well defined, fixed indicators that are tracked over multiple visits.  Projects supporting drug supply need to focus on the supply of artesunate and injectable quinine. The evidence Overview Activities in pursuit of improvement of diagnosis and treatment  Strengthening laboratory capacity (with activities managed by IDI);  Improving malaria diagnosis by working with health staff to increase the percentage of malaria diagnoses that are laboratory confirmed; and  Strengthening treatment of malaria at health facilities. This includes the management of severe malaria as well as uncomplicated malaria. The Cooperative Agreement specified that, “At the facility level, we will establish sound parasitological diagnosis of fever cases. Management of severe malaria will be improved through early recognition of danger signs and pre-referral and referral level treatment.” The year 2 annual reported noted that “the project experienced some challenges that were beyond its scope and yet critical to achievement of planned results. The challenges included: frequent stock outs of ACTs …. Quinine (oral and injectable), IV fluids and related supplies.” Strengthen laboratory capacity Summary of the proposed approach and summary of progress during each year of the project as reported by SMP The year 2 Annual Report listed various constraints to laboratory diagnosis: “…inadequate laboratory staff compared to the work load… ; stock out of laboratory reagents… ; poor quality microscopes… and a limited budget to carry out routine supervision by the District Laboratory Focal Persons… A number of HC IIIs do not have microscopes…. The MoH is yet to roll out RDTs to health facilities without microscopes.”” The CA stipulated that “IDI and MC will work with RBM partners to… strengthen microscopy in laboratories and introduce RDTs at health center II to complement microscopy… the focus will be on the performance of the laboratory technicians, which will be assessed during quarterly support supervision and an external quality assurance system… [A JUMP] 9 days course will be held at IDI’s facilities…. With the MOH, these ‘Peer Trainers’ will then carry out a cascade training targeting health workers and laboratory technicians at HCIV and HCIII. Each cascade session will last 6 days…” During the first two years of the project IDI adapted the existing laboratory training course called JUMP to train 500 laboratory staff in microscopy and RDT. As shown by Figure 16, “The pre-training accuracy Report for the final evaluation of the USAID/Uganda Stop Malaria Project 22 Pre and post test results-Microscopy 0 10 20 30 40 50 60 70 80 90 100 1 3 5 7 9 11 13 15 17 19 Personal identification number (PIN) Marks (%) Pre test- Slide reading Post test- Slide reading of laboratory staff in terms of correctly reading positive and negative blood slides was very low” and accuracy improved dramatically as a result of the training. This covered 75% of public health facilities with laboratories in the focus districts. In year 3 another 594 laboratory technicians/assistants were trained. Some districts had training coverage below 80% due to lack of a district laboratory focal point. Trainees were each followed up within 6 weeks of training by a team of 3 laboratory specialists from IDI, NMCP and the National Public Health Reference Laboratory. A system of External Quality Assessment (EQA) was also introduced for 136 health facilities (4 per district) whereby a sample of blood smear slides were re-read at district level (with discordant readings then read again at national level by IDI or the Central Public Health Laboratories). Blood slide reading accuracy increased from 75% in Q2 of Year 3 to more than 90% by Q2 of Year 4 when another 176 laboratory staff were trained. The external quality assurance (EQA) activity continued in year 5, however, the quarterly reports do not provide any statistics on the accuracy of slides. Summary of the progress as monitored by ISS As shown in Figure 17, ISS data suggest that the percentage of health facilities with at least one laboratory worker trained in malaria diagnosis increased sharply during year 3. However, by year 4, many hospitals, HCIV’s and especially HCIII’s in the SMP￾supported districts lacked any laboratory workers trained in malaria diagnosis. According to ISS data, the availability of trained lab workers dropped significantly further by Q3 of year 5. 5 Improve diagnosis Baseline – The 2009 MIS found that “the proportion of children under five with fever who received a diagnostic test prior to treatment of fever from the health facility for both the SMP-supported districts and entire country was 17%. 5 HCII’s are not featured in this analysis as they have no laboratories. The indicator does not appear to have been measured during year 4 Q1 round of supervision. Figure 16 Figure 17 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 23 The year 2 Annual Report noted that “Clinicians show reluctance to respect negative laboratory results when a syndromic assessment appears indicative of a malaria diagnosis.” The year 3 to year 5 Annual Reports presents data showing an increase in the testing ratio from 38% during Q1 to 70% by year 5 (see Figure 18) . This increase was attributed to the increased availability of RDTs. Note that this graph, taken from an SMP quarterly report, is mislabeled. The testing ratio is NOT “the percentage of children with fever who received a diagnostic test before treatment”. In fact, the testing ratio is usually 2 or more times greater that this stated percentage. This is because the testing ratio includes in the numerator ALL tests, including negative tests. Hence, it is common for the testing ratio to be greater than one and the target for this indicator should be greater than 1.0.6 Analysis of HMIS data show that the testing ratio over the last 12 months was higher in 6 SMP￾supported districts (average = 0.76) than in 6 non-SMP-supported districts (average = 0.58). This is reflected by Figures 19 and 20 which show that a higher percentage of health facilities in the SMP￾supported districts had a testing ratio of greater than 1.0 and a lower percentage had a testing ratio of less than 0.3. The evaluation team found that most health workers still lack confidence in negative RDTs.  30 (55%) of 55 health workers said that if an RDT test is negative they are NOT confident that the patient has malaria;  20 (71%) of 28 OPD clinicians said that if an RDT test is negative then they consider treating for malaria based upon clinical suspicion. 6 As pointed out by the PMI/Uganda Senior Malaria Advisor, testing ratio = (reported number of positive malaria tests / reported number of malaria cases) / the proportion of malaria tests that are positive. Figure 18 Figure 19 Figure 20 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 24 During year 5, to design a communication campaign to promote prompt testing, SMP visited a sample of health facilities “… to identify current knowledge, attitudes and practices regarding malaria testing in general and RDT use in particular. [A]... report was compiled and used among other literature in and outside Uganda to inform the strategy design process.” During their field visits, the evaluators found two HCII’s which had recorded a remarkably high percentage of positive RDT tests in their lab registers. At one facility, for which HMIS data is shown in Figure 21, 93 of the last 100 RDTs were recorded as positive. Such a very high malaria test positive rate raises the question of whether the RDTs are being misread by poorly trained HCII staff or whether the RDT test kits themselves may be defective. Staff of IDI acknowledged that they had heard reports of their being defective RDTs in Uganda. Strengthen treatment of uncomplicated malaria at health facilities During year 2, “frequent stock outs of ACTs in many health facilities coupled with the high cost of ACTs in private outlets… led health workers and patients to resort to use of ineffective medicines (chloroquine and SP) for treatment of malaria.” SMP attempted to intervene (“[SMP] Monitored ACT stock outs routinely at health facility and relayed information about stock outs from SURE at the national level to the facilities...”), however, the evaluation team was left with the impression that the project had quite limited control over the supply of essential drugs. Between year 3 and year 4, the supply of ACTs to health facilities improved considerably and remained adequate until present (see Figure 22) Training of health workers in management of uncomplicated malaria was delayed: “Orientation of the health workers [in case management] did not take place-awaiting approval of the new National Malaria Treatment Policy by MoH top management.” Finally, in year 4, 5,651 health workers (vs a target of Figure 21 Figure 23 Figure 22 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 25 4,567) were trained with the 4 day Integrated Management of Malaria (IMM) in-service training course covering management of uncomplicated malaria, management of severe malaria and use of RDTs. This resulted in a major increase in the % of health facilities with staff trained in management of malaria (see Figure 23).7 Beginning around Q2 of year 5, however, the training coverage (% of health facilities with at least 2 health workers trained in management of malaria) began to drop and SMP’s Q3 quarterly report of year 5 observed that “Although SMP-supported districts to train many health workers, especially in integrated management of malaria (IMM), many districts have recruited new health workers that are not well conversant with the new guidelines of malaria treatment. During ISS for July – September 2013 quarter, SMP together with the districts will identify the number of new health workers recruited so that these health workers are trained in IMM in the first half of SMP year six.” Review of the IMM course materials shows that the course did not instruct in management of diarrhoea (one of the top three causes of morbidity in Uganda) and promoted an approach to management of febrile illness that was overly complex. Course materials provide a rapid, superficial overview of how to take a complete physical exam (e.g. “listen for rhonchi, crepitations... any heart sounds such as murmurs, rubs and gallops”). SMP reports note that “Health workers participating in the IMM trainings sometimes struggled to conduct a complete medical history and physical exam.” In this respect, the IMM training materials are similar to the Uganda Clinical Guidelines, the most common job aide now found at health facilities in Uganda: stuffed with words and topics suitable only for doctors and clinical officers, not user friendly and not making use of the integrated management of childhood illness algorithm that has been endorsed by the Ugandan MoH. With the 4 day IMM course, an opportunity was missed to train large numbers of health workers in a practical integrated case management approach. This was in part due to the decision to train on in-patient care of malaria (the participant is to learn that “acidosis” is defined as Plasma bicarbonate < 15 mmol/L”) as well as management of uncomplicated malaria and pre-referral treatment. For health workers needing an easy reference following IMM training and for health workers who were not able to attend IMM training, up-to-date, user friendly job aides are frequently not available at health facilities. Of the 19 health facilities visited by the evaluation team, 14 had copies of the 523 page Uganda Clinical Guidelines and 5 could locate a copy of the 129 page training guide from the IMM course. The only malaria treatment guidelines that health workers had posted to walls or available on their desks were either charts printed in 2005 or IMCI job aides. At the time of the evaluation, SMP had just begun training of providers in use of a recently developed job aide. This job aide, developed in collaboration with the Maternal and Child Health Division of the MoH, promoted used of an n integrated approach to diagnosis and treatment of febrile illnesses. The evaluators did not observe this new job aide at any of the health facilities they visited. Unfortunately, the ISS supervision checklist does not include a sufficient number and variety of well￾defined criteria with which to objectively assess and track health worker knowledge and practice. The closest thing to such an item is question CM5 which asks the supervisor to “Observe if health workers are giving treatment according to the National Malaria Treatment Guidelines.” The checklist then defines correct treatment as “Right drug, right dose & schedule”. Such minimal criteria do not permit ISS data to be used to track progress with health worker knowledge and skills. More importantly, such restricted assessment criteria do not facilitate mentoring on the most important aspects of care (e.g. assessment of danger signs, lab confirmation of malaria diagnoses, adequate exam for other causes of febrile illness, etc…). 7 For the ISS of Q1 of year 3, less than 30% of the health facilities in the 34 districts were visited. This may explain why an anomalously high percentage of health facilities had trained health workers during this round of ISS. With Figure17 and Figure 3 the data point for Q3Y3 has been shaded out. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 26 Strengthen treatment of severe malaria The Cooperative Agreement notes that “Health workers at HCII and HCIII will be trained on the role of timely referral in the management of severe malaria. Recognition of danger signs is of critical importance in this regard and emphasis will be given to this in preparing new job aids…. The health workers will be trained on how to calculate the dosage of rectal artesunate to give children suspected to have severe malaria and how to administer the medicine.” During years 1 and 2, SMP trained almost 4,000 health providers in management of severe malaria and “clinical audits”. “Prior to SMP’s work, none of the health workers in any of the health facilities visited had received any training focusing on the management of severe malaria in the last four years.” The intervention known as the “clinical audit” is not well described in SMP reports. SMP’s technical team explained to the evaluators that it involved a 3 day visit each quarter to each targeted hospitals and HCIV to review the care provided for severe malaria. These were phased in until by year 4 they were conducted in more than half of the 89 hospitals and HCIVs in the 34 districts. SMPs annual and quarterly reports include anecdotes suggesting that the clinical audits were having a positive impact such as more careful administration of IV quinine. “At Buliisa HC IV and Kibaale HC IV, waiting time for suspected severe malaria cases in OPD before a consultation has improved from more than 1 hour and 2 hours, respectively, to at most 30 minutes….” “At Masindi Hospital, clinicians in OPD consultation rooms routinely conduct emergency assessment and prescription of emergency treatment for severely ill patients.” “… at Ntwetwe HC IV, recognition of the severely sick patients is promptly done (within 25 minutes) through an established triage system and treatment is initiated in a timely manner Presence of a “functioning triage system” appears to be the only indicator of performance or quality of care that is reported on in more than one SMP report. i.e. the % of hospitals or HCIVs with a “functional triage system”. The year 4 report noted that“100% of hospitals and HC IVs (compared to 85% in year 3) maintained a functional triage system for timely recognition of severely ill patients followed by appropriate treatment.” Unfortunately, the definition of this indicator and the means of assessing it are not discussed further and no other data are provided on it in subsequent reports. SMP reports note that some people want to conduct clinical audits in non-SMP districts. “The tool needs to be streamlined so that it can be used at health facilities without the support of the districts, and development of critical variables identified to track performance improvement…”. “NMCP plans to use the revised tool to roll out clinical audits across the country.” However, “The districts continue to wait for SMP funding (for transport to the health facilities, and for day allowances) to conduct clinical audits….” A major constraint to management of severe malaria has been the absence appropriate drugs for pre￾referral treatment. National policy has now endorsed a switch from parenteral quinine to parenteral artesunate for treatment of severe malaria and rectal artesunate for pre-referral treatment of severe malaria. During Q3 of year 5, “It was … noted during clinical audits that all the HC IVs (54/54 HC IVs) and hospitals (35/35 hospitals) have started using injectable Artesunate for treatment of severe malaria cases….. “ However, as of September when the evaluation team made their visits only 3 of 10 higher level HC’s (III, IV, hospital) visited in SMP-supported districts had ever received injectable artesunate and only 1 of 7 HCII’s had ever received artesunate suppositories. In the absence of alternatives, the preferred practice has been to give IM injections of diluted quinine for pre-referral treatment of severe malaria. But this is a task for which many HCII’s lack any qualified staff and lack the injectable quinine (unless they obtain some from a nearby HCIII, HCIV or hospital). SMP’s Q1 year 5 quarterly report remarks that “The lack of parenteral quinine in HC IIs for pre-referral treatment of severe malaria hampers malaria case management in the districts.” The result is, as noted by ISS, 30% or more of children referred for management of severe malaria are still not given pre-referral medication for malaria. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 27 5.2.2 Support access to ACTs in the home and community Findings At the request of PMI and NMCP, SMP stopped working on this component in year 3. However, the MoH now seems to support community-based treatment of malaria that would improve access to ACTs at the community level. The rationale for resuming support for this activity is that a large proportion of febrile illness is still managed at home without care from a health facility. The Malaria Consortium (MC) UNICEF and others have continued working on integrated community case management (ICCM) through Village Health Teams in selected districts, but will soon phase out their activities. Conclusion To support access to ACTs at community level NMS and NMCP need to develop a clear strategy to continue supply of commodities and to provide supportive supervision of VHTs respectively. Recommendations The US government (USG) should support any strategy for community-based treatment of malaria that is embraced by the MoH (e.g. Integrated Community Case Management). But the MoH should avoid massive overload of VHTs; and serious attention must be given to VHT motivation and refresher training. The evidence The Cooperative Agreements stated that “We will support NMCP to implement high quality home￾based management of fever in all 45 districts …” It also summarized some of the constraints to the performance of community medicine distributors (CMDs): “CMDs volunteerism has negatively affected their motivation causing some to either provide poor or no service… The supply of anti-malarials to CMDs is often irregular and sometimes CMDs have to walk long distances to health facilities to collect medicines …” “By rolling out high quality HBMF in the 45 districts we will ensure that the PMI target of at least 85% of children with fever receiving an ACT within 24 hours of fever onset is reached.” Even during year 1 of the project, however, “NMCP, with PMI’s agreement, requested that all further activity regarding CMDs and HBMF be put on hold until long-term availability of ACTs can be established.” SMP then modified its strategy to work with Village Health Teams (VHTs). However, the year 2 annual report notes that “Community mobilization activities… , have not been scaled up to most of the SMP target districts mainly due to changing strategies for community mobilization.” “… the focus shifted from Village Health Teams (VHTs) to strengthening health facilities to be able to reach out to the communities through Health Assistants (HAs).” This work with Health Assistants has been limited to malaria education activities and has not in any way involved community-based distribution of ACTs. The 2011 DHS (Figure 24) found that 33% of children 0-59 months in SMP-supported districts with fever in the 2 weeks preceding the survey had received ACTs the same day or the next day after onset of the fever. Even if we accept findings from the LQAS surveys (Figure 25 -- as already discussed, LQAS findings should be reviewed with caution) at least 40% of children with fever are not being treated promptly with an effective anti-malarial.8 8 Note that a different indicator has been measured with the LQAS surveys: the denominator is limited to children less than 2 years of age (as opposed to less than 5 years of age) and the numerator consists of children given ACT within 24 hours (as opposed to the same day or the next day after onset of the fever). Strictly speaking, PMI has not set a target for the indicator measured with the LQAS. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 28 The Malaria Consortium (MC) UNICEF and others have continued working on integrated community case management (ICCM) through Village Health Teams. In the case of MC this has involved supply of RDTs as well as drugs (ACTs, amoxicillin for pneumonia and ORS/zinc for diarrhea). Evaluators met with the malaria focal persons and VHTs in two districts (Mpigi and Kyenjojo) where ICCM had been implemented for the past two years. In both cases these informants were enthusiastic supporters of the approach. In both cases, district staff noted that the challenge, now that MC and UNICEF are phasing out their support, is for NMS to continue supply of commodities and the health system to provide supportive supervision of VHTs. None of the informants thought these challenges would be easy to confront. Meanwhile, the acting Program Manager of the NMCP and the Assistant Commissioner of the Resource Center of the MoH both expressed support for a VHT strategy that would improve access to ACTs at the community level. 5.3 Systems Strengthening Operationalizing Malaria related policies and guidelines Findings  SMP provided technical and financial support to NMCP to complete the development of more than 20 important policies, guidelines and training materials;  The impact of these policies and guidelines is yet to be assessed since most of them are at their early stages of implementation Conclusion  The development of the Malaria control policy, the NMCP strategic plan and M&E plan and the malaria program review were key milestones in providing a strategic approach to malaria control in Uganda. Recommendation The MoH needs additional support to implement many of the current policies and guidelines The evidence The Cooperative Agreement (CA) specified that: “National policies and guidelines will be updated when necessary to reflect state-of-the-art knowledge and be effectively disseminated to and implemented at the district/facility level.” Figure 24 Figure 25 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 29 Over the 5-year period, SMP provided technical and financial support to develop or review over 20 policies/guidelines/training manuals. The key policies, guidelines and documentation that were finalized with SMP support include:  Malaria control policy 2010/11 – 2019/20  National Malaria Control Strategic Plan 2010/11 – 2014/15 + M&E plan  NMCP- National Communication Strategy 2010/11 – 2014/15.  Three-year Rolling Implementation Plan (2010/13).  Malaria Program Review (MPR),  National implementation guidelines for parasite based diagnosis of malaria  Training manuals/guidelines for: MiP, IPTp, Lab EQA, RDT & Microscopy, LLIN distribution, & National malaria M&E training curriculum During years 1 and 2, SMP supported review and revision of national malaria policies and guidelines on LLIN distribution. Also during this period, SMP supported the MoH Resource Centre to update the NMCP web page on the MoH website for dissemination of NMCP policies, resources and publications. In year 3 SMP supported the NMCP to undertake the Malaria Program Review (MPR) and provided technical and financial support to NMCP to develop its first ever Annual Work Plan 2011/12, the Strategic Plan 2010/15; the National Malaria Control Policy 2010/11-2019 and training manuals for malaria in pregnancy. SMP also provided technical support to NMCP to draft the phase 1 report on Global Fund Round 7 funds. In year 4, SMP provided technical and financial assistance to NMCP to finalize the National Malaria Control Strategic Plan 2010/11 – 2014/15, Monitoring and Evaluation Plan for the National Malaria Control Strategic Plan 2010/11 – 2014/15, Three-year Rolling Implementation Plan (2010/13) and National Communication Strategy 2010/15. Approval of the Malaria Control Policy during year 4 permitted in-service training to commence using the Integrated Malaria Management (IMM) course. In year 5, SMP supported NMCP to develop national implementation guidelines for parasite based diagnosis of malaria. The implementation / roll-out of guidelines has been heavily dependent on SMP support. 6. THE EFFECTIVENESS OF THE PROJECT BUILDING THE CAPACITIES OF NMCP AND DISTRICTS AS LAID OUT IN THE COOPERATIVE AGREEMENT AND THE MONITORING AND EVALUATION (M&E) FRAMEWORK, AND WITH REGARD TO IMPROVING CAPACITY TO PROPERLY MANAGE MALARIA CONTROL IN UGANDA 6.1 Strengthen the M&E Capacity of NMCP Findings  SMP seconded an M&E specialist to NMCP for 3 years. She was effective at training staff and increasing production of strategic information (e.g. MPR Report). However, by the time the M&E specialist left, there was no counterpart within NMCP to carry on the work. NMCP is now too weak to undertake M&E  Since year 3, half of NMCP staff have left without replacement.  SMP support for the MoH Resource Center (RC) has helped develop the national HMIS Conclusion  NMCP understaffing has impacted on NMCP participation in SMP supported activities in general. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 30 Recommendations  A “flag bearer” or “country champion” is needed to advocate for raising the status of malaria control in Uganda. Finding and supporting such a flag bearer should be one type of BCC intervention. USG projects should support the strengthening of linkages9 between NMCP and other divisions of the MoH (Resource Center, MCH, and Quality Assurance). These other divisions will be able to implement malaria-related activities in a way that complements the role of the NMCP.  GOU or its development partners should again fund the secondment of a seasoned M&E specialist to the NMCP. This secondment should depend upon the staff achieving concrete deliverables. The appropriate deliverables should be identified on the basis of a needs assessment of the NMCP. The evidence The Cooperative Agreement notes that “M&E has historically been weak within the NMCP.” “STOP Malaria will second to NMCP a qualified M&E specialist with particular strengths in database management, statistics, and mapping/GPS data… One of the specialist’s first activities will be to support finalization of the M&E plan (with a costed implementation plan)…” SMP will “Build the capacity of NMCP/MOH staff on topics such as data management, use of mapping software; data interpretation including secondary analysis of the 2008 MIS/AIS data set.” The M&E specialist is also to support coordination and standardization of monitoring tools/indicators and survey coordination – areas which the CA identifies as in need of particular attention. Before the secondment, a MOU was to be signed with MOH that establishes the intention of the MOH to take over this position by FY 2011.” In year 1, SMP seconded an M&E officer to the NMCP, provided 7 laptops and office furniture and supported installation of a local area network in the NMCP offices. The project subsequently funded the installation at the Resource Center of the server now used for the DHIS2 database. SMP’s year 1 Annual Report notes that “The M&E capacity, skills and training needs of NMCP staff were documented ... Data sharing was deemed inadequate, and the staff was unable to utilise the available data.” “... there was a lack of tracking of support supervision reports.” “NMCP’s interest in using available HMIS data is still limited due to a general lack of confidence in these data; the NMCP prefers to collect its own data during district visits” During year 2, 11 NMCP staff were trained in supportive supervision. However, due to “Inadequate staffing at NMCP to cope with competing priorities”, only 13 of the 23 districts were reached with the new supportive supervision approach.” By year 3, however, the project reported that 100% of district health teams received a supervisory visit from national or zonal NMCP personnel in the past year. During the third and final year of her contract, the SMP-seconded M&E specialist supported analyses of malaria related data from the previous 10 years. The findings from this analysis informed the 2011 Malaria Programme Review which was largely financed by SMP. The M&E Specialist also supported the development of the NMCP M&E Plan, 2011 – 2014 which was finalized in year 4 of the project. By the time that the M&E specialist left in 2011, there was no counterpart within NMCP to continue her work. The aide memoire of the Malaria Programme Review drew attention to the limited capacity and stature of the National Malaria Control Programme: 9 The evaluation team owes this insight to the former M&E specialist. She noted that some other divisions of the Ministry of Health had capacities and were interested in supporting implementation of malaria control activities even when the NMCP itself was unable to make sufficient progress with those activities. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 31 “The positioning of the NMCP within the MoH organogram is low. The implication of this is a restricted decision space on policy, technical and resource allocation matters. It minimizes the mandate and authority of the programme to properly head and guide malaria policy and implementation activities…” Since this was written in 2011, the capacity of the NMCP has further weakened as 6 of 11 staff have left the program without replacement. SMP has endeavoured to assist the NMCP for the last two years of the project by supporting almost quarterly meetings of the Roll Back Malaria partnership. “These meetings have strengthened the RBM partnership in the country and have been used as avenues to advocate for malaria issues.” SMP informed the evaluation team that they have also collaborated with the VOICES III project to form a high level malaria advocacy group in Uganda. SMP staff noted that there is need for further support of this activity once VOICES concludes at the end of this year. To the strengthening nationwide of the Health Management Information System (HMIS) of the MoH, SMP has supported the Resource Center of the MoH to develop and print harmonized HMIS tools. The project has SMP has attempted to support NMCP supervision visits to districts. The project reports that NMCP have visited close to 100% of districts in the last 12 months. The evaluation team found that NMCP visited only 4 of 9 districts in the last 6 months (and only 1 had a report on the supervision). 6.2 Strengthen capacity at district level for malaria M&E Findings SMP trained records assistants and facility in-charges at more than 80% of health facilities in data quality assessment. However, the available evidence shows that improvements in data quality have been limited. SMP succeeded at developing the M&E capacity of district biostatisticians and the data use practices of health facilities. SMP also developed some other aspects of district capacity. Conclusion SMP training improved data reporting, timeliness and accuracy. However, SMP planning was not well integrated with district planning and tended to by-pass the constraints of district capacity. (NOTE: The original scope of work did not define how the project was to strengthen district capacity for management of malaria control other than M&E). Recommendations Plans for future support of supervision and other malaria control activities need to be integrated into district planning, budgeting and the existing supervisory processes. For nationwide impact, USG should find ways to provide additional support to the Resource Center (RC) of the MoH for further development of the DHIS. The DHIS software should be configured to reduce entry of inconsistent data. District biostatisticians should be trained and encouraged to regularly download the disaggregated DHIS data and review it to identify health centers with inconsistent data (e.g. ANC1 < IPT2; reported malaria tests performed < reported positive malaria tests) as well as with low performance (i.e. low IPT2 uptake or low testing ratio). This will go a long way to improve data quality and targeted support supervision for weak performing health facilities. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 32 Evidence Summary of progress as reported by the project SMP’s year 2 Annual Report noted that “Data is rarely or not used at all at the primary generation sites (health facilities). This means that decision making in these facilities is not effective since it is not based on evidence, this hugely affects implementation of project activities in these areas.” Project interventions included  data management trainings for district biostatisticians and district HMIS focal persons;  purchase of GPRS modems for district HMIS offices;  training of 4057 staff at health facilities in the use of the revised HMIS tools;  development with the Resource Center of guidelines for data quality assessment (DQA) and manuals for training in data demand and use (DDU);  Training of total of 2,788 health facility staff (in-charges and facility records assistants), district biostatisticians and district HMIS focal persons in Data Quality Assessment (DQA) and Data Demand and Use (DDU);  data quality assessments were then carried out As shown in Figure 26 and Figure 27, graphs presented in SMP annual and quarterly reports suggested some improvement since year 3 in the completeness and timeliness of district reporting to national level. Figure 26 Figure 27 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 33 The project also monitored data usage by assessing whether current graphs of malaria data were displayed on the walls of health facilities. Every health facility was provided with pre-printed graphs for key indicators to plot charts and hang on the notice boards for easy reference. With this intervention, SMP documented some increase in data use from 37% of health facilities in Q4 of year 4 to 49% in Q1 of Year5. Findings from ISS data During each round of integrated supportive supervision, supervisors reviewed OPD registers, counted the number of malaria cases registered during a month and compared this to the number of malaria cases reported by the health facility to the district and national level. Figure 28 shows ISS findings on the percentage of health facilities in SMP-supported districts for which the difference between registered malaria cases and reported malaria cases was 5% or less. During each round of supervision for the last 3 years, between 30% and 40% of health facilities were found to have reported inaccurate data. Little progress appears to have been made with this ISS indicator. ISS has also assessed hospitals, HCIVs and HCIIIs for the presence of trained records assistants. Figure 29 suggests that there has been a modest increase in the availability of this cadre over the last 2 to 3 years. Findings from review of HMIS data Evaluators reviewed the last 12 months of HMIS data from a sample of 6 SMP-supported districts and 6 non-SMP-supported districts. Data were assessed for completeness of report submission (whether any report at all was submitted) and consistency of the data (whether the report had ANC1> IPT2; IPT2>IPT1; malaria tests > malaria positives; and malaria cases > positive malaria tests). In both SMP￾supported districts and non-SMP-supported districts, the great majority of health facilities (93% of SMP￾supported facilities versus 89% of non-SMP-supported districts) submitted at least 11 monthly reports during the last 12 months. Findings concerning the consistency of the data are shown in Figures 30 and 31. Figure 28 Figure 29 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 34 4% 51% 45% Distribution of SMP-supported health facilities by % of HMIS reports without inconsistencies, 09/12-08/13 <50% 50%_89% 90%-100% What these figures show is that in SMP-supported districts as well as in non-SMP-supported districts there are some health facilities which report inconsistent data on at least half of the entire monthly reports they submit. Such poorly performing health facilities, shown as the red slice of each pie, are less common in SMP-supported districts (4% of health facilities) than in non-SMP-supported districts (7% of health facilities). Conversely, facilities which seldom report inconsistent data, shown as the green slices, are as common in SMP-supported districts (45% of health facilities) as they are in non-SMP-supported districts (47% of health facilities). To explain the large number of health facilities submitting at least one report in the last year with inconsistent data, SMP staff noted that “The quality of data is greatly affected by the lack of HMIS primary Tools including health facility registers. These have been lacking for the last 3 years in most of the SMP districts since SMP was given funds to print HMIS Tools for 5 out 34 districts by PMI/USAID. In year5 SMP embarked on supporting districts to conduct data quality assessments and we believe that this will improve the situation with time once uniform tools are available in all health facilities since HMIS under review.” Findings from the evaluation field visits Evaluators interviewed district biostatisticians in each of the 9 districts visited. The biostatisticians, in non-SMP-supported districts as well as SMP-supported districts, appeared skilled and motivated. They all were able to access data from the DHIS and had all done some analysis of the data to produce graphs. Biostatisticians in SMP-supported districts were familiar with the HMIS-strengthening activities sponsored by the project and felt that these were helping to improve data quality and data use. Findings concerning the project’s impact on other aspects of district capacity to manage malaria control activities The SMP Cooperative Agreement observes that “If scaled-up service delivery and increased client demand is to be sustained, institutions at the district and national levels must continue providing inputs and support well past the end of this project …” This raises the question of the extent to which SMP has strengthened district capacity to manage malaria control activities (beyond the strengthening of M&E/HMIS). SMP reports note that the project has developed some other aspects of district capacity: 1. ISS has developed the technical capacity of individual district supervisors in supervision and mentoring lower level staff. Figure 31 Figure 30 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 35 2. SMP’s approach to distribution of ANC LLINs depended upon districts to warehouse the nets and manage their periodic supply to health facilities using district vehicles. This built up district capacity for such logistics work. In important respects, however, SMP planning was not fully integrated into district planning and budgeting. The project often by-passed the constraints of district capacity – e.g. limited vehicles, SDA, etc. The District Health Officer of Masaka District said to one of the evaluators that, “Unlike other projects, SMP did not work through the district planning process” 6.3 Strengthen District Supportive Supervision Findings  From years 3 to 5, SMP spent about $2 million on quarterly “Integrated Supportive Supervision” (ISS) of 50% to 70% of HC’s in SMP districts.  ISS enhanced district quarterly review meetings including the DHTs and health centre in-charges and laboratory staff to improve service delivery Conclusions  ISS built the capacity of individual district staff for malaria supervision and “mentoring”  ISS was depended on SMP for vehicles/SDA and therefore not sustainable by districts.  There is insufficient evidence to show that ISS mentoring improved health worker knowledge and practice. Recommendations  Supervision checklists should include questions to objectively assess health worker knowledge.  Supervision checklists need to be broadened to provide for supervision of other health services in addition to malaria prevention, diagnosis and treatment (i.e. supervision of management of other causes of febrile illness).  Plans for future support of supervision and other malaria control activities need to be integrated into district planning, budgeting and existing supervisory processes. Evidence The Cooperative Agreement summarized some of the constraints to effective supervision by district staff of service delivery. “Major constraints at district level responsible for this include lack of prioritization and poor planning, lack of transport, reliance on allowances before a supervisor moves out of station, lack of supervision checklists and lack of mentoring skills among supervisors leading to health workers being fearful of supervision.” To strengthen supervision, the Cooperative Agreement indicated that “STOP Malaria will provide supplementary support such as safari-day allowances and accompanying supervision teams occasionally to mentor team members and ensure that the scheduled support supervision occurs.” However, no mention was made in the Cooperative Agreement of using project vehicles for such supervision. “Once a supervisor is proficient, the trainers [will] not need to accompany the supervisors to the health facilities unless they expressed a particular need for assistance … Thus the budget for supportive supervision reduces over time… our teams will be able to work with the district supervisors to advocate for regular funding for support supervision, thus addressing the issue of sustainability.” During years 1, district staff were trained in supportive supervision, but “most of the 13 districts did not have funds to support regular support supervision visits to health facilities by district level supervisors. Thus, it was not possible during the first year for the supervisors to conduct follow up visits...” after training. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 36 “Districts do not regularly conduct support supervision to the lower level health facilities due to lack of adequate funding. …supervision specific to malaria services is not routinely included in the district plans.” In Q4 of year 2, the integrated supportive supervision tool was piloted in 20 of 23 districts. Beginning in Q1 of year 3 and continuing to the present SMP has conducted ISS was conducted in all 34 districts, visiting 50% or more of all health facilities since Q3 of year 3 (Figure 32). “Every quarter, SMP supervises all hospitals and HC IVs while lower level facilities (HC III and HC II) receive at least two rounds of support supervision in a year.” SMP reports note that ISS focused on “mentoring” of facility staff and the experience developed the individual skills of district staff in malaria supervision. It is remarkable that none of the 5 SMP districts health offices visited by the evaluation team could provide copies of completed checklists or summaries of key findings from ISS for each health facility. One district Malaria focal person showed copies of matrices of scores for each health facility that had been feedback from SMP. However, the district health teams interviewed by the evaluation team could not produce summaries of action points or other issues that needed to be followed up after ISS. As noted in the section related to strengthening treatment of malaria, the ISS supervision checklist does not include a sufficient number and variety of well-defined criteria with which to objectively assess and track health worker knowledge and practice. The items included do not permit ISS data to be used to track progress with health worker knowledge and skills. More checklist items are needed to facilitate mentoring on important aspects of care (e.g. assessment of danger signs, lab confirmation of malaria diagnoses, adequate exam for other causes of febrile illness, etc…). The checklist focuses exclusively on supervision of malaria prevention and treatment services and does not attempt to assess any other services provided by health facilities. “SMP conducted integrated support supervision with all SMP partners (JHU, Malaria Consortium, IDI and CDFU).” ISS has depended entirely upon the project for all transport and all SDA. NMCP endorsed the ISS checklist. There is no documentation of the checklist or the approach being used in any non-SMP districts. “Although SMP’s desire is for districts and NMCP to conduct quarterly ISS as per the national support supervision guidelines, SMP’s experience is that neither the districts nor NMCP are able to execute this mandate (especially ISS focused on malaria services) without SMP funding. There is a need to advocate to the districts and NMCP to include malaria specific ISS to districts within their annual work plans and budgets.” 7. HOW THE PARTNERSHIP BETWEEN JHU, MALARIA CONSORTIUM, IDI, CDFU, UHMG WORKED IN TERMS OF COST EFFECTIVENESS AND PROCESS/IMPLEMENTATION EFFICIENCY Findings from SMP annual and quarterly reports Delay in recruitment of the Chief of Party contributed to delays in implementation and spending during the first 2 quarters of the project. Once the current Chief of Party arrived in month 8 of year 1, implementation speeded up considerably. Figure 32 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 37 As discussed in the section related to BCC activities, due to circumstances beyond the control of the project, there were repeated changes in the community mobilization strategy. These changes made CDFU’s work challenging. During year 4, “There was a gap in the distribution of nets to eight districts in the Central Region for several months at the beginning of 2012 due to challenges in amending the Year 4 sub-contracts with the Uganda Health Marketing Group (UHMG)… This contributed to not meeting the target for pregnant women receiving LLINs at their first ANC visit.” “Lastly, in August 2012, Malaria Consortium discovered 6,047 LLINs missing out of the 708,650 LLIN received in March/April 2011. Malaria Consortium reported the missing nets to SMP and filed a report with the local authorities.” Findings from interviews with the representatives of the partner organizations The SMP partnership included: JHU CCP – managing the project, M&E, ANC LLIN distribution, supervision, BCC , Malaria Consortium - technical oversight, strengthening malaria case management, IPTp and LLIN distribution, IDI – strengthening malaria microscopy, CDFU - community-mobilization and UHMG – (sub-contracted by Malaria Consortium) to distribute LLINs in 8 districts Each partner was asked to comment upon the effectiveness of the partnership and their relationship with other partners. They uniformly testified that there was a smooth, effective and mutually respectful working relationship among the partners. Each partner was seen to have complementary expertise and capacities and was given an appropriate, well defined role. As noted in the section dealing with ISS, “SMP conducted integrated support supervision with all SMP partners (JHU, Malaria Consortium, IDI and CDFU).” Evaluators hypothesized that ISS thus helped to integrate the partners through joint field work. Findings from review of SMP budgets and financial information With delays of some months in project start-up, SMP expended only 46% of the obligated budget. During years 2, 3 and 4, the project was able to expend 70% or more of considerably larger obligated budgets.10 Findings from overall assessment of project implementation and monitoring SMP succeeded in conducting and completing a large number of activities at national level (development of policies and guidelines) as well as in the 34 focus districts. The work appears to have been monitored closely and reported on in suitable detail. SMP was implemented according to plan and successfully completed a large number of activities at national level (policies, guidelines) and in the 34 focus districts. Conclusions The evaluators conclude that the project has been effectively managed. In particular: The SMP partners worked well together. Their capacities and roles were complementary and respected by each other:  CCP provided effective overall management and M&E for the project; 10 The evaluation team was unable to interpret the budget information for year 5 which shows that the project was obligated more than 3 times as much as in the work plan budget in order to fund sub-awards several times what has been awarded during previous years. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 38  MC played an effective role with malaria technical oversight;  IDI did an effective job strengthening malaria microscopy;  CDFU was effective with community mobilization through Health Assistants; and  UHMG complemented the partnership with its experience in distribution of commodities. The quarterly coordination meetings enhanced the partnership and provided a platform for joint planning for SMP interventions. Progress at national level was constrained by the increasing limitations of the NMCP Recommendation  The USG should use a similar partnership for implementing future malaria control programs. 8. THE FACTORS ASSOCIATED TO THE MAJOR SUCCESSES AND PERFORMANCE WEAKNESSES 8.1 Key strengths Each of the implementing partners had had a long experience in the areas they were focusing on (i.e. MC for case management and LLIN distribution, IDI for training, and CDFU for BCC activities). The level of expertise of each partner was high and SMP benefited from the previous experience of the partners working in other areas on similar interventions. The coordination framework (quarterly meetings and mutual respect) helped the partnership to make joint planning and have a platform for review of progress and common understanding of the project interventions. 8.2 Major Weaknesses  SMP was not in direct control of LLINs and drug supply. This limited progress on these interventions (e.g. LLIN distribution did not start until Year 3).  Use of outdated job aids that do not reflect the new malaria treatment guidelines plus the clinical guidelines that are quite bulky and not user-friendly.  Due to new recruitments and turnover, large scale trainings in IPTp and laboratory diagnosis of malaria now need to be repeated.  SMP did not directly work within the district planning and budgeting framework (by-passing the district capacity limitations). This brings sustainability/continuity of SMP ISS into question.  The positioning of the NMCP within the MoH organogram is low. The implication of this is a restricted decision-making space on policy, technical and resource allocation matters. It minimizes the mandate and authority of the NMCP to properly head and guide malaria policy and implementation activities.  Since year 3, half of NMCP staff have left without replacement. NMCP understaffing has impacted on NMCP participation in SMP supported activities in general. Report of the final evaluation of the USAID/Uganda Stop Malaria Project 39 ANNEX 1 – THE CALENDAR FOR THE EVALUATION No Activity Time Frame (Weeks) Responsible 1 2 3 4 5 6 7 1 In-briefing by USAID: Introduction of the evaluation team, discussion of the SOW and initial presentation of the proposed evaluation work plan 09/2 Consultants 2 Initial meeting with SMP 09/3 Consultants 3 Submission of draft electronic inception report to USAID 09/8 Consultants 4 Pretest instruments in Mukono 09/9 Consultants 5 Data collection in Central region (Kayunga), 09/10-13 Fred Matovu 6 Data collection in Central region (Masaka and Lyantonde) 09/10-13 Festus Kibuuka 7 Data collection in Central region (Mpigi and Mityana) 09/10-13 Robert Pond 8 Data collection in Eastern (Kumi and Palsisa) 09/16-19 Fred Matovu 9 Data collection in Mid-Western (Hoima) 09/16-19 Festus Kibuuka 10 Data collection in Mid-W (Kyenjojo) 09/16-19 Robert Pond 11 Interviews with Kampala Stakeholders 09/23-27 9/30 -10/4 Consultants 12 Interviews with USAID/Kampala 09/23-27 9/30 -10/4 USAID/Consultants 13 Data entry and analysis 09/23-27 9/30 -10/4 Consultants 14 Oral Presentation 10/10 Consultants 14 Submission of draft evaluation report 10/12 Consultants 16 Final Report 10/27 Consultants Report of the final evaluation of the USAID/Uganda Stop Malaria Project 40 ANNEX 2: PERSONS INTERVIEWED No Name Organization/ Designation SMP 1 Abesiga Harriet Technical Assistant Mid-Western Regional Office 3 Asimwe James Technical officer- SMP Mid-Western Regional Office 4 Barbara Evelyn Kunihira M&E officer IDI 5 Basil Tushabe Executive Director CDFU 6 Bright Asiimwe DCOP/M & E Manager Stop Malaria Project 7 Catharine Chime Mukwakwa Chief of Party Stop Malaria Project 8 Dr. Ester Kaggwa Research Monitoring & Evaluation Adviser 9 Dr. Godfrey Magumba Uganda Country Director Malaria Consortium 10 Dr. Mugwanya Edward Team Leader SMP Central Region 11 Dr. Samuel Sudida Gudoi Senior Technical Advisor 12 Dr. Sekabira B. Umaru Deputy Head of Training IDI 13 Dr. Susan Naikoba Head of Training IDI 14 Jim Kamanyo Finance Officer 15 Linda Lukandwa Finance Manager 16 Mugenyi Chris Rwabogo Team Leader SMP Mid- Western Region 17 Namara Linda Data officer IDI 18 Paul Oboth Laboratory Training Coordinator IDI 19 Pherister Nakamya M& E Specialist Uganda AIDS Commission/Formerly with SMP 20 Stella Zawedde Muyanja Technical Trainer IDI USAID/PMI 1 Daryl Martyris USAID SMP AOR 2 BK Kapella PMI/CDC Senior Malaria Advisor MOH 1 Carol Kyozira Principal Bio-Statistician 2 Dr. Edward Mukooyo Assistant Commissioner RC NMCP 1 Agaba Bakita Bosco Epidemiologist 2 Dr. Myers Lugemwa RME Team Leader NMCP 3 Dr. Okui Albert Peter Ag. Programme Manager In each of the 9 districts visited (Mityana, Kyenjojo, Mpigi, Kumi, Kayunga, Pallisa, Masaka, Hoima, Lyantonde) District Health Officer District Malaria Focal Point District Laboratory Focal Point District Health Inspector District Biostatistician At each health facility visited: Report for the final evaluation of the USAID/Uganda Stop Malaria Project 41 No Name Organization/ Designation Kayunga – Nkokonjeru HC3, Kayunga hosp., Nakatovu HC2; Kumi – Kumi hosp., Kumi HC4, Nyero HC3; Pallisa – Pallisa hosp., Butebo HC4, Kibale HC3; Mityana – Mityana hosp., Malangala HC3, Bukkalamuli HC3, Miseebe HC2, Namigaru HC2; Kyenjojo – Kyenjojo hosp., Kisojo HC3, Rwaitengya HC2; Mpigi – Mpigi HC4, Nswanjere HC3, Kibumbiro HC2; Kumi – Kumi hosp., Kumi HC4, Nyero HC3; Kayunga – Kayunga hosp., Nkokonjeru HC3, Nakatovu HC2; Pallisa – Pallisa hosp., Butebo HC4, Kibale HC3; Masaka – Bukoto HC3, Nyendo Senyange HC2; Hoima – Buhimba HC3, Kogoroya HC4; Lyantonde – Lyantone hosp., Kabatema HC2, Mpumudde HC3 In-charge or most senior person available A health worker who manages febrile illnesses A health worker who provides antenatal care The most senior laboratory worker (if there was a lab) The person in charge of the drug store The records assistant (if there was one) A client exiting after treatment of a child with febrile illness A client exiting after antenatal care Report for the final evaluation of the USAID/Uganda Stop Malaria Project 42 ANNEX 3: DOCUMENTS REVIEWED 1. Stop Malaria Project Health Facility Assessment Survey Report September 2011 2. Ministry of Health, The Health Management Information System Volume 3 District/ HSD Procedure Manual August 2010 3. Ministry of Health, Data Quality Assessment Manual, Tools and Guidelines for Implementation 4. Uganda Demographic and Health Survey 2006 5. Uganda Malaria Indicator Survey (MIS) 2009 6. Uganda Demographic and Health Survey 2011 7. Uganda Joint Behaviour Change Communication Survey, October 2012 8. Uganda National Malaria Control Policy June 2011 9. President’s Malaria Initiative Uganda Malaria Operational Plan (Mop) FY 2007 10.President’s Malaria Initiative Uganda Malaria Operational Plan (Mop) FY 2008 11.President’s Malaria Initiative Uganda Malaria Operational Plan For FY 2009 Final Submitted November 12, 2008 12.President’s Malaria Initiative Uganda Malaria Operational Plan For FY 2010 Draft November 2009 13.President’s Malaria Initiative Uganda Malaria Operational Plan For FY 2011 Final, November 23, 2010 14.President’s Malaria Initiative Uganda Malaria Operational Plan For FY 2012 September 20, 2011 15.President’s Malaria Initiative Uganda Malaria Operational Plan FY 2013 16.The Uganda Stop Malaria Project Annual Report Y1:September 26, 2008 – September 30, 2009 17.The Uganda Stop Malaria Project Annual Report Y2:October 01, 2009 – September 30, 2010 18.The Uganda Stop Malaria Project Annual Report Y3:October 31 2010 – 30th September 2011 19.The Uganda Stop Malaria Project Annual Report Y4 October 1st, 2011– September 30TH, 2012 20.The Uganda Stop Malaria Project Quarterly Performance Report Year 4 October 1ST, 2011– September 30TH, 2012 21.The Uganda Stop Malaria Project Quarterly Performance Report October 1st – December31st2012 22.The Uganda Stop Malaria Project Quarterly Performance Report January 1st – March 31st2013 23.The Uganda Stop Malaria Project Quarterly Performance Report April 1st – June 30th 2013 24.The Uganda Stop Malaria Project Work plan Year 1: October 2008 - September 2009 25.The Uganda Stop Malaria Project Work plan Year 2: October 2009 - September 2010 26.The Uganda Stop Malaria Project Work plan Year 3: October 2010 - September 2011 27.The Uganda Stop Malaria Project Work plan Year 4: October 2011 - September 2012 28.The Uganda Stop Malaria Project Work plan Year 5: October 2012 - September 2013 29.Assessing Malaria Treatment and Control in Selected Health Facilities 4th Quarter support supervision report October 2010 30.Assessing Malaria Treatment and Control in Selected Health Facilities 2009 4th Quarter Support Supervision Report July 2010 31.Integrated Support Supervision Report Year 3, Quarter 2 March 2011 32.Integrated Support Supervision Report Year 3, Quarter 3 May 2011 33.Integrated Support Supervision Report Year 3, Quarter 1 December, 2010 34.Integrated Support Supervision Report Year 3, Quarter 4 December 2011 35.The Uganda Stop Malaria Project Districts Health Facility Integrated Support Supervision Report Year 4, Quarter 4 36.The Uganda Stop Malaria Project Districts Health Facility Integrated Support Supervision Report November–December2011 37.The Uganda Stop Malaria Project Districts Health Facility Integrated Support Supervision Report Year 4 Quarter 2 2012 38.The Uganda Stop Malaria Project Districts Health Facility Integrated Support Supervision Year 5, Quarter 1 Report January 2013 39.The Uganda Stop Malaria Project District Health Facility Integrated Support Supervision Year 5, Quarter 2 report June 2013 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 43 40.Ministry of Health Monitoring & Evaluation Plan For National Malaria Control Strategic Plan 2010/11 – 2014/15 41.Uganda National Malaria Control Strategic Plan 2010/11 - 2014/15 42.Stop Malaria Project Journey for the past 5 years Achievements, Challenges and Recommendations 43.The Uganda Stop Malaria Project Performance Monitoring Plan Prepared and Submitted to USAID on 9 January, 2009 44.The Uganda Stop Malaria Project Performance Monitoring Plan Prepared and Submitted to USAID Revised January 18, 2012 45.The Uganda Stop Malaria Project Performance Monitoring Plan 2008 – 2013 Prepared and Submitted to USAID Revised January 31, 2013 46.The Uganda Stop Malaria Project Performance Monitoring Plan Prepared and Submitted to USAID on 9 January, 2009 Revised May 2009 47.Integrated Management of Malaria Training Facilitator’s Manual National Malaria Control Programme (NMCP) Ministry of Health March 2012 48.Integrated Management of Malaria Training A Practical Guide For Health Workers National Malaria Control Programme (NMCP) Ministry of Health March 2012 49.Routine Distribution of Long Lasting Insecticidal Nets through ANC Implementation Guide for Managers at District and Health Sub District Levels National Malaria Control Programme, Ministry of Health, Uganda, 2011 50.Routine Distribution of Long Lasting Insecticidal Nets through ANC Implementation Guide for National Planners Malaria Control Programme, Ministry of Health, Uganda, 2011 51.Routine Distribution Of Long Lasting Insecticidal Nets through ANC Implementation Guide for Practitioners Malaria Control Programme, Ministry of Health, Uganda, 2011 52.Health Management Information System Data Management, Demand and Use Health Facility Trainers’ Manual April 2012 53.Health Management Information System Data Management, Demand And Use Trainers’ Manual April 2012 54.The Uganda Stop Malaria Project District Health Facility Integrated Support Supervision Year 5, Quarter 2report June 2013 55.Management of Severe Malaria: A Practical Handbook Third Edition World Health Organisation 56.Quinine, an old anti-malarial drug in a modern world: role in the treatment of malaria Jane Achan1*, Ambrose O Talisuna2, Annette Erhart3, Adoke Yeka4, James K Tibenderana5, Frederick N Baliraine6,Philip J Rosenthal6 and Umberto D’Alessandro3 57.Uganda Country Development Cooperation Strategy 2011-2015 58.WHO 2009: A strategic framework for malaria prevention and control during pregnancy in the African region. World Health Organisation Geneva, AFR/MAL/04/01 59.SMP Annual Budgets 60.Malaria Program performance Review (MPR) report May 2011 61.LQAS Community Survey report, 2012 62.SMP Support supervision (ISS) tool Report for the final evaluation of the USAID/Uganda Stop Malaria Project 44 ANNEX 4 – LOCATION OF THE CLUSTERS OF THE 2006 DHS, THE 2009 MIS AND THE 2011 DHS Report for the final evaluation of the USAID/Uganda Stop Malaria Project 45 ANNEX 5 – STATEMENT OF WORK USAID/Uganda: Stop Malaria Project (SMP) Evaluation Background The President’s Malaria Initiative (PMI)/Uganda’s Stop Malaria project (SMP) was established as a flagship project to increase coverage and use of key life-saving malaria interventions in support of the Uganda National Malaria Strategy and the National Malaria Control Program (NMCP). This project was to geographically complement the Northern Uganda Malaria, AIDS, and Tuberculosis project (NUMAT), and be able to work in all areas of Uganda except those supported through NUMAT. NUMAT has now closed but its activities continue to be implemented through the new Northern Uganda Health Integration to Enhance Services (NUHITES) project. The objective of SMP is to support the goals of the national malaria program; specifically to achieve Uganda’s Presidential Malaria Initiative (PMI) targets in prevention and treatment of malaria. The main three results under this objective were to: 1. Improve and implement malaria prevention programs in support of the National Malaria Control Strategy. 2. Improve and implement malaria diagnosis and treatment activities in support of the National Malaria Control Strategy. 3. Improve national capacity of the National Malaria Control Program to monitor and evaluate malaria interventions. The abovementioned main results are further subdivided into intermediate results, whose expected targets, level of effort expected and indicators are outlined in the project proposal, work plans and PMPs. Purpose of Assignment The purpose of this evaluation is to provide evidence to guide strategic targeting and investment for future USAID/Uganda malaria interventions. Evaluation Questions The evaluation will answer the following questions: 1. To what extent has SMP improved delivery of key (global standard) malaria interventions in the districts of operation? Key intervention areas cover preventive, curative and systems strengthening. 2. What are the factors associated to the major successes and performance weaknesses? 3. How well did the partnership between JHU, Malaria Consortium, IDI, CDFU, UHMG work in terms of cost effectiveness and process/implementation efficiency? 4. How effectively is the project building the capacities of NMCP and districts as laid out in the cooperative agreement and the monitoring and evaluation (M&E) framework, and with regard to improving capacity to properly manage malaria control in Uganda? Methodology The evaluators should consider a range of possible methods and approaches for collecting and analyzing the information required to answer all evaluation questions. The methodology will be discussed with and approved by USAID/Uganda prior to implementation. The methodology may include document review, secondary data analysis, key informant interviews, Focus Group Discussions (FGDs), client exit interviews and observation of service delivery as necessary. Team is encouraged to use the sample evaluation design matrix provided in Annex 1. Document Review Evaluation team will acquire and review key documents prior to the start of in-country work. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 46 Documents include:  SMP request for applications (RFA)  SMP Program Description in the cooperative agreement  SMP work plans  SMP quarterly and annual reports  PMI malaria operational plans (MOPs)  USAID/Uganda CDCS and other strategy documents  USAID/Uganda and USG/Uganda operational plans (OPs)  GOU health strategies, policies, guidelines, protocols  Uganda’s 2009 malaria indicator survey (MIS) report  Uganda’s 2011 demographic and health survey (DHS) PMI Uganda may be contacted to source some of these documents Interviews Key Informant Interviews may include  USAID/Uganda health team  PMI team  GOU staff, including national level (MOH, NMCP), district and facility level  SMP staff, including prime recipient and sub-recipients  Participants of SMP training/supervision programs  Health facility staff and clients at SMP target health facilities Other The evaluation team may implement direct observation of SMP activities and/or service provision at SMP-assisted districts and health centers. Team Composition, Skills, and Level of Effort (LOE)  Evaluation team will be composed of three consultants: One Team Leader, one malaria program expert and one organizational capacity expert  Team Leader will be a senior evaluation expert with over 15 years of experience evaluating and/or implementing health programs and with knowledge of and experience in malaria programs. Team Leader will have played substantive role in more than five other evaluations and played team leader role in a minimum three related evaluations  The Malaria Program expert will be a holder of an MPH with extensive malaria programming experience in Sub-Saharan Africa and skills in monitoring and evaluation of malaria programs. S/he will have played a substantive role in a minimum of three evaluations in health  The Organizational Capacity Expert will be a holder of a master’s degree in development studies, business administration, health economics or other relevant social sciences and demonstrated skills/experience in assessing organizational capacity and partnerships Performance Period Evaluation is scheduled to begin in the last week of August 2013 and be completed not later than end of October 2013. Deliverables and Products 1. In-brief: introduction of the evaluation team, discussion of the scope of work and initial presentation of the proposed evaluation work plan, and presentation of the inception report detailing the team’s interpretation of the assignment, an evaluation design and methodology, analytical plans, sampling, interview tools, field visits and work schedule. 2. Weekly progress reports: brief informal reports summarizing progress, challenges and constraints and describing the evaluation team’s response 3. Oral presentation: power point presentation. The oral presentation should, at a minimum, cover the major findings, conclusions, recommendations, and key lessons. The evaluation team Report for the final evaluation of the USAID/Uganda Stop Malaria Project 47 will liaise with the mission to agree on the dates, audience, venue and other logistical arrangements for this briefing. The presentation will be held before departure. 4. Draft evaluation report: the report should comply with the USAID’s evaluation report standards set out in Annex 2. 5. Final draft report: complete report incorporating comments from USAID and other stakeholders. 6. Final report: Team Leader will submit a final report incorporating final edits and formatting for wider sharing. 7. Cleaned data sets: The contractor will share the cleaned data sets with USAID. Report for the final evaluation of the USAID/Uganda Stop Malaria Project 48 ANNEX 6 - ASSESSMENT OF THE REPORT OF THE BCC SURVEY The report of the October 2012 Uganda Joint BCC Survey raises as many questions as it answers. 1. Most importantly, why weren’t more robust methods used to evaluate the various multi￾million dollar BCC campaigns? Much more persuasive evidence would be available if household knowledge, attitudes and practices (KAP) had been assessed prior to and following the BCC campaigns and assessed in both intervention and control communities. Such methodologies would have permitted straightforward judgments to be made regarding a) improvements over time; and b) improved outcomes in intervention vs. control communities. While it is true that non-project communities were likely exposed to some of the mass media, the BCC campaigns were geographically targeted. Hence, residents of project communities were more likely to have been exposed and the impact of these BCC interventions could best have been judged by comparison of intervention and non￾intervention districts. 2. Why didn’t those who analyzed the data at least attempt to assess differences between project districts and non-project districts in the extent of exposure and in KAP outcomes? 3. In the absence of a more robust comparison group (i.e. baseline or control communities) analysis focused on the association between reported exposure to BCC messages and reported KAP outcomes. The results from analysis of the crude data are presented in table 41. From the statistics presented, it appears that the p-values appearing in Table 41 were estimated by assuming that the sample was selected using simple random sampling. This is not appropriate given that cluster sampling (15 subjects per village x 540 villages) was used. The effect of this cluster sampling is that the effective sample size was smaller (by a factor equal to the “design effect”) and the true p-values larger than those reported in Table 41. 4. As implied by the report, reported exposure provides an imperfect measurement of true exposure to the BCC interventions. In research parlance, reported exposure is not an ideal “independent variable”. This is because confounding influences can affect the association between reported exposure and reported KAP outcomes. The analysts use various statistical methods (Propensity Score Matching and logistical regression) to attempt to limit the influence of these confounders. However, none of these methods addresses the most important limitation of the independent variable: people who report hearing a BCC message may be biased to also report compliance with the message. For example, some of those who report that they heard a message promoting malaria testing might be reluctant to admit that they did not get a malaria test the last time that they had malaria. Conversely, people who recently got a malaria test might be more likely to recall hearing a BCC message promoting this practice. 5. The report provides no explanation of the complex multivariate analytic methods used. There are multiple alternative ways to carryout either propensity score matching or logistic regression. Some ways are more prone than others to yield misleading results. The analytic methods must be described before we can have confidence in the reported results. For Report for the final evaluation of the USAID/Uganda Stop Malaria Project 49 example, how was cluster sampling taken into account? As noted above, standard error calculations that ignore clustering generate under-estimates of p-values. But the problem does not stop there. As noted by Li et al11 “… ignoring the multi-level structure [i.e. cluster sampling] in both stages of the propensity-score-weighting leads to severe bias in estimating the average treatment effect…”. This is because the influence of any uncontrolled confounders which are associated with the clustering (e.g. exposure to non-project interventions) can be heightened by propensity score matching. 6. The significance of some of the reported findings cannot be interpreted without reviewing the questionnaire. For example, what is meant by “seeking treatment for fever”? Does this include care sought from a pharmacist or care sought from a community medicine distributor? 7. There are numerous unexplained inconsistencies in the sample sizes and some other statistics cited in the report: a. Why are the sample sizes after propensity score matching (see the graphs on pages 83 and 84) consistently larger than the sample sizes of the crude data (see Table 41)? This is likely the result of the mathematical modeling involved in generating data that were propensity score matched. However, if the resulting artificial sample sizes were used instead of the real sample sizes then this would have resulted in artificially low p-values. b. According to Table 45, 536 subjects were exposed to the Power of Day One Campaign. However, according to Table 46, 481 were exposed; c. According to Table 4, there were 1,350 urban subjects. However, according to Table 42, there 1,440 urban subjects (and why doesn’t this table also show the number of rural subjects and the percentage of them that were exposed?); d. According to Table 4, there were 1,639 subjects sampled in the 6 SMP districts. However, according to Table 48 there were 1,589 subjects in these 6 districts, whereas, according to Table 49, this number was somewhere between 2,785 and 3,261; e. According to the paragraph at the bottom of page 88, 75% of respondents exposed to the United Against Malaria Campaign took some action. However, according to the graph on page 89, 95% of respondents exposed took at least one action. 8. Since the bulk of the United Against Malaria Campaign took place in Kampala, how can it be that exposure in Kampala (28% in Table 42) was lower than in other urban areas (31% in Table 42)? 9. The report should explain how confidence intervals have been adjusted when “data dredging” (making multiple comparisons). When large numbers of tests are performed, some of the tests will produce false results merely as a result of random statistical error. According to the Bonferroni method12, for example, if we are conducting 5 tests, each with an individual confidence interval, and we want a 95% confidence level for the overall set of 5 findings, then the individual confidence interval for each separate test should be 99%. The report notes that the lower end of the 95% confidence intervals for some supposedly 11 Li F, Zaslavsky AM and LandrumMB. Propensity Score Weighting with Multilevel Data 12 Bland M and Altman DG. Multiple significance tests: the Bonferroni method. BMJ 1995;310:170 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 50 significant odds ratios are as low as 1.02. Any adjustment of such confidence intervals to take account of the multiple comparisons might make the 95% confidence interval overlap 1.0 (i.e. zero effect). 10. The report presents odds ratios as if they are equivalent to measures of relative risk. Other than for rare outcomes (probability of less than 5%) such an interpretation overstates the size of effects. For example, on page 100 an odds ratio of 1.91 (Table 50) is misinterpreted to mean that “… respondents exposed to the [Stop Malaria in your Community] campaign were 91% more likely to seek treatment for malaria than those who were not exposed.” If 60% of unexposed persons sought treatment and 74% of exposed persons sought treatment (i.e. 0.74/0.60 = 1.24 or 24% more likely), then the odds ratio would be (0.74/26)/(0.60/0.40) = 1.91. In fact, the report sometimes even misinterprets odds ratios in this way even when the report acknowledges that the 95% confidence interval overlaps 1.0 (for example, see the odds ratio of 1.71 in Table 47 and the last sentence on page 94). 11. In conclusion, without further analysis and without further explanation of the analytic methods, given the apparently modest effect of BCC exposure on most KAP outcomes (87% of those exposed versus 84% of those not exposed reported sleeping under an ITN) it is not possible to reach conclusions beyond those in the Executive Summary of the report: “Exposure to any of these malaria communication interventions was associated with a net increase in testing before treatment of 4% among women and 8% among men… There were no net effects associated with exposure to malaria communication for other key outcomes such as getting treatment within 24 hours of fever onset, and sleeping under a mosquito net.” Report for the final evaluation of the USAID/Uganda Stop Malaria Project 51 Annex 7 Field data collection instruments Questionnaire for district level Start with the District Health Officer and the District Malaria Focal Person then continue with the district biostatistician, the district health inspector and the person responsible for supporting laboratories in the district Name of interviewer ________________________ 2. Date __________________ 001 Name of district __________________ Name of region ___________________ Names of 5 health facilities selected for possible visitation Name of health facility Level (Hospital, IV, III or II) Facility A Facility B Facility C Facility D Facility E Describe the district. When was this district first designated as a district, what is the catchment population and how many health facilities of each type do you have in this district? How many functional vehicles and motorcycles are available to be used by the district health office? 002 Estimated population 003 Number of communities 004 Number of hospitals 005 Number of HCIVs 006 Number of HCIII’s 007 Number of HCII’s 008 Number of active Health Assistants working outside of health facilities 009 Number of Community Medicine Distributors actively distributing anti-malarials Equipment and electricity available to the district health office – How many of the following are available to be regularly used by the district health officer? Equipment / electricity Number 010 Vehicle 011 Motorcycle 012 Computer 013 Printer 014 Average number of hours of electricity per day -including generator Report for the final evaluation of the USAID/Uganda Stop Malaria Project 52 Training – I want to ask you about training related to malaria control that has taken place in this district in the last 5 years. For each type of training please tell me whether it has taken place and if so, when the training took place, and about how many health workers were trained. If the informant does not know the number, record “98” for the number. Training topic Took place in the last 11 months Yes No Took place 12 to 35 months ago Yes No Took place 36 to 59 months ago Yes No Number of health workers trained 021 Integrated malaria management (IMM- diagnosis, treatment of uncomplicated and treatment of severe malaria) 1 2 1 2 1 2 022 What is the target number of health workers to be trained in management of malaria? [How many health facility staff in this district treat malaria?] 023 What % of health facility staff who treat malaria now need refresher training in management of malaria? [if there is time, ask why] 024 Microscopy and RDTs 1 2 1 2 1 2 025 What is the target number of laboratory workers to be trained in management of malaria? [How many staff in this district perform microscopy or RDT tests?] 026 Malaria data quality assessment (DQA) 1 2 1 2 1 2 027 Integrated ANC, LLIN distribution and malaria in pregnancy (MiP) 1 2 1 2 1 2 028 What is the target number of health workers to be trained in IPTp? [How many health facility staff in this district provide ANC care ?] 029 Training of health assistants to work at schools and in the community 1 2 1 2 1 2 030 Training hospital and HCIV staff for clinical audits 1 2 1 2 1 2 031 Other(specify) ___________________ 1 2 1 2 1 2 Name of district: ______________________________ What malaria control activities has your district health office performed in the last 2 years (prompt) Malaria control activity Yes No 015 Supply coartem and other anti-malarials to health facilities 1 2 016 Supply of LLINs to health facilities 1 2 017 Supply of LLINs to sites for mass distribution 1 2 018 Training health staff in malaria case management 1 2 019 Training health staff in IPTp 1 2 020 Supervision of health facilities 1 2 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 53 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 54 Name of district: _________________________________ Supervision of health facilities -- Now, I would like to ask you a few questions about supervision of health facilities performed by district health staff. 032 What percentage of the health facilities in this district have been visited in the last 3 months for technical supervision including supervision of malaria control % of HF’s supervised in the last 3 months 033 What percentage of the health facilities in this district have been visited in the last 12 months for technical supervision including supervision of malaria control % of HF’s supervised in the last 12 months 034 Each time that the district staff go for supervision of health facilities, how many district staff participate? 035 What transport is used for district supervision of health facilities? 1= District vehicle 2= District motorcycle 3= SMP vehicle or motorcycle 4= Vehicle of another project (specify the project name) ____________________ 5= Other (specify) ___________________ 6= No transport is available 036 What is the source of funds for SDA for district staff who go on supervision 1= District budget 2= SMP 3= Another project (specify the project name) ____________________ 4= Other (specify)_____________________ 5= No SDA is paid Do you have any report from supervision of the following health facilities in the last 6 months [Note this may include a copy of the checklist or a summary of the action points from the supervision visit]: Do you have any report from supervision of the following health facilities in the last 6 months? If yes, please show it to me. Yes, observed Yes, not observed No 037 Facility A (read the name from page 1) 1 2 3 038 Facility B (read the name from page 1) 1 2 3 039 Facility C (read the name from page 1) 1 2 3 040 Facility D (read the name from page 1) 1 2 3 041 Facility E (read the name from page 1) 1 2 3 Technical support and supervision from NMCP 042 Have staff from NMCP come to this district in the last 6 months to provide technical support or supervision? 1= Yes 2= No 043 If yes, is there any report of this visit? [ask to see the report] 1= Yes, observed 2= Yes, not observed 3+ No [If there is a report from NMCP, please briefly note on the back of this page any issues discussed] Report for the final evaluation of the USAID/Uganda Stop Malaria Project 55 Name of district: ________________________ The role of SMP 044 Please list the most important ways that SMP has helped strengthen malaria control in this district [Do not prompt for answers. Ask “Any other ways?”. Circle all answers] 1= Support for training 2= Support for ISS 3 = Assistance with supply of drugs 4= Supply of LLINs 5= Quarterly review meetings of health facilities 6= Other _______________ 7= Other _______________ 8 = Other _______________ 045 Which of the following activities has SMP supported in this district? [Prompt for each] 1= Training in IMM 2= Training in MiP 3= Training in DQA 4= Training of HAs 5= Training in clinical audits 6= Integrated supportive supervision (ISS) 7= Data quality workshop for district biostatisticians 8= Quarterly review meetings of health facilities 9= Quarterly review meetings of districts (at zonal level) 046 Will the district be able to continue the current approach to supervision after the SMP project ends? 1=Yes 2=No 047 If no, why not? (Do NOT prompt. Circle all responses) 1= Shortage of vehicles; 2= Shortage of funds for fuel and vehicle maintenance; 3=Shortage of funds for SDA; 4=Other (specify) 5=Other (specify) 048 Please describe how you think supervision will change after the end of the SMP project? 049 When the SMP project comes to an end, do you think it likely that district vehicles can be used for supportive supervision 1=Yes 2=No 050 If not, why not? 051 Did ISS visits often identify problems with a health facility that were beyond the ability of the health facility staff and the ability of district health staff to solve? 1=Yes 2=No 052 If yes, please give examples of these problems that could not be solved at the level of the health facility or at the level of the district health office Report for the final evaluation of the USAID/Uganda Stop Malaria Project 56 Name of district: _____________________ 053 Do you think it is likely that quarterly review meetings for health facilities will continue after the end of SMP 1=Yes 2=No 054 If not, why not? 055 Please describe how you think quarterly review meetings may change after the end of the SMP project? 056 Please tell us about any challenges or problems with the support provided by SMP 057 Have staff of SMP talked with you and other district staff about how malaria control activities might continue or might change after the project ends? 1=Yes 2=No 058 If yes, what has been said about the future of malaria control activities 059 What is the single most important thing that SMP has accomplished? 060 SMP was a partnership of several organizations including the ministry of health. Please tell us what you know about who these partners were and what their roles were? Additional topics to discuss if time permits: LLINs -- What are the major factors constraining use of LLINs by households owning LLINs? What has the district done to help address these constraints? What has SMP done? IPTp -- What are the major factors constraining coverage with IPTp? What has the district done to help address these constraints? What has SMP done Report for the final evaluation of the USAID/Uganda Stop Malaria Project 57 Name of district: _____________________________ Lab confirmation of malaria -- What are the major factors constraining laboratory confirmation of malaria? What has the district done to address these constraints? What has SMP done? Diagnosis and treatment of fever -- What are the major factors constraining prompt and effective management of fever? Are there important constraints to effective management of persons with fever who seek care from a health facility? What has the district done to address these constraints? What has SMP done? BCC – Describe the district has done to improve household knowledge, attitudes and practices related to malaria prevention and treatment. Are Health Assistants active? How many health assistants are active in the community? What has SMP done? What about radio? Home and community access to anti-malarials -- Briefly, what are the constraints to improving home and community-based management of fever and what are the possible ways of moving forward in this area? Are there any active Community Medicine Distributors in this district? Is yes, how many? Who supplies and supervises them? If not, why not? Questions for the laboratory focal person Training – I want to ask you about training of laboratory staff in this district in the last 5 years on the topic of malaria diagnosis. Please tell me whether in-service training has taken place and if so, whether the training was in the last 11 months or 12 to 35 months, and about how many health workers were trained. Training topic Took place in the last 11 months Yes No Took place 12 to 35 months ago Yes No Took place 36 to 59 months ago Yes No Number of health workers trained 061 Training of laboratory staff on malaria diagnosis 1 2 1 2 1 2 062 How many staff in this district perform microscopy or RDT tests? External Quality Assurance (EQA) for laboratory 063 How many of the laboratories at health facilities in this district participated in the external quality assurance (EQA) program that measured the accuracy of malaria microscope slides 064 During the last round of EQA, how many of these laboratories had at least 80% accuracy 065 Why do you think some of the laboratories had less than 80% accuracy? 066 What percent of health facilities in the district now have RDTs in stock? 067 Why do stocks of RDTs sometimes run out? Report for the final evaluation of the USAID/Uganda Stop Malaria Project 58 Name of district: ____________________________ Health management information system / M&E (questions for the biostatistician) Questions for the Health Inspector – I want to ask you some questions about Health Assistants and the work they perform to provide health education in schools and in the community. 072 How many Health Assistants work in this district? 073 In the last 5 years, how many HAs have been trained with support from SMP to provide health education on malaria in schools and in the community? 074 In the last 6 months, what percent of these trained HAs have received supervisory support visits? 075 In the last 3 months, what percent of these trained HAs have submitted quarterly monitoring forms? 076 Please show us a report that has been submitted by an HA in the last 3 months 077 In the last 3 months, how many schools have HAs visited to provide health education on malaria? 078 Please describe any challenges with this work of the HAs and with health education for malaria in this district. After the end of SMP will the district be able to sustain the support of HAs? Yes=1 No=2 Questionnaire for a facility audit 068 Please describe what has been done to improve M&E / HMIS is this district during the last 5 years. Has the M&E capacity of the district improved? If so, in what ways? 069 Over the last 5 years, has SMP organized any training or workshop to develop your skills in management and analysis of malaria data 1= Yes 2=No 070 Please show me any examples of graphs or tables or reports prepared by the district that involved analysis and presentation of malaria data. [If a graph or table is observed, briefly describe it in the space below: 1= Observed 2= Not observed 071 Can you please export some data for me from the DHIS. I would like an excel sheet providing the form 105 data for June 2013 for each of the following 5 health facilities (see page 1). I would also like an excel sheet providing the data for July 2013 for each of the following 5 health facilities (see page 1). [Some tips for obtaining these excel sheets from DHIS: Select “Reports”; then select “Data set report”; then select the month; then select the specific health facility; then export all the data from form 105] 1= Excel sheets provided 2= DHIS could not be accessed (e.g. no power) 3=Biostatistican lacks the skills to obtain the excel sheets Report for the final evaluation of the USAID/Uganda Stop Malaria Project 59 3. Number of observations, exits & service provider questionnaires completed at this facility SERVICE PROVIDER INTERVIEWS (OPD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SERVICE PROVIDER INTERVIEWS (ANC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . EXIT INTERVIEWS (CARETAKERS OF UNDER FIVES) . . . . . . . . . . . . . . . . . . . . . . . . . . EXIT INTERVIEWS (PREGNANT WOMEN) . . . . . . . . . . . . . . . . . . . . . . . . . . Date of interview______________ Name of interviewer_________________ 1. Facility Identification 001 NAME OF FACILITY __________________________________________________ 002 DISTRICT __________________________________________________ 003 SUB-COUNTY __________________________________________________ 004 TYPE OF FACILTY: Hospital HCIV HCIII HCII 009 STAFFING LEVEL Please tell me the number of staff in this facility by cadre. Please also indicate how many of these staff treat malaria and how many of these staff administer IPTp Cadre Number of Staff in cadre Number that treat malaria Number that administer IPTp Medical officer Clinical officer Comp. Nurse Enrolled Nurse Registered Nurse Nursing Aide Enrolled Midwife Registered Midwife Lab technologist Lab technician Lab Assistant Microscopist Health Educator/ Assistant Report for the final evaluation of the USAID/Uganda Stop Malaria Project 60 SECTION 1: INFORMATION ON PERSON PARTICIPATING IN FACILITY AUDIT NO. QUESTIONS CODING CLASSIFICATION GO TO 115 May I begin the interview now? YES . . . . .1 NO . . . . . . . . . . . 2 END 017 How many years have you been working in this facility? IF LESS THAN ONE YEAR, INDICATE HOW MANY MONTHS YEARS MONTHS 018 Are you in charge of this health facility YES . . . . . . 1 NO . . . . . . . . . . . . 2 INFORMATION ON SERVICES HEALTH FACILITY OPERATIONS 024 Does this facility routinely admit patients for overnight/inpatient care? YES …………………………………………….1 N0………………………………………………..2 025 INDICATE HOW MANY BEDS THIS FACILITY HAS FOR ADMISSIONS NUMBER OF BEDS 1025 How many blood transfusions have been provided at this health facility in the last 6 months SUPPORT SUPERVISION 039 Now, I would like to ask you a few questions about external supervision this facility may have received. When was the last time a supervisor from OUTSIDE this facility visited for technical supervision including of malaria control activities? WITHIN THE LAST 3 MONTHS.................……..1 MORE THAN 3 MONTHS AGO……………..…..2 NEVER SUPERVISED FROM OUTSIDE FACILITY.......................................................3 3,→Q41 040 Please show me the supervision book and show me where this most recent visit for technical supervision is recorded. 1= Supervision book is not available 2= Supervision book was observed but there is no record of the most recent technical supervision 3=Supervision book was observed and seen to include a record of the most recent technical supervision 041 Record any technical issue related to malaria control that was noted from the most recent visit for technical supervision. If there is no issue related to malaria control, record at least one issue related to the delivery of health services that was recorded. CASE MANAGEMENT 048 Does the facility have any staff trained in Integrated Management of Malaria (IMM) in the last 2 years? If yes, how many health workers have been trained in the last 2 years? Number trained PUT “00” IF NO ONE WAS TRAINED “98” if unknown 049 Is malaria treatment based on lab diagnosis? YES….1 NO……2 050 Is there a laboratory at this health facility? YES….1 NO……2 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 61 053 Record if the facility has the following guidelines, posters and job aides Available & Observed Available but not Observed Not Available a) Integrated Management of malaria, Practice guide for Health Workers 1 2 0 b) Flow chart on malaria in pregnancy, 2nd edition, December 2005 1 2 0 c) Diagnosis and management of severe malaria (poster printed by MoH with Malaria Consortium, reprinted by SMP) 1 2 0 d) Flow chat on management of malaria, December 2005 1 2 0 e) Other malaria job aide (specify) 1 2 0 PROVISION OF IPT/ANC SERVICES 054 Does this facility provide ANC services? YES …………………………………………………1 NO …………………………………………………2 2,→Q74 059 Does the health facility offer IPTp? YES ………………………………….1 NO ……………………………………2 (Check register or observe the mother taking SP) 1 →Q61 060 Why does the facility not offer IPTp LACK OF SP ……………………………….1 LACK OF CUPS …………………………….2 LACK OF WATER…………………………….3 LACK OF WATER PURIFICATION TABS….4 LACK OTHER SUPPLIES …………………..5 OTHER____________________________97 (SPECIFY) 061 Does the health facility administer IPTp under DOTs? YES, OBSERVED …………………..1 YES, NOT OBSERVED ……………2 NO……………………………………..3 1,2,→Q63 062 Why doesn’t the facility administer IPTp under DOTS? NO WATER ………………………….1 NO CUPS, OTHER SUPPLIES…….2 STAFF NOT TRAINED ……………..3 OTHER _____________________97 (SPECIFY) BEHAVIOUR CHANGE COMMUNICATION 100 How many Health Assistants/ Health Educators work at the health facility? 102 How many Health Assistants conducting community awareness activities in the catchment area of this health facility? 104 Please show me examples of any flip charts or other materials used for education about malaria at this health facility or in the community 1= Materials observed 2= Materials exist but are not observed 3=No materials Report for the final evaluation of the USAID/Uganda Stop Malaria Project 62 Reporting (HMIS) 109 Does the facility have a Records Assistant or someone for data entry? YES………………………………………..1 NO…………………………………………2 110 In the last 3 years has the Records Assistant at this facility received training in data quality assessment? YES……………………………………….1 NO…………………………………………2 111 In the last 3 years has the person in￾charge of the facility received training in data quality assessment? YES……………………………………….1 NO…………………………………………2 112 Please show me the form 105 from June and the form 105 from July of 2013 1= Both forms observed 2=Only one form observer 0=Forms not observed 113 Assess whether the cells for reporting malaria testing have been completed properly on both forms 1=Malaria testing data reported properly on forms for both months 2=Malaria testing data reported properly for only 1 mont 3=Malaria testing data not reported properly on either form\ 0= No forms observed Assess whether the cells for reporting stock outs of malaria drugs have been completed properly on both forms 1=Anti-malarial stockouts reported properly on forms for both months 2= Anti-malarial stockouts reported properly for only 1 mont 3= Anti-malarial stockouts not reported properly on either form\ 0= No forms observed 114 Is there evidence of data analysis and utilization? ( Check for Malaria graph) MALARIA GRAPH OBSERVED…………1 OTHER EVIDENCE___________________97 (SPECIFY) NO ………………………………………….0 THANK YOU Report for the final evaluation of the USAID/Uganda Stop Malaria Project 63 Questionnaire for staff in the ANC clinic PROVISION OF IPT/ANC SERVICES 059 Does the health facility offer IPTp? YES ………………………………….1 NO ……………………………………2 (Check register or observe the mother taking SP) 1 →Q61 060 Why does the facility not offer IPTp LACK OF SP ……………………………….1 LACK OF CUPS …………………………….2 LACK OF WATER…………………………….3 LACK OF WATER PURIFICATION TABS….4 LACK OTHER SUPPLIES …………………..5 OTHER____________________________97 (SPECIFY) 061 Does the health facility administer IPTp under DOTs? YES, OBSERVED …………………..1 YES, NOT OBSERVED ……………2 NO……………………………………..3 1,2,→Q63 062 Why doesn’t the facility administer IPTp under DOTS? NO WATER ………………………….1 NO CUPS, OTHER SUPPLIES…….2 STAFF NOT TRAINED ……………..3 OTHER _____________________97 (SPECIFY) 063 How many health workers at this facility provide ANC care? 064 How many health workers in this facility have been trained to offer IPTp under DOTs? NUMBER OF HEALTH WOKERS [IF NO STAFF TRAINED, RECORD “00”] 065 Is there safe water ready for use within the ANC area for mothers to take IPT? YES, OBSERVED …………………..1 YES, NOT OBSEVED ………………2 NO……………………………………..3 066 Does this facility have an adequate quantity of the following IPTp Commodities? YES NO CUPS 1 0 WATER 1 0 PURIFICATION 1 0 TABS JERRYCANS 1 0 069 Is IPTp recorded correctly in: YES NO ANC Register 1 0 ANC Card 1 0 [The register should be “1” for IPTp1 in the IPTp column. “2” for IPTp2 in the IPTp column, “C” for completed in the IPTp column.] [If the mothers are present check a few cards to confirm whether the cards are filled in correctly] 070 Do you currently have SP in the ANC clinic YES …………………………………..1 NO …………………………………….2 071 What challenges do you experience LACK OF SP ………………………………..1 LACK OF WATER …………………………..2 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 64 during administration of IPTp to mothers? LACK OF CUPS …………………………….3 LACK OF JERYCANS ……………..………4 LACK OF WATER PURIFICATION TABS..5 LACK OF OTHER SUPPLIES………….…6 LOW COVERAGE WITH ANC2+ ……….7 MOTHER’s HAVE SIDE EFFECTS ….…8 MOTHER’s AVOID MEDICINES……..…11 LACK OF TRAINING OF STAFF……….12 OTHER _____________________97 (SPECIFY) 072 Do you dispense LLINs through the ANC? If yes, are they free? YES FREE ……………………………………………1 YES SUBSIDIZED COST…………………………2 YES FULL COST ……………………………………3 NO …………………………………………………….4 Go to the Questionnaire for a provider of ANC services Report for the final evaluation of the USAID/Uganda Stop Malaria Project 65 Final evaluation of the Stop Malaria Project Questionnaire for interview of provider of ANC care 1. Facility Identification 301 NAME OF FACILITY 302 DISTRICT 303 SUB-COUNTY EXPERIENCE AND TRAINING 304 Do you provide antenatal care in this health facility? 1= Yes 2= No No Find another provider 305 What is your current technical (or medical) qualification? SPECIALIST...............................................................1 MEDICAL DOCTOR.............................................2 CLINICAL OFFICER..............................................3 REGISTERED MIDWIFE........................................4 ENROLLED MIDWIFE ..........................................5 REGISTERED NURSE ............................................6 ENROLLED NURSE...............................................7 NURSING ASSISTANT.........................................8 NURSING AIDE......................................................9 COUNSELOR..........................................................10 OTHER __________________________..11 (specify) 306 In what year did you start working in this facility? IF YEAR IS NOT KNOWN, PROBE AND MAKE THE BEST ESTIMATE YEAR SUPERVISION 307 Now, I would like to ask you some questions about supervision you have personally received. This supervision may have been from a supervisor either in this facility, or from outside the facility. Do you receive technical support and supervision in your work? IF YES, ASK, When was the most recent time? YES, IN THE PAST 3 MONTHS…………………1 YES, IN THE PAST 4-6 MONTHS ……………...2 YES, IN THE PAST 7-12 MONTHS …………….3 YES, MORE THAN 12 MONTHS AGO ...………4 NO…………………………………………………..5 5,  Q121 308 How many times in the past 12 months has your work been supervised? NUMBER OF TIMES 309 The last time you were personally supervised, did your supervisor do any of the following? YES NO DK a) Check your records or reports CHECKED RECORD 1 2 8 b) Observe your work OBSERVED 1 2 8 c) Provide any written comment (either positive or negative) on your performance WRITTEN FEDDBACK 1 2 8 d) Discuss problems you have encountered DISCUSS 1 2 8 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 66 123 How many doses of sulphadoxine-pyrimethamine (SP) are recommended during pregnancy? ONE ....................................................................1 TWO.....................................................................2 THREE .................................................................3 FOUR ...................................................................4 MORE THAN FOUR ............................................5 124 When should mothers take the first dose of sulphadoxine￾pyrimethamine( SP) during pregnancy? [Prompt: “When the mother is how many weeks pregnant?”] Below 16 weeks (4 months) of pregnancy..........1 At 16 weeks of pregnancy....................................2 After 16 weeks of pregnancy ..............................3 125 After a pregnant woman has received a first dose of SP, how many weeks should pass before she is given a second dose of SP. [ record “13” if the response is “next trimester”] NUMBER OF WEEKS 127 What medications do you usually give pregnant women who are diagnosed with malaria in the following category? 128 Pregnancy below 16 weeks Oral Coatem ........................................................1 Oral Fansidar .......................................................2 Oral Quinine .........................................................3 IM Artesunate.......................................................4 IM Quinine............................................................5 IV Quinine.............................................................6 IV Artesunate........................................................7 Rectal artesunate..................................................8 Other ________________________________97 129 Pregnancy greater than 16 Oral Coatem ........................................................1 Oral Fansidar .......................................................2 Oral Quinine .........................................................3 IM Artesunate.......................................................4 IM Quinine............................................................5 IV Quinine.............................................................6 IV Artesunate........................................................7 Rectal artesunate..................................................8 Other ________________________________97 130 Has this facility experienced a stock out of SP in the past 3 months? 1=YES 2=NO 131 Has this facility experienced a stock out of coatem in the past 3 months? 1=YES 2=NO 131 Has this facility experienced a stock out of quinine tablets in the past 3 months? 1=YES 2=NO 132 If a rapid diagnostic test for malaria is negative are you confident that the patient does not have malaria 1=YES 2=NO THANK YOU Report for the final evaluation of the USAID/Uganda Stop Malaria Project 67 QUESTIONNAIRE FOR THE LABORATORY FIND THE MOST SENIOR HEALTH WORKER INVOLVED IN THE DELIVERY OF LAB SERVICES. Do you have the following items available? (a) Is item present? 079 (b) If item/s available, are they in working order? ITEMS REQUIRED FOR LABORATORY EXAMINATION AVAILABLE OBSERVED AVAILABLE NOT OBSERVED NOT AVAILABLE YES NO Nd 401 - Binocular Microscope 1 2 8 1 2 8 402 - Slides and coverslips 1 2 8 1 2 8 403 - Giemsa stain or field stain 1 2 8 404 - Hematocrit Centrifuge 1 2 8 1 2 8 405 -- HB Estimation Machine 1 2 8 1 2 8 406 -- HB test strips 1 2 8 1 2 8 407 - Rapid diagnostic test (RDT) for Malaria 1 2 8 408 - Glucometer or means to measure blood glucose 1 2 8 411 Is there a skilled human resource available to carry out malaria tests? Number LAB TECHNOLOGIST LAB TECHNICIAN LAB ASSISTANT MICROSCOPIST 412 Are there a lab personnel available at all times (including nights)? YES .................................................1 NO....................................................2 413 What techniques do you use to diagnose malaria in this laboratory? Microscopy ……………………………1 Rapid Diagnostic Tests (RDTs) ……… 2 414 Does the facility have any dysfunctional microscopes? YES ........................................................1 NO............................................................2 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 68 415 How many lab technicians / assistants now working here have been trained in-service in the last 3 years in Microscopy/ RDT? Technicians trained in microscopy Technicians trained in RDTs Assistants trained in microscopy Assistants trained in RDTs 416 Is there a system for sending malaria microscopy slides to an outside lab for quality assurance? YES………………………………………………………1 NO………………………………………………………2 417 If yes, how many months ago was this last done? 418 Is there any report of this outside quality assurance? 1= Yes, report observed 2= Yes, but no report observed 3= No report 0= No outside quality assurance is performed 419 The last time that malaria slides were sent to an outside lab, what % of slides were reported to have been read accurately? 420 At present, does this lab use the official HMIS lab register or do they use some other type of register to record their results? 1= Official HMIS lab register 2= Some other type of register is used 0= There is not lab register for recording results 421 Is the register up to date? UP TO DATE…………1 NOT UPTO DATE……2 No Register………………0 422 Are the name and age of the patient recorded as well as the result? 1= Yes 2= No 0= No register 423 Did the facility experience any stock out of lab supplies in the last three months? YES……………………………………………………1 NO……………………………………………………2 424 Has anyone in the laboratory or anyone in the health facility recently calculated the percentage of malaria tests which have been positive? 1= Yes, statistic observed 2=Yes, but no statistic was observed 0=No Report for the final evaluation of the USAID/Uganda Stop Malaria Project 69 QUESTIONNAIRE FOR INSPECTION OF THE DRUG STORE 096 Are there updated stock cards at the facility store? YES……………………………………1 NO……………………………………….2 097. CHECK TO SEE IF EACH OF THESE MEDICATIONS IS AVAILABLE IN THE FACILITY TODAY. 101. THEN CHECK TO SEE IF THEY HAVE HAD A STOCKOUT OF THIS MEDICATION IN THE LAST 3 MONTHS Drug Observed Not Available ND Stock-out No-Stock Out Never supplied Coatem (Artemether/ Lumefantrine) <5 Coatem (Artemether/ Lumefantrine) for >5 SP(Fasidar) IV Quinine Oral Quinine Artesunate( Rectal) Artesunate( injectable) 5% Dextrose Blood for transfusion 099 What is done when the Health Facility experiences stock-outs of different medicines and supplies PROVIDES PRESCIPTION …………………………………1 OTHER _________________________________97 (SPECIFY) Report for the final evaluation of the USAID/Uganda Stop Malaria Project 70 Questionnaire for interview of provider who treat malaria Facility Identification 101 NAME OF FACILITY 102 DISTRICT 103 SUB-COUNTY EXPERIENCE AND TRAINING 109 Do you treat malaria at this health facility 1= Yes 2= No STOP 110 What is your current technical (or medical) qualification? SPECIALIST...............................................................1 MEDICAL DOCTOR.............................................2 CLINICAL OFFICER..............................................3 REGISTERED MIDWIFE........................................4 ENROLLED MIDWIFE ..........................................5 REGISTERED NURSE ............................................6 ENROLLED NURSE...............................................7 NURSING ASSISTANT.........................................8 NURSING AIDE......................................................9 COUNSELOR..........................................................10 OTHER __________________________..11 (specify) 113 In what year did you start working in this facility? IF YEAR IS NOT KNOWN, PROBE AND MAKE THE BEST ESTIMATE YEAR SUPERVISION 118 Now, I would like to ask you some questions about supervision you have personally received. This supervision may have been from a supervisor either in this facility, or from outside the facility. Do you receive technical support and supervision in your work? IF YES, ASK, When was the most recent time? YES, IN THE PAST 3 MONTHS…………………1 YES, IN THE PAST 4-6 MONTHS ……………...2 YES, IN THE PAST 7-12 MONTHS …………….3 YES, MORE THAN 12 MONTHS AGO ...………4 NO…………………………………………………..5 5,  Q121 119 How many times in the past 12 months has your work been supervised? NUMBER OF TIMES 120 The last time you were personally supervised, did your supervisor do any of the following? YES NO DK 12002 Check your records or reports CHECKED RECORD 1 2 8 12003 Observe your work OBSERVED 1 2 8 12006 Provide any written comment (either positive or negative) on your performance WRITTEN FEDDBACK 1 2 8 12008 Discus problems you have encountered DISCUSS 1 2 8 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 71 PRESCRIBING PRACTICES DO NOT READ RESPONSES TO THE RESPONDENT 121 What signs show that a child may have severe malaria? DONOT READ. CHECK ALL THAT IS MENTIONED. INABILITY TO EAT/DRINK ..................................1 VOMITING EVERYTHING....................................2 FEBRILE CONVULSIONS ...................................3 LETHARGIC…………………………………………4 OTHER_______________________________ 97 (SPECIFY) 122 What is included in the pre-referral treatment of malaria among children CHECK ALL RESPONSES MENTIONED MANAGEMENT OF CONVILSIONS ...................1 MANAGEMENT OF HIGH TEMPERATUES .......2 MANAGEMENT OF DEHYDRATION ..................3 MANAGEMENT OF HYPOGLYSEMIA (Low blood sugar) ................................................4 GIVE IM QUININE................................................5 GIVE RECTAL ARTESUNATE.............................6 OTHER _______________________________97 1122 If a child has signs of severe malaria and you cannot treat severe malaria at your health facility, what medicine would you give to the child before you sent him or her to another health facility? [If the route of administration is not specified, ask whether the medicine would be given orally, IM or IV] Oral Coatem ........................................................1 Oral Fansidar .......................................................2 Oral Quinine .........................................................3 IM Artesunate.......................................................4 IM Quinine............................................................5 IV Quinine.............................................................6 IV Artesunate........................................................7 Rectal artesunate..................................................8 Other ________________________________97 126 What medications do you usually give children below 5 years of age who you diagnose as having uncomplicated malaria? Oral Coatem ........................................................1 Oral Fansidar .......................................................2 Oral Quinine .........................................................3 IM Artesunate.......................................................4 IM Quinine............................................................5 IV Quinine.............................................................6 IV Artesunate........................................................7 Rectal artesunate..................................................8 Other ________________________________97 127a If you think that a woman may be pregnant and has uncomplicated malaria, what medication do you prescribe to treat the malaria? “It depends on how far advanced the pregnancy is ........................................................1 Coatem ........................................................2 Fansidar .......................................................3 Quinine .........................................................4 Other ________________________________5 Don’t know …………………………………….98 127b If the pregnancy is less than 16 weeks what medicine would you prescribe for uncomplicated malaria? Oral Coatem ........................................................1 Oral Fansidar .......................................................2 Oral Quinine .........................................................3 IM Artesunate.......................................................4 IM Quinine............................................................5 IV Quinine.............................................................6 IV Artesunate........................................................7 Rectal artesunate..................................................8 Other ________________________________97 129 What medications do you usually give to treat patients who are hospitalized for severe malaria? Oral Coatem ........................................................1 Oral Fansidar .......................................................2 Oral Quinine .........................................................3 IM Artesunate.......................................................4 IM Quinine............................................................5 IV Quinine.............................................................6 IV Artesunate........................................................7 Rectal artesunate..................................................8 Other ________________________________97 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 72 130 Has this facility experienced a stock out of the following drugs in the past 3 months? READ OUT THE DRUG NAMES FOR DRUGS THAT THE FACILITY NO LONGER STOCKS, CIRCLE “N/A”. NO YES N/A SP 0 1 2 Artemether/Lumefantrine 0 1 2 Oral Quinine 0 1 2 IV Quinine 0 1 2 Artesunate(Rectal) 0 1 2 Now, I would like to ask you some questions about rapid tests and testing for malaria. Please let me know if you strongly agree, somewhat agree, somewhat disagree or strongly disagree with the statements below 131 If a rapid diagnostic test is positive then I am confident that the patient has malaria Strongly Agree Somewhat Agree Somewhat Disagree Strongly Disagree 132 If a rapid diagnostic test is negative then I am confident that the patient does not have malaria Strongly Agree Somewhat Agree Somewhat Disagree Strongly Disagree 134 I would consider treating a patient with an anti-malarial based on clinical suspicion, even if they have a negative RDT test result. Strongly Agree Somewhat Agree Somewhat Disagree Strongly Disagree Please complete the sentence below 135 I believe that rapid diagnostic tests (RDTs) make it _______ for me to do my job. INTERVIEWER: Probe, does it make it easier or harder to do your job or it does not make a difference? Easier No Difference Harder Don’t Know THANK YOU Report for the final evaluation of the USAID/Uganda Stop Malaria Project 73 Annex 8 – Questionnaire for a client exit interview Final evaluation of the Stop Malaria Project Questionnaire for a client exit interview 1. Facility Identification ID NUMBER: ___________________________________ 301 NAME OF FACILITY 302 DISTRICT 303 SUB-COUNTY 304 FACILITY NUMBER 305 TYPE OF FACILITY REGIONAL REFERRAL HOSPITAL 01 GENERAL HOSPITAL 02 OTHER HOSPITAL 03 HEALTH CENTER IV 04 HEALTH CENTER III 05 HEALTH CENTER II 04 OTHER__________________________ 96 (specify) 306 OWNERSHIP GOVERNMENT........................................01 PRIVATE NOT FOR PROFIT(PNFP)........02 OTHER __________________________ 96 2. Information about Interview 307 INTERVIEW DATE 308 NAME OF THE INTERVIEWER NO. QUESTIONS CODING CLASSIFICATION GO TO VERIFY THAT RESPONDENT PROVIDED CONSENT 314 May I begin the interview? CLIENT AGREE 1 CLIENT REFUSES 2 STOP What is the highest level of school you attended: primary, secondary, or post – secondary? NONE...............................................................................1 SOME PRIMARY .............................................................2 COMPLETED PRIMARY..................................................3 O Level........................................................4 A LEVEL ...........................................................................5 UNIVERSITY/TERTIARY ..................................................6 Do you listen to the radio almost every day, at least once a week, less than once a week or not at all? ALMOST EVERYDAY ...............................................1 ATLEAST ONCE A WEEK .........................................2 LESS THAN ONCE A WEEK .....................................3 NOT AT ALL ...............................................................4 Do you watch television almost every day, at least once a week, less than once a week or not at all? ALMOST EVERYDAY ...............................................1 ATLEAST ONCE A WEEK .........................................2 LESS THAN ONCE A WEEK ......................................3 NOT AT ALL ...............................................................4 329 Do you have children? YES ............................................................................1 NO ..............................................................................2 2,→Q331 330 When did you give birth to your youngest child? Month…………….. Don’t know month …….96 YEAR…… Don’t know year…97 331 Were you or the person you brought for care Yes .....................................................................1 STO P 322 323 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 74 today visiting the clinic for malaria related illness? No........................................................................2 332 Did you or the person you brought for care today have blood taken from his/her finger or heel for testing? Yes............................................................................1 No..............................................................................2 332a Did the provider offer or suggest testing for malaria? Yes ...................................................................................1 No ....................................................................................2 Don’t Remember.................................................................3 332b Did the provider EXPLAIN to you or the person you brought for care the RESULTS OF THE TEST? Yes ...................................................................................1 No ....................................................................................2 Don’t Know .......................................................................3 Please let me know if you strongly agree, somewhat agree, somewhat disagree or strongly disagree with the statement b elow 332c I believe I was given the correct treatment as per the outcome of the test result Strongly Agree 3 Somewhat Agree 2 Somewhat Disagree 1 Strongly Disagree 0 333 Were you or the person you brought for care today found to have malaria? Yes............................................................................1 No..............................................................................2 334 Did you or the person you brought for care today receive treatment for malaria? Yes...............................................................................1 No ...............................................................................2 2,→Q337 335 What drugs were you or the person you brought for care given today? Any other drugs? RECORD ALL MENTIONED ANTI-MALARIAL DRUGS Coartem.......................................................................1 Oral Quinine.................................................................2 Injectable quinine .......................................................3 Atersunate...................................................................4 Other ____________________________................... (SPECIFY) ANTIBIOTIC DRUGS Pill/Syrup .....................................................................5 Injection .......................................................................6 OTHER DRUGS Panadol .......................................................................7 Aspirin ...................................................................... ..8 Ibuprofen ....................................................................9 OTHER ____________________________________ (SPECIFY) Don’t know ................................................................98 336 Did a provider explain to you or the person you brought for care HOW TO TAKE THE ANTI-MALARIAL TABLETS? YES ............................................................................1 No...............................................................................2 All,→Q337 338 Record who the patient was Respondent ................................................................1 Respondent’s child ......................................................2 Respondent’s relative who is a child ..........................3 Respondent’s other relative ........................................4 Other ...........................................................................5 (SPECIFY) 339 How old is the child you bought for malaria related treatment today? IF CHILD IS LESS THAN ONE YEAR OLD, INDICATE AGE IN MONTHS. CHILD’S AGE YEAR……………………….. MONTHS……………………. 340 Did the provider inquire WHETHER CHILD is YES ............................................................................1 2,→Q337 1,4,5→ Q353 2,3→339 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 75 able or unable to drink or breastfeed at all? NO ...............................................................................2 342 Did the provider inquire whether the child has had convulstions? NO YES Convulsions with this sickness 0 1 349 How long after the fever started did you bring the child for treatment at this facility? THE SAME DAY.....................................................................1 ONE DAY AFTER FEVER STARTED .................................. 2 TWO DAYS AFTER FEVER STARTED ............................... 3 THREE OR MORE DAYS AFTER ........................................ 4 DON’T KNOW / NOT SURE ...........97 350 Did you provide any remedies/medications to the child at home before you bought him/her for treatment? YES .....................................................................................1 NO .......................................................................................2 351 What medications/drugs did you give the child before you bought him/her this facility today? Oral Quinine.................................................................1 Atersunate...................................................................2 Coartem ......................................................................3 Other ____________________________ (SPECIFY) ANTIBIOTIC DRUGS Pill/Syrup ....................................................................4 OTHER DRUGS Panadol .......................................................................5 Aspirin ...................................................................... ..6 Ibuprofen .....................................................................7 OTHER ____________________________________ (SPECIFY) Don’t know…………………………………………….97 352 How soon after the fever began did you give the child these drugs? THE SAME DAY.....................................................................1 ONE DAY AFTER FEVER STARTED ..................................2 TWO DAYS AFTER FEVER STARTED ...............................3 THREE OR MORE DAYS AFTER ........................................4 DON’T KNOW / NOT SURE ...........97 All,→Q353 353 Do you own a mosquito net? YES ..........................................................................1 NO ............................................................................2 2,→Q356 354 Is it an ITN, that is a net that has been treated with a chemical to protect you from mosquito bites? YES ..........................................................................1 NO ............................................................................2 DON’T KNOW ........................................................98 355 Where did you get the mosquito net? HOUSEHOLD DISTRIBUTION….....................................................................................1 ANC CLINIC ..................................................................................2 PURCHASED FROM MARKET ........................ ……………………………. ……….3 OTHER ___________________________________________97 (SPECIFY) DON’T KNOW ……………………………………………………...98 356 Did you sleep under a mosquito net last night? YES ...........................................................................1 NO ...............................................................................2 2,→Q358 357 Was the net you slept under last night treated with a chemical? YES ..........................................................................1 NO ............................................................................2 DON’T KNOW ...........................................................3 358 INTERVIEWER CHECK Q327 Respondent ................................................................1 Respondent’s child ......................................................2 Respondent’s relative who is a child ..........................3 Respondent’s other relative ........................................4 Other ...........................................................................5 359 Did the child you brought for care sleep under a net last night? YES ...........................................................................1 NO ...............................................................................2 2,→Q361 360 Was the net the child slept under last night treated with a chemical? YES ...........................................................................1 NO ...............................................................................2 DON’T KNOW…………………………………………98 All,→Q361 1,4,5Q 361 2,→Q353 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 76 361 During this or previous visit did a provider offer you an ITN free of charge or offer to sell you one? IF THERE IS AN INDICATION THAT THE CLIENT WILL PICK UP OR BUY THE ITN ELSEWHERE WITHIN THE FACILITY, THAT COUNTS AS PROVIDER GIVING OR CLIENT PURCHASING FROM PROVIDER YES, OFFERED FREE THIS VISIT ............................1 YES, OFFERED FREE PREVIOUS VISIT...................2 YES, OFFERED TO SELL ME ONE THIS VISIT .......................3 YES, OFFERED TO SELL ME ONE THE PREVIOUS VISIT............ 4 NO, NOT OFFERED ...................................................5 362 What is the best medicine to treat a child who is sick with malaria? FANSIDAR ..................................................................................1 CHLOROQUINE ..........................................................................2 METAKELFIN ..............................................................................3 MEFLOQUINE .............................................................................4 ARTEMETHER/LUMEFANTRINE ...............................................5 QUININE ......................................................................................6 COARTEM ...................................................................................7 HERBAL REMEDIES ..................................................................9 DON’T KNOW/NOT SURE…………………………………...96 OTHER ___________________________________________97 (SPECIFY) 363 How soon after the on-set of malaria should somebody suspected of having malaria be taken for treatment? THE SAME DAY.....................................................................1 ONE DAY AFTER FEVER STARTED .................................. 2 TWO DAYS AFTER FEVER STARTED ............................... 3 THREE OR MORE DAYS AFTER ........................................ 4 WHEN THE FEVER IS TOO HIGH/TOO HOT.......................5 DON’T KNOW / NOT SURE ...........96 OTHER ________________________________________97 (SPECIFY) 366 Do you think pregnant women should be given anti-malarial tablets to prevent them from getting malaria? YES ...........................................................................................1 NO ..............................................................................................2 DON’t KNOW………………………………………………………..96 367 What dugs/medications should pregnant women be given to prevent them from getting malaria during pregnancy? FANSIDAR ..................................................................................1 CHLOROQUINE ..........................................................................2 METAKELFIN ..............................................................................3 MEFLOQUINE .............................................................................4 ARTEMETHER/LUMEFANTRINE ...............................................5 QUININE ......................................................................................6 COARTEM ...................................................................................7 HERBAL REMEDIES ..................................................................9 DON’T KNOW/NOT SURE………………………………………96 OTHER ___________________________________________97 368 How many doses of anti-malarial tablets should a pregnant woman take during a pregnancy to prevent her from getting malaria? ONE ............................................................................1 TWO ............................................................................2 MORE THAN TWO ....................................................3 DON’T KNOW/ NOT SURE........................................96 370 Have you heard of any malaria prevention and treatment messages on the radio? YES .............................................................................................1 NO ...............................................................................................2 DONT REMEMBER .....................................................................3 371b What messages did you hear I SHOULD SLEEP UNDER A MOSQUITO NET.....................................1 PREGNANT WOMEN SHOULD TAKE MEDICATIONS TO PREVENT MALARIA DURING PREGNANCY................................2 PERSONS SUFFERING FROM MALARIA SHOULD BE TREATED WITHIN THE FIRST 24 YEARS AFTER ONSET OF FEVER ...............3 ONE SHOULD BE TESTED FIRST BEFORE THEY ARE TREATED FOR MALARIA........................................................................................4 OTHER________________________________________________97 (SPECIFY) ANC CLIENT, CONTINUE. IF NOT ANC CLIENT, GO TO Q.385 Report for the final evaluation of the USAID/Uganda Stop Malaria Project 77 EXIT INTEVIEW FOR ANC CLIENTS ONLY 372 Did you visit the antenatal clinic today? YES ............................................................................1 NO................................................................................2 IF NO, STOP INTERVIEW 373 Is This your first pregnancy? YES ............................................................................1 NO................................................................................2 374 Is this your first antenatal visit at this facility for this pregnancy? YES .............................................................................1 NO................................................................................2 375 How many weeks pregnant do you think you are? IF RESPONSE IS IN MONTHS, CALCULATE WEEKS, USING 4 WEEKS PER MONTH WEEKS............................... 376 When did you attend your first antenatal visit for this pregnancy? WITHIN THE FIRST THREE MONTHS OF PREGNANCY...............................................................1 BETWEEN THE FOURTH AND SIXTH MONTH .....................................................................................2 BETWEEN THE 6TH AND 9TH MONTH .......................3 DONT KNOW/DON’T REMEMBER ............................4 377 During this or previous visits, has a provider given or prescribed any anti-malarial tablets to protect you from malaria? SHOW THE CLIENT TABLETS OF FANSIDAR YES, THIS VISIT.......................................................1 YES, PREVIOUS VISIT.............................................2 YES, BOTH PREVIOUS VISIT AND THIS VISIT.........3 NO................................................................................4 DON’T KNOW.........................................................98 378 How many times during this pregnancy have you swallowed the tablets given you to prevent malaria? ONE TIME ...................................................................1 TWO TIMES ...............................................................2 MORE THAN TWO TIMES .........................................3 DONT KNOW/DONT REMEMBER .............................4 379 Did the provider ask you to take the tablets in front of him or her? YES, ..........................................................................1 NO, ............................................................................2 380 Did you take them? YES,..........................................................................1 NO,.............................................................................2 1,→Q383 381 If not, why? I WAS HUNGRY ............................................................1 DONOT LIKE THE SIDE EFFECTS ...............................2 OTHER ___________________________________97 (SPECIFY) 383 Did the provider explain to you the number of doses of anti-malarial drugs you need during this pregnancy? YES .............................................................................1 NO ..............................................................................2 384 Did the provider explain why you need to use an insecticide treated net during pregnancy? YES ............................................................................1 NO...............................................................................2 THANK YOU