Formative Evaluation of Quality of Care Initiative by Ministry of Health |i FORMATIVE EVALUATION OF QUALITY OF CARE INITIATIVES BY MINISTRY OF HEALTH – UGANDA REPORT FINAL REPORT August, 2010 This document was produced for review by the United States Agency for International Development (USAID/Uganda). It was prepared by The Mitchell Group, Inc. for the Uganda Monitoring and Evaluation Management Services, herein referred to as UMEMS, number 617-C-00-08-00012-00 Acknowledgements The Evaluation of the Quality of Care (QoC) initiative was made possible with the support from the United States Agency for International Development (USAID) through the Uganda Monitoring and Evaluation Management Services (UMEMS), which is managed by The Mitchell Group, Inc. (TMG), located in Washington, D.C. This report was written by Dr. Helen Smits, Dr. Andrew Balyeku, Dr. Fred Bagenda, Dr Larry Adupa, Dr. Edgar Mulogo, Dr. Nathan Nshakira, and Dr. Hizaamu Ramadhan. The authors are very grateful to USAID staff for their support and active participation in the review, UMEMS staff, including Patricia Rainey, Nestore Jalobo, Collin Obote and Caroline Kasabiti who provided excellent logistical and other support in Uganda and work space for the team in Kampala. The support from TMG/HQ assured that the evaluation could be mobilized and completed on time. We would especially like to thank the participants in the review—the staff of the Ministry of Health, Uganda, HCI and HIVQUAL program, the Regional Hospital staff, the districts, health facilities, Civil Society Organizations (CSOs) and the People Living with HIV/AIDS that participated in the interviews and the Focus Group Discussions (FGDs). Formative Evaluation of Quality of Care Initiative by Ministry of Health |ii Table of Contents ACKNOWLEDGEMENTS ................................................................................................................................ II LIST OF ACRONYMS AND ABBREVIATIONS .......................................................................................... IV EXECUTIVE SUMMARY ................................................................................................................................ VI 1. INTRODUCTION ............................................................................................................................................ 1 1.1 BACKGROUND ........................................................................................................................................ 1 1.2 HEALTH SYSTEM CONTEXT IN UGANDA ................................................................................................ 1 1.3 INITIATIVE............................................................................................................................................. 1 1.4 OBJECTIVES OF THE FORMATIVE EVALUATION ...................................................................................... 2 2. METHODOLOGY ...................................................................................................................................... 4 2.1 DESIGN DESCRIPTION ............................................................................................................................ 4 2.2 SAMPLING .............................................................................................................................................. 4 2.3 LIMITATIONS .......................................................................................................................................... 4 3. FINDINGS .................................................................................................................................................... 5 3.1 OBJECTIVE 1: IMPROVEMENT OF QUALITY IN COMPREHENSIVE HIV/AIDS CARE AND TREATMENT SERVICE DELIVERY ............................................................................................................................................. 5 3.1.1 Effectiveness in improving quality of HIV/AIDS services ................................................................. 5 3.1.2 Key factors facilitating or hindering improvements in HIV/AIDS services .................................... 11 3.1.3 Collaboration and coordination ..................................................................................................... 12 3.2 OBJECTIVE 2: DEVELOPMENT OF CAPACITY OF MINISTRY OF HEALTH TO IMPROVE AND MAINTAIN QUALITY OF COMPREHENSIVE HIV/AIDS PROGRAM ....................................................................................... 14 3.2.1 Capacity building in the MoH ........................................................................................................ 14 3.2.2 Integration and spread of QI approaches ....................................................................................... 20 3.2.3 Factors affecting the institutionalization of QoC initiative ............................................................ 20 3.2.4 Evidence of sustainability of QoC initiative ................................................................................... 25 3.2.5 Key lessons learned from the design and implementation .............................................................. 28 3.3 OBJECTIVE 3: QOC SUPPORT OF ACCREDITATION IN FACILITIES PROVIDING HIV SERVICES ................ 30 3.3.1 Accreditation support ..................................................................................................................... 30 3.3.2 Factors facilitating and hindering accreditation ............................................................................ 30 4. RECOMMENDATIONS ........................................................................................................................... 32 4.1 NATIONAL-LEVEL RECOMMENDATIONS ............................................................................................... 32 4.1.1 Organization and Structure ............................................................................................................ 32 4.1.2 Policy .............................................................................................................................................. 32 4.1.3 Operations/practice. ....................................................................................................................... 32 4.2 SUB-NATIONAL LEVEL RECOMMENDATIONS ....................................................................................... 33 4.2.1 Regional level ................................................................................................................................. 33 4.2.2 District level.................................................................................................................................... 33 4.2.3 Health facility level ......................................................................................................................... 33 ANNEXES ........................................................................................................................................................... 34 ANNEX 1: PLACES VISITED ................................................................................................................................ 34 ANNEX 2: SCOPE OF WORK .............................................................................................................................. 35 ANNEX 3: FACILITY-LEVEL FINDINGS .............................................................................................................. 38 ANNEX 4: BIBLIOGRAPHY ................................................................................................................................. 40 Formative Evaluation of Quality of Care Initiative by Ministry of Health |iii List of Acronyms and Abbreviations AIDS Acquired Immunodeficiency Syndrome ANC Ante-Natal Care ART Antiretroviral Therapy ARV Antiretroviral CSO Civil Society Organisation DAC District AIDS Committee DAT District AIDS Task Force DHO District Health Officer DHT District Health Team DQA Data Quality Assessment FBO Faith Based Organization FGD Focus Group Discussion FSG Family Support Groups HAART Highly Active Anti-Retroviral Therapy HC Health Centre HCT HIV Counselling and Testing HIV Human Immunodeficiency Virus HMIS Health Management Information System HRH Human Resources for Health HSSP Health Sector Strategic Plan IRCU Inter Religious Council of Uganda JICA Japanese International Cooperation Agency JMS Joint Medical Stores MCH Maternal and Child Health M&E Monitoring and Evaluation MoH Ministry of Health MOU Memorandum of Understanding MS Medical Superintendent MSF Medecins San Frontiers NMS National Medical Stores NGO Non Governmental Organisation OPD Out Patient Department OVC Orphans and Vulnerable Children PHA People Living with HIV/AIDS PHC Primary Health Care PNFP Private Not for Profit PMTCT Prevention of Mother to Child Transmission QI Quality Initiative QIT Quality Improvement Team QoC Quality of Care RH Reproductive Health Formative Evaluation of Quality of Care Initiative by Ministry of Health |iv RQT Regional Quality Team RRH Regional Referral Hospital TASO The Aids Support Organisation TB Tuberculosis UAC Uganda AIDS Commission UMEMS Uganda Monitoring and Evaluation Management Services UN United Nations USAID United States Agency for International Development USG United States Government Formative Evaluation of Quality of Care Initiative by Ministry of Health |v Executive Summary A major Quality of Care initiative (QoC) was initiated in Uganda in 2005 that was supported through various U.S. government-funded PEPFAR projects. The initial aim was to ensure that antiretroviral therapy (ART) care provided in Uganda was of high-quality in light of the quick rollout of ART in the country. This evaluation was designed to provide insights regarding the QoC initiative’s effect on effectiveness, collaboration and coordination, capacity building, institutionalization, sustainability and accreditation, outline the hindering and facilitating factors, highlight lessons learned from implementation of the initiative, and offer recommendations. This evaluation is based on a formative cross-sectional survey conducted in July 2010 at the national, regional, district and facility levels of the health care delivery system. Key Findings a) The QoC initiative has largely been effective in improving the quality of HIV/AIDS services at all levels. Given the rapid scale-up of ART services nationally and to the level of HC IV, the MoH and partners realized that the quality of care was potentially poor in many of the facilities. The QoC program has been instrumental in establishing a national structure to ensure that all facilities accredited and providing ART receive training and supervision services that will enable staff to identify and correct their problems. The spread of the initiative into the districts was built along the original ART scale-up plan that involved regional rather than district level. The regional teams have been effective in ensuring rapid uptake of QI processes at ART sites. One year following establishment of the teams, district QITs are gradually being formed that constitute the first attempt to do this at that level with respect to ART services. At the facility level, improvements in￾patient flow, patient record management, indicator tracking and data use have been made, based on QI meetings. The facility QI teams were modelled on the existing ART teams. b) The QoC initiative has established a number of activities to increase collaboration and coordination among partners implementing QI activities within ART services. A framework has been established to push joint planning at the national level. However, collaboration within districts remains weak because the collaboration structures (especially the District Health Teams) have not yet been adequately harnessed for QoC implementation. Consequently, facilities supported by the programs often do not communicate with each other, records are not shared or aggregated and there is parallel supervision and mentoring. c) The QoC initiative has strengthened capacity planning and training at the national level. Regional training and mentoring capacity has also been developed, and districts for the first time are now engaged in ART clinical care. As a result, most ART clinics have developed QI competence. However, critical gaps still exist that undermine the effectiveness of the QI structures. These include inadequate involvement of managers, lack of tangible incentives and reliance on external support for morale, guidance and logistics. QoC is still in its “infancy stage,” even where facilities have graduated or implemented services for more than two years. This is particularly true at HC and the district hospital levels. d) Several factors facilitate the institutionalization of QI activities. First, national quality indicators have been selected that are key elements in the provision of services and are attainable by lower-level facilities. Facilities in a particular district were encouraged to work on three to four similar indicators, thus facilitating the district team’s work. Formative Evaluation of Quality of Care Initiative by Ministry of Health |vi Significant staff training and awareness-raising among a critical mass of district and facility staff in the country provide a base of competency that facilitates further instutionalisation. The ART team composition promotes participation of the affected population in QI efforts. Despite the impressively quick implementation of the QI initiative at ART clinics throughout the country, institutionalization remains weak. The ART program has not been adequately instutionalised yet and many workers still view QI as part of ART. The approach has not yet effectively mobilized managers who can serve as the engines for institutionalization and continuity. The design also did not build links to programs like Yellow Star. The QI program thus remains dependent on continued external PEPFAR funding to provide the separate budget envisioned by managers for supervision, expenses, and the like. This may partly be attributable to unrealistic expectations by the implementers. Key Lessons Learned • Diminishing commitment to, and sponsorship of, top management at facilities like Iganga Hospital can be a significant basis for poor performance. • The pace of the QI improvements depends on the capacity of very constrained resources. Failure to properly address the linkages between structural constraints such as understaffing, lack of space and furniture and gross underfunding increases the risk of failure beyond the short-term • The process of accreditation of facilities for ART services also seemed skewed against FBO facilities. Only two districts had ever used QI to address accreditation. • Little evidence existed that facilities were prioritizing and sequentially implementing improvement cycles. Many of the deeper problems in districts and facilities would require such an approach in order to achieve more notable successes. Consequently, simple internal activities are revised instead of enacting real long-term solutions. For example, QI has not yet been applied to the serious problems of stockouts and lack of availability of CD4 testing even though these are the kinds of problems for which QI is well-suited. • Improvement in data integrity and analysis at the ART clinics was critical to determining gaps that the QI team could address. The introduction and use of revised ART patient records made it easier for the teams to identify problems. Though it would take longer, all QoC efforts should first start with data quality initiatives. Such efforts are designed to establish and maintain consistent data definition, analysis and presentation rules so that facilities or departments can achieve a single version of the real data and save time identifying real problems. Key Recommendations a) After strengthening the QAD in the MoH, the QoC initiative should be housed within the Quality Assurance Department to facilitate preparation, budgeting and implementation that were planned under the health sector framework. This will ensure government leadership, be adopted by other departments and result in common approaches that neither confuse nor burden front-line workers. Formative Evaluation of Quality of Care Initiative by Ministry of Health |vii Formative Evaluation of Quality of Care Initiative by Ministry of Health |viii b) Efforts to align organizational culture and value change with QI should flow from top management to front-line staff. It should be done in a way that is clearly visible and consistent with the leadership structure. c) A capacity-building plan should be developed for all levels that includes structured training and support. Pre-service training in quality of health care should be developed jointly with training institutions. In-service training should not only single out national quality targets but also develop quality improvement skills among managers and develop awareness about achieving long- term quality goals. d) A process should be developed for reporting quality indicators at all levels of the system whereby aggregated national information is made available to top leadership in the MoH and other stakeholders. e) A strategy should be designed to integrate the QoC initiative into other service areas at the health facility level, using the PDSA cycle approach. One or two high-volume service areas could be chosen and collaborations developed to get them started on QI. f) The regional quality teams should operate within the existing regional technical support supervision framework. g) The QI initiative should be integrated into the existing HIV/AIDS coordination and reporting structures at the district level, e.g., the District AIDS Committee and District AIDS Team. h) QI collaboratives should be initiated at the district level to deal with those administrative issues that are limiting success, such as stock outs, human resources policies and lack of CD4 testing availability. i) Administration and key facility leadership need to understand and own the benefits of change. Major differences exist at the facility level and the approach adopted needs to be adapted to the circumstances at each level. j) Consideration should be given to providing incentives to facilities that meet their goals and maintain their gains. For example, providing modest financial incentives to successful teams has been used with success in other settings. k) Facility leadership, especially among nursing officers, is instrumental in quality improvement; consequently, QI work should be included in the basic job description of all managers so QI becomes an integral part of supervision. l) Greater attention should be paid to sustainable methods of QoC. Implementation should be congruent with targets established by facilities and should strengthen rather than increase the burden on the existing systems and structures, which are already overstretched. 1. INTRODUCTION 1.1 Background Uganda’s population is approximately 32 million. The country’s GDP is projected to increase to 6.4 per cent at basic prices in 2009/10. The HIV/AIDS prevalence has gone down from a peak of 30 per cent in the 1980s to 6-7 per cent in 2005/6. The prevalence is higher for women, at 7.5 per cent than it is for men, at 5 per cent (UHSBS 2004/05). While significant expansion has occurred in the availability of ART services, important problems remain with respect to the quality of health services. 1.2 Health System Context in Uganda Public health services in Uganda are provided in National Referral Hospitals (NRHs) and Regional Referral Hospitals (RRHs), General Hospitals, Health Centre (HC) IVs, HC IIIs, HC IIs and Village Health Teams (HC Is). Standard-setting and quality assurance, resource mobilization, capacity development, and technical support supervision are among the core functions of the MoH headquarters, which seeks to provide a supportive environment for delivery of high-quality services. Regional hospitals are semiautonomous units that are directly connected to the national office, with no defined operational linkage to districts. The delivery of health services is decentralized to districts and health sub-districts (HSDs). The District and Municipal Local Governments manage public general hospitals and HCs and provide supervision and monitoring of all health activities in their respective areas. Though this includes the private sector, public-private partnerships remain weak at the district level. The lower-level HSDs have responsibility for the planning, organization, budgeting and management of the health services at this and the lower health centre levels. The national strategy provides for ART service delivery up to HC IV, though a number of HC IIIs have been accredited, especially in the urban and private sector. The rapid expansion of ART in Uganda has resulted in an increase in the number of facilities providing ART services in the public and private sector from 110 in 2004 to 374 by the end of September 2009. These facilities are located in all districts (old districts) in the country. Currently over three-quarters of HC IVs are providing ART services. Uptake of ART services has also increased from 17,000 active ART clients in 2003 to 200,213 clients by September 2009. Most (91.5 per cent) are adults over the age of 15. Based on the national standard that those with CD-4 T cell counts of less than 250 cells per micro litre should be in treatment, approximately 54 per cent of individuals in need of ART were receiving treatment by September 2009. 1.3 QoC Initiative The Ministry of Health and the USAID Mission in Uganda identified quality of care as an integral component of the rapid scale- up of comprehensive HIV/AIDS services. The Quality of Care (QoC) initiative was launched in 2005. From 2005 to 2007, the QAP and HIVQUAL supported the MoH in implementing the QI interventions in ART clinics. From 2008, QAP transformed into HCI and continued to support the Ministry together with HIVQUAL. QI Formative Evaluation of Quality of Care Initiative by Ministry of Health |1 interventions have occurred in 257/360 (72 percent) of accredited ART facilities in the country. During the last five years, the implementation of the QoC initiative was based on the Uganda Ministry of Health’s experience developing and supporting the scaled-up delivery of ART services in Uganda, its global experience in quality assurance and quality improvement in health services, and the technical assistance provided by the two organizations (HIVQUAL and URC-CHS) funded by the U.S. Government that enabled implementation of QoC. The QoC program is currently coordinated and directed by the, Ministry of Health and its main objective is developing a sustainable quality improvement system for HIV/AIDS service delivery. The three strategic objectives of the QoC are to: 1) Improve the quality of comprehensive HIV/AIDS care and treatment service delivery to adults and children at national, regional, district and sub-district facilities; 2) Develop the capacity of the Ministry of Health to improve and maintain the quality of the national comprehensive HIV/AIDS program; and 3) Support accreditation of public, private and PNFP facilities providing HIV/AIDS services. QoC implementation initially focused on strengthening capacity to support quality improvement in ART services, and therefore dealt initially with building the team at headquarters engaged in the ART scale-up program. It also focused on ART programs aiming to start with one site in each district. The rationale for this approach was to build a critical mass of sites across the country providing high-quality ART services, and ones with the capacity to facilitate the expansion of quality improvement in ART services to other facilities in the same district or sub-region. In order to expand and sustain ART quality improvement, the initiative adopted a decentralized capacity-building approach that initially focused on taking capacity to support QI to the regional level (by building QI support teams within each health region). Members of regional QI teams were selected from among the health staff already trained and engaged in QI activities at the ART sites within each region. It also focused on developing district QI teams that would support scaling up QI at lower-level health facilities as part of an effort to prepare more health facilities for accreditation and inclusion in the national ART scale-up. The evaluation team found that the approach described above was appropriate and effective, and considers it an innovation that is worth adopting as part of the planned QI roll-out to other health services. 1.4 Objectives of the Formative Evaluation The purpose of the evaluation was to determine whether QoC is on track to achieve its program objectives and to assess the program’s strengths and challenges in order to make programmatic and management improvements in MoH partner support to the initiative. The evaluation was intended to (a) make recommendations for strengthening institutionalization of QI in HIV/AIDS care and treatment services as well as devise lessons learned and best practices for the benefit of other MoH programs and/or for future programming and; (b) assist MoH and its partners in enhancing programs that provide HIV care and treatment services to adults and children. The key evaluation questions were based on each QoC objective: Formative Evaluation of Quality of Care Initiative by Ministry of Health |2 1. Improve the quality of comprehensive HIV/AIDS care and treatment service delivery to adults and children at national, regional, district and sub-district facilities. a) To what extent is the QoC initiative effective in improving the quality of HIV/AIDS services at the national, regional, district, sub-district and facility levels? b) What have been key factors facilitating or hindering improvements in HIV/AIDS services? c) To what extent is the QoC initiative effective in collaborating and coordinating with other partners implementing QI activities at the national, regional, district, sub￾district and facility levels? 2. Develop the capacity of the Ministry of Health to improve and maintain the quality of the national comprehensive HIV/AIDS program. a) How effective have partners been in building capacity in the MoH to improve and maintain the quality of HIV care and treatment services at the national, regional and district levels? b) To what extent has the QoC initiative developed the capacity of other relevant departments in MoH to integrate QI approaches? c) What have been the key factors facilitating or hindering the institutionalization of quality improvement activities at different levels of the MoH? d) What evidence is there that the project will be sustained in the long-term? e) What are the key lessons learned from the design and implementation of the QoC initiative? 3. Support accreditation of public, private and PNFP facilities providing HIV/AIDS services. a) How effective has the QoC initiative been in supporting MoH in the accreditation of facilities for ART? b) What have been the key factors facilitating or hindering support for accreditation of public, private and NGO facilities providing HIV/AIDS services? Formative Evaluation of Quality of Care Initiative by Ministry of Health |3 2.METHODOLOGY 2.1 Design Description The evaluation contained a formative cross-sectional survey that focused on the national, regional, district and facility levels of the health care delivery system. It involved the collection of primary and secondary data through desk records review, interviews and observations. The main data collection, conducted in June-July 2010, consisted of gathering information at all of these levels using four separate data collection tools (omitting the HSD level since it was not included in the QoC design). The main data sources were (1) National Level QI team members from MoH and partner organizations (HCI and HIVQUAL) (2) Regional QI Teams/facilitators (3) the District QI Team and (4) Site QI Teams. The national tool consisted of a set of approximately 30 structured and open-ended questions regarding QoC design, leadership, coordination, capacity-building at all levels, institutionalization, sustainability, activity integration, policies and hindering and facilitating factors. At the facility level, relevant QI records (documentation journals and minutes of QI team meetings) were reviewed. The facility tool contained questions regarding team membership, training, staffing levels, leadership and management, supervision and monitoring and providers’ perceptions of QI. Focus Group Discussions with clients at the facilities were conducted to ascertain the clients’ views on the quality of care they received at the clinics. 2.2 Sampling The evaluation team conducted a three-stage sampling process. The first stage of sampling took place at the regional level, where a random sample of six (Arua, Fort Portal, Jinja, Masaka, Mbale and Lira) of the 11 MoH regions were selected. Kampala, the seventh region, was used to test the tools. The second stage of sampling consisted of randomly selecting two districts in each region from which one was selected. The third stage involved selecting a list of all facilities that were reported as having QI and then randomly selecting four to five facilities in each region to visit. During the third stage of sampling we also ensured that both HCI- and HIVQUAL-supported facilities were included in the sample. Six regional teams, six district teams 25 health facility ART clinics (19 HCI, 6 HIVQUAL) and conducted interviews and six focus group discussions. Four additional facilities were included that were conducting QI activities. (See annex 1) This ensured that facilities and teams visited had valuable experience in QI. 2.3 Limitations • Because the ART program was changing rapidly during the period studied, it was not possible to identify a clear baseline of quality against which to measure changes. • The study did not include an examination of QI efforts in other (non-Ugandan) ART programs in order to draw comparisons. • The scope of the study did not involve comparing the degree of ART quality improvement between facilities supported by different projects Formative Evaluation of Quality of Care Initiative by Ministry of Health |4 3.FINDINGS 3.1 Objective 1: Improvement in Quality Comprehensive HIV/AIDS Care and Treatment Service Delivery 3.1.1 Effectiveness in improving the quality of HIV/AIDS services It is clear that the QoC has put in place a working mechanism to define measure and improve the quality of ART services. A framework for the inclusion of quality parameters in service scale-up has been developed. In addition, quality improvement steps, tools and capacity￾building processes have been defined, tested and are in use. As a result, there are now staff with QI knowledge, skills and early experience throughout the Ugandan health system. However, a complete framework for QI has not been developed that is harmonized with QA for all health services and programming for AIDS response; only annual work plans have been developed by the HIVQUAL and HCI. The project approach to QoC has not fully exploited existing mechanisms/efforts for general improvement in health care, monitoring and evaluation and routine reporting of health services and general service support programs. This is discussed in greater detail below. A. National level i. A documented description of the QoC has been developed as it is intended to be, with the vision, mission and five strategic goals that have been incorporated into the HSSP III. This documentation was developed over the last 12 months (July 2009 to June 2010) and is based on the experience gained in implementing the two projects since 2005. The QoC technical core team at MoH developed the Uganda Quality of Care programme 2010-2014 strategic plan, with support from its partners. ii. The support that the two projects provided to QoC implementation in ART was channelled through two streams in the Ministry of Health – HCI, through the clinical services department, and HIVQUAL, through the ACP which falls under the National Disease Control Department. iii. The Ministry has a Quality Assurance Department under the Directorate of Planning whose role in implementation of the two support projects has been limited to having three members on the steering committee and one member on the QoC core technical team. iv. A collaborative management structure has been established that is composed of the QoC steering committee and the QoC core technical team. These structures are not clearly linked to the management and policy structures of MoH that would facilitate institutionalization of QoC initiatives. v. Clear policy guidance has not been established to streamline integration of QoC into sectoral plans, budgets and activities. vi. A number of indicators, guidelines and tools for ART programme implementation have been developed by QoC partners (HIVQUAL-data collection, IDI-CAP, JICA￾5S, HCI-collaborative support) and disseminated to different facilities. The guidelines and tools include documentation journals, training manuals, and supervision checklists. vii. These programmes were initiated in health facilities where ART services were being Formative Evaluation of Quality of Care Initiative by Ministry of Health |5 provided. Because the Regional Referral Hospitals contained capacity in clinical service delivery and support, the immediate approach to build support systems for QI had to focus at the regional level. Subsequently, as a result of sustainability and the need to expand and scale-up QI to other health facilities within the districts, it became necessary to build capacity at the district level as well. Currently, about 40 district QITs have been trained. B. Regional level i. Regional teams are comprised of staff from the hospital and DHT. They are considered extensions of the national teams and do not have specific work plans. They only provide supervision when someone from the centre comes down with resources The regional teams have very limited linkages with district teams. “They give me a title and responsibilities without resources, therefore I cannot work” Regional ART coordinator “There is lack of joint planning between the centre and the region, they just call us and say that tomorrow we have supervision field work.” Regional trainer ii. The regional QI process does not conform to the regional technical support supervision structure and system. Consequently, the two processes are delinked. Some of the regional hospital ART or QI team members are members of the regional teams. For instance, in one region, of the eight members of the QI team at the regional referral hospital, two were participating in regional QI activities. In another region, the one person actively serving as a regional QI facilitator was based at the regional referral hospital but was not participating in the QI team of the hospital. In another region the team of six that has ceased to function designated the district records officer as a secretary of the team. In another region, the regional QI facilitator was an ART and QI team leader at an HCIV. iii. Efforts to link the two levels have involved including district QI team members in the regional QI team. The QoC is one of the first concerted efforts to involve DHTs in facility-based HIV clinical care. C. District Level i. QI work was only initiated in the Districts in 2009. ii. Teams existed where they were expected to be in place. They were comprised entirely of representatives of implementing sites, held regular, well-attended meetings, and have maintained current records. The QI teams in all six districts that we visited had undergone training, and consisted of members invited to take part in three-day workshops (learning sessions) held at the national level. iii. The district QI process is directly supported by only one of the two projects in terms of training and funds. As a result, the district teams work more closely with the HCI￾supported health facilities. iv. As noted, the district QITs were established later than the regional QITs. Linking the two is challenging because their focus is very different. District QI teams are actively engaged in mentoring and coaching at ART sites, both as an independent effort (with no accompaniment from regional or national teams), and on occasion when regional/national teams come to visit sites in the districts. The relationship between Formative Evaluation of Quality of Care Initiative by Ministry of Health |6 the two teams is improving but remains weak. v. While many district teams had established guidelines for QI initiatives, the team members were not well-informed about what they were expected to do. Moreover, not all district teams have access to the continuously revised tools. The role of the districts is management, but district staff often view QI as more of a technical endeavour. vi. In some districts the leadership of the district QI teams was based on workload distribution across the DHT rather than based on capacity considerations and existing responsibilities that would grant them a comparative advantage in providing HIV care. In one district, a nursing officer was chosen to lead the team who was not involved in delivery of HIV care rather than the district HIV focal person or those providing ART care. By contrast, the AIDS focal person in another district led the QI team and this appeared to contribute to team effectiveness in that district. vii. In all districts with established QITs, the teams prepared reports on the sites, but did not identify particular district problems or develop accompanying work-plans for district-level action. The teams rely on the supervision plans, which are vertically funded by the QI programs. Nevertheless, QI activities are reported on at DHT meetings and through these sessions many participating facilities have been able to obtain quick solutions for problems like ARV stockouts by borrowing from other facilities or following up with NMS. viii. Though the vertical QI funding mentioned above ensured that supervision was provided and data collection was conducted, District QITs were not offered guidance on implementing PDSA cycles at their level. They were also not guided on how to connect with QI/QA initiatives at the district level, particularly the national program Yellow Star. QI was and is still implemented on a project basis... ix. Though efforts to integrate QI into DHT activities are commendable, the overall clinic-based design of QI did not adequately assess and harness the district leadership role in providing ART support. D. Facility Level i. All facilities visited were informed about QI initiatives and contained QI Teams implementing QI initiatives with the exceptions of Iganga hospital and Rubongi hospital. The teams generally consisted of both clinical and non-clinical staff at the hospitals, including clinicians, laboratory, pharmacy, counsellors, nurses and records clerks. Iganga and Rubongi hospitals have undergone changes in their management and consequently no longer maintained the required leadership in QI. The administrative staff in these facilities were not adequately involved in the QI and could not support the process without clinical leaders. ii. Of the 24 health facilities with QI teams, 20 reported continuous improvement in the quality of ART care provided, and four of these 20 said they had experienced great improvement as a result of their QI processes. This is demonstrated by the change in national quality indicators they set out to improve. iii. Specific areas of improvement at the facilities included comprehension of the ART guidelines and the use of patient data to improve areas like record retrieval, client follow-up and reduction of patient waiting time. All facilities had locally developed ART management flow charts but only those with active teams updated performance graphs displayed in open places. They also experienced general improvement in the Formative Evaluation of Quality of Care Initiative by Ministry of Health |7 integration of ART, TB and HCT services. iv. However, a heavy workload has remained a big problem relative to the small facility space, limited furniture and limited staff size. This common problem has been resolved through a number of solutions generated in the QI process. These include making appointments for clinic patients on days with typically low attendance and promoting client involvement in non-technical tasks like registration, card retrieval, and filing. Nevertheless, the problem has not been fully resolved, especially in facilities like Kiswa health centre III with a very large number of clients (more than 600 OPD cases per day), where staff reported there was “no time” to conduct QI. v. Many clinical staff mentioned that issues that have not changed much that require external intervention include availability of drugs and other commodities, infrastructure development and staff allowances. A number of staff in facilities visited stated that QI processes are currently increasing the workload due to the increased amount of time required for data processing and planning. QI is viewed as an initiative that still requires nurturing. vi. Each team had a senior clinical member (in terms of years served or professional level) who served as the team leader for coordinating and guiding team meetings and sharing lessons learned. Whereas team senior-level clinicians at the ART clinics assumed leadership, nursing staff at the ART clinics managed the day-to-day QI activities. vii. In some health facilities, staff Clients waiting to access services at Bar HC-III trained in QI were either transferred or rotated. This negatively affected the enthusiasm, pace and continuity in QI activities at those respective health facilities. viii. QI planning and review meetings took place at all sites, and QI was integrated into the ART teams and other AIDS services meetings, e.g., nutrition and HIV (Supported by Nulife), ART scale-up meetings (Supported by NUMAT), and paediatric ART scale￾up supported by Baylor College of Medicine). The frequency of meetings varied, with some facilities conducting more than one per month either on the day of the ART clinic or afterwards to address QI issues as they arise and monitor progress on interventions; others, especially those with limited resources, reported less frequent meetings. ix. In most facilities, there was evidence that the PDSA approach had been applied to QI. Problems were identified after reviewing clinic activities, work plans that were developed, and the implementation of actionable points for QI that were initiated. Despite this paper work, there was little evidence that individual roles in the improvement process had been communicated to all staff, especially those outside the ART clinic. x. In some facilities QI awareness grew, with PDSAs used in the male ward in Tororo Hospital, and in Entebbe hospital and Bukuuku HCIV plans were made to expand QI to the OPD and MCH. Formative Evaluation of Quality of Care Initiative by Ministry of Health |8 Formative Evaluation of Quality of Care Initiative by Ministry of Health |9 xi. The introduction of QI appears to have led to significant changes in facilities that already provided ART services. Improvements were noted in areas such as completions of ART patient cards, which resulted in improved patient monitoring and assessment, follow-up and adherence to treatment. Structural challenges such as frequent ART and OI drug stock outs and lack of CD4 count machines have continued to undermine QI efforts. QI initiatives appear to have had a limited impact on facilities that already had severe structural challenges. For example, in Bar HC-III there were few staff trained, and some of those who were had been transferred so there is no access to CD4 counts. In Kuluva Hospital, however, HIV/AIDS programme support by IRCU and PACE provided a good basis for implementing QI. xii. Most facilities had not recently conducted cross-collaborative meetings and networking (joint trainings and experience sharing) with other facilities, especially in the Eastern, Central and Northern regions. xiii. In most facilities, there was evidence of improvement based on the indicators they were tracking. The improvements were illustrated in the charts and flow diagrams in the reports. The table below summarizes some of the indicators assessed during the facility assessment (See annex 3). Client at Kuluva Hospital that had received a Water guard, jerrycan and mosquito net provided by PACE project Formative Evaluation of Quality of Care Initiative by Ministry of Health |10 Table 1: Assessment of the performance of the health facility QI teams in improving the quality of HIV/AIDS services Roles of health facility QI teams Assessment1 Comments 1) Improve referral of HIV+ persons to ART sites and their selection for treatment +++ • Most facilities have improved internal referrals, which was one of the first problems addressed. • Mechanisms for external referral were also developed. 2) Improve care that conforms to ART clinical guidelines +++ • The areas that were improved include increased enrolment, and TB assessment. • In some places the people providing care do not have the necessary clinical skills. • Though QI has helped spread out the high patient load over several clinic days, quality is compromised in a number of facilities because patient numbers are still high and staff work loads are high as well. 3) Maximize adherence rates ++++ • The completion and retrieval of patient registers have improved. • The involvement of expert clients in patient follow-up and defaulter tracking and communication are viewed as a success. 4) Ensure accurate lab testing + • Neither the internal nor the external Lab QA is linked yet to the QI processes. 5) Ensure drug availability and supplies + • This core problem has been inadequately addressed by QITs. It should be a focus of District activities. 6) Improve patient flow. ++++ • Patient flow has improved and patient waiting times have decreased. • There are flow charts and stations. • The number of clinic days has increased. 7) Ensure that care and treatment guidelines, job aids, essential equipment and drugs are in place + • This is not directly addressed by QI. 8) Conduct teams’ self-assessment + • This is performed weakly and is not guided by specific tools. 9) Produce rapid cycle improvement (plan-do-study-act) +++ • Though many were aware of PDSA cycles and are following a similar pattern, the activities are arranged to solve problems during meetings rather than be small additive solutions. 10) Build on successes ++ • This has not been demonstrated yet. The current implementation approach does not offer guidance on how success that has been achieved can be sustained. 11) Share lessons learned among the team members and across teams ++++ • This is done through participation in district QI teams and regional training sessions. 12) Combine clinical and non-clinical staff of the hospital +++ • They are mainly ART teams that include non-clinical staff but not administrative staff.. 13) Develop an action period work plan +++ • Many facilities have incomplete plans. • The work plan has not been communicated to all implementers. 14) Conduct a simple analysis in order to use the data they are gathering for decision-making +++ • This is being done through the registers, plotting data, recording documentation journals, and making decisions based on this data for the ART clinic. 15) Prepare run charts with the assistance of experts in data management ++++ • They have posted them on the walls.                                                             1 Assessment based on Delphi technique involving consensus of six consultants: + = Strongly Disagree, ++ = Disagree, +++ = Agree, ++++ = Strongly Agree 3.1.2 Key factors facilitating or hindering improvements in HIV/AIDS services A. Facilitating Factors • A great deal of interest has been expressed by funding partners, especially the U.S. Government and JICA. Funding continues to be available for QA/QI initiatives. • Activities are taking place across a number of facilities; these are termed collaboratives in HCI and networks in HIVQUAL. Despite the organizational and reporting differences across the QI initiatives, the implementation processes and principles are essentially the same. • The technical core team at MoH ensured that the partners interacted with each other and with the programme managers routinely. • There are full-time focal point persons at the national level for the two major QI initiatives who accelerate the process. • There is a national training team across all initiatives that has provided for sharing of resources and standardization of trainings. • The teams at the regional, district and facility levels have been receiving funds for meetings, coaching and supervisions. This has facilitated data collection at the implementing sites • The implementing sites are represented on the district teams. • There is on-site programme support (IRCU, NUMAT, STAR, JCRC and TASO), and referral support for services not available at the sites, especially paediatric HIV services and CD4 count services. • Facility QI teams provide teamwork, strong leadership and training in QI. B. Hindering Factors • The QI initiatives and other projects are run vertically with different indicators and different reporting arrangements. No system exists for reporting from the implementing facilities to the districts and then to the national level. Some districts report at the national level while others do not. • Although the two projects are seen as extensions of previous interventions, they have not built on previous QI projects, e.g., Yellow Star, EGPAF QI initiatives. • The core technical team meetings operated parallel to the ACP management meetings and QI is seen as detached from ACP even though they are linked • The location of HIVQUAL in ACP and HCI in clinical services does not promote integration and good coordination. For example, there is a perception that the required QI data elements within ACP are designed to inform central management decisions compared to QI data elements based outside this institutional framework. • The QI initiatives focus on ART clinics yet they should link with all other services. This undermines sustainability and tends to portray ART as a super program. Synergies should also exist with other programs especially those with a high patient volume, e.g., reproductive health and malaria. • The nursing department in MoH is not involved in the quality initiatives at the central level yet nurses conduct most of the implementation of the AIDS response and QI process. Consequently QI is not included in the job descriptions and oversight duties of nurses Formative Evaluation of Quality of Care Initiative by Ministry of Health |11 • There is weak collaboration between HIVQUAL and HCI, especially at the district and facility level where differences in tools and data collection requirements appear to be very large. • At times there are too many workshops, which interferes with service delivery. • Staff transfers and rotations interfere with active QI work. • There is inadequate material and technical support for QI activities. • There are perennial stock outs of ARTs and OI drugs, and limited access to affordable CD4 count testing. • Support and incentives are lacking that would help successful facilities hold on to the gains they have made Discussion The implementing partners have financially supported the QA/QI initiatives and the Ministry of Health has provided leadership, as required to accelerate the processes. The focal point persons for the two initiatives at the national level have facilitated sharing of resources and standardization of trainings, which have produced positive results with respect to service provision. However, sharing of resources is not occurring at the district level, where the two initiatives are operating vertically. This has resulted in different reporting arrangements whereby some districts are being bypassed when reporting to the national level. Even at the Ministry level, the operations of the technical teams are detached from the ACP. This does not augur well for long-term integration of the initiatives, which in turn will impact their sustainability. 3.1.3 Collaboration and coordinating The evaluation team assessed the QoC initiative’s collaboration and coordination with its partners. This was judged in terms of leadership, coordination and collaboration structures and functionality of these structures. A. National level • The Core Technical Team (CTT) developed a draft strategic plan for QoC that has not been reviewed yet by the Steering Committee. • The collaboration and coordination structures in place are the QoC Steering Committee and CTT. • The steering committee includes MoH and partners supporting QoC with regard to HIV/AIDS care. This forum presents an opportunity for the main partners to meet and make strategic decisions. The core technical team is composed of heads of ACP and sub-programs such as PMTCT, paediatric AIDS, nutrition and laboratory services. The steering committee is supposed to meet quarterly but last met in September 2009. The main cause of the delays was the committee’s failure to achieve a quorum due to competing priorities. • The CTT is composed of technical staff from MoH, HCI and HIVQUAL project staff, Baylor College of Medicine, JCRC, MJAP, Mildmay and NUMAT. The CTT meets monthly but the process for reporting to the steering committee or elsewhere is unclear. Formative Evaluation of Quality of Care Initiative by Ministry of Health |12 • An Annual QoC conference that brings together the regional coordinators and key stakeholders and implementers is held annually. This brings together most of the key implementers and has facilitated a sharing of experiences. • Despite efforts at coordination there are still fragmented approaches to quality improvement for ART care, as evidenced by different indicators, training curricula, reporting mechanisms, areas of focus and scale of intervention. • The collaboration and coordination of the QoC initiative is channelled through two areas of the Ministry of Health -- HCI, through the clinical services department, and HIVQUAL, through the ACP which comes under the National Disease Control Department. “QoC should have been under the Quality Assurance Department but this has not happened. It is still under the clinical and community services department.” Key informant, Core Technical Team/Clinical Services Department B. Regional level • Little coordination and collaboration of the partners has occurred at this level. The regional trainers are an extension of the national technical team. • In some instances, the regional team members are not part of the existing technical support supervision structure at the regional referral hospital. Examples include Masaka and Lira. “There is no harmonization and coordination of the implementation partners at our regional level.” Key informant, Fort Portal Regional Hospital C. District level • There was initially no participation in the coordination and collaboration of partners at the district level. However, late in 2009 district quality teams were trained and established that are becoming more involved in this area now. “There is very little involvement of the district in the coordination of quality improvement activities so they just transfer staff without consideration.” Key informant, Entebbe hospital • In the districts where Quality Improvement Teams have been formed, support has been provided for project review meetings, which are held quarterly. “It was at such a meeting that we identified the problem of loss to follow- up among clients attending ART clinics in the facilities supported by different partners.” Key informant, Kyenjojo District • The district teams meet regularly and submit activity and financial reports on the projects. • Clear guidance is generally not provided on how the districts and regional hospitals collaborate and coordinate their activities. • In the districts visited, QoC activities were not included in either the quarterly or annual plans. However in all of these districts, progress on QoC activities was cited in the DHT meetings. • Coordination and collaboration is particularly good in instances where facility team members are included in the district QI teams. Formative Evaluation of Quality of Care Initiative by Ministry of Health |13 D. Facility level • Clinic “Site Teams” have been formed to improve service delivery in facilities providing ART services. These committees function at different levels in the health facilities, and meet on a weekly or monthly basis to see whether their respective interventions are on course. • Some staff members of the QI teams have interacted with those at other facilities through trainings in the East and North. The western and central regions have also conducted collaborative meetings. In some districts, there are heath facility QI team members on the district QI teams, which enable them to share experiences across facilities. • There has also been collaboration between the health facilities and the regional/central teams in the districts of Masaka, Kyenjojo and Kabarole. This has included reported support supervision/mentoring sessions with feedback recently. Discussion The QoC initiative has been effective to some extent in developing collaboration and coordination among QoC partners. A draft strategic plan has been prepared and the coordinating structures are in place, including the Steering committee and Core Technical Team (CTT). The CTT brings the partners together in regular joint meetings and in an annual implementers meeting where experiences are shared. The Quality Assurance Department has improved with respect to both leadership and staffing, and can now ably take on the role of overseeing QoC coordination. The district-level teams are working well with the health facility “site teams” and have regular meetings and report to the centre. However, clear guidelines have not been issued on the role and responsibilities of regional trainers, and their coordination and collaboration with the districts is still weak. The regional teams are not part of the existing technical support supervision structure, which would probably have streamlined their coordination with the districts. 3.2 Objective 2: Development of the Capacity of the Ministry of Health to Improve and Maintain the Quality of a Comprehensive HIV/AIDS Program 3.2.1 Capacity-building in the MoH Our overall finding is that capacity-building was a central element in the support that both projects provided to MoH. Six main elements of capacity-building are evident in the QoC implementation process as described in reviewed documents and interviews with respondents at different levels: 1) training; 2) provision of tools and materials; 3) on-going coaching and mentoring; 4) mobilization and provision of resources; 5) data and information management; and 6) structural adaptations to enhance QI. Our findings below on capacity-building are structured in accordance with these six elements Training HCI and HIVQUAL projects have conducted training at four main levels: a) national level (through HCI collaboration with IRCU); b) regional level (primarily through Regional Formative Evaluation of Quality of Care Initiative by Ministry of Health |14 Referral Hospitals); c) district level (through members of the district health teams, and representatives of ART clinical staff reached by the initiative); and d) health facility level (especially focused on the staff and systems of the ART clinics, and on selected clients as volunteers in different areas of service provision). Training has occurred in all of the facilities that were visited. In 10 of the 24 facilities training has occurred in the last 12 months. The objective of the QoC initiative and the support provided by the HCI and HIVQUAL projects was to build upon existing knowledge and skills in general health care, AIDS￾specific care and treatment (including ART); and health care management (e.g., team￾building, support supervision, etc.). We did not find either a comprehensive and systematic description of the specific quality improvement training envisioned at different levels or a structured training needs assessment of different stakeholders. We found that the general experience regarding QI training was that staff at the different levels were trained through a combination of QI learning workshops (usually lasting one to three days), provision and exposure to a range of written materials on QI, and mentoring and coaching during site visits. The expectation was that staff trained through workshops would impart their QI knowledge and skills at their respective stations of deployment. This would be done mainly through structured learning sessions with staff and volunteers as part of continuing medical education and professional development programs at health care and health management settings. At all levels of focus (national, regional, district and health facility), gaps were reported in personnel who would receive training in QI, both in terms of the number available and their skills, especially in ART and health management. For example, HCI had to second a staff member to MoH to coordinate the initiative, and there were unfilled positions in the district teams and all health facilities visited in the Lira and Arua health regions. Very few staff in the health facilities visited had received training in comprehensive AIDS care. All of the health facilities visited in the Lira, Masaka, Jinja, Fort-Portal and Arua health regions reported frequent transfers of staff both between facilities and out of the districts, often resulting in ‘loss of staff’ with ART and/or QI training. Although HCI project documents reviewed indicated that the project generally facilitated access of staff in supported health facilities to ART training, no such experience was reported at the health facilities visited in Lira and Arua health regions. QI teams have been formed; 13 of the 24 facilities have developed work plans and meet weekly or monthly and as part of routine staff meetings. In general, QI teams visited at the district and health facility levels cited successful efforts to share skills acquired in QI training workshops with other staff. An example of local QI capacity-building was the one-hour CME session on 7 February 2010 on Quality of Care Initiative in HIV/AIDS, presented by Dr. Aliga Simon and Dr. Drazu Stephen at Kuluva Hospital. The session was attended by 11 staff spanning all hospital departments, including Administration and the Nursing School. However, this effort has not reached all staff or maintained QI as a staff priority in all cases. Staff in the maternity/MCH interviewed by the study team had heard about the AIDS QI process and appreciated the need to reflect on quality and its improvement in MCH, but had not yet had the opportunity to apply it in their work. Formative Evaluation of Quality of Care Initiative by Ministry of Health |15 Tools and materials for QI The objective of the reviewed HCI and HIVQUAL support activities was to develop, provide, and promote utilization of necessary QI-related tools and materials. This includes materials to enhance understanding of the basic QI elements, and tools to guide QI practice, e.g., QI planning and documentation, the collection and processing of QI data; and communication of QI progress both within the health care site and with other stakeholders. Interviews with respondents and records reviewed indicate that the materials and tools provided were useful in the QI implementation process, especially in on-site training of other staff, and in documenting the QI improvement process. In general, the tools for communicating QI information, such as graphs and reporting formats, were not displayed on notice boards in hospitals. However, HCIVs and a few HCIIIs displayed graphs on key indicators. In some of the health facilities visited, the charting and discussion of the progress made regarding QI indicators of focus was not sustained, particularly when targets were achieved and sustained for some time. Reasons for that included reduced motivation, transfer of staff who were key in leading and promoting communication, work overload, inadequate funding, and lack of incentives. Visits to the health facilities and district teams revealed that the main mechanism for distribution of QI tools was learning sessions. This largely reaches those who attend these sessions, but this depends on their vigilance with regard to selecting all necessary materials, and making them available for use (and necessary reproduction) at the ‘home’ station. As a result, we found that the tools were not uniformly distributed; for example, a modified site coaching and mentoring guide was found in use in most facilities visited in Arua, Masaka, and Jinja regions, but not in Lira and Fort-Portal regions. Some of the District and Regional QI team members interviewed said there were no specific coaching and mentoring tools at the district and facility levels (only the ART clinic); which may explain the absence of formal and documented coaching processes at these levels. In addition to materials to support QI, the QoC implementation process included the provision and utilization of tools and materials to enhance clinical care, including basic AIDS care tools such as ART cards, pre-ART cards, and AIDS service registers. The initiative provided buffer stocks of such stationery supplies to health facilities, and facilitated advocacy for them at the district and national level to ensure due provision of future stocks. It also supported training and coaching of health unit staff to enhance their skill in making proper use of the tools, and provided on-going support through random checks to ensure correct use of the tools (e.g., entry of records, processing data forms, and correct computation). Local analysis and application of QI and general AIDS care data was emphasized, and is well appreciated at all levels, as the best way to detect and take timely action regarding problems regarding the use of tools. At most health facilities visited, we found that health care teams, often limited in number and skills, and overwhelmed by their patient load found the demands they received for AIDS care and QI data overwhelming and difficult to meet. A district team Formative Evaluation of Quality of Care Initiative by Ministry of Health |16 observed that data load complaints at health facilities are also often linked to long-term work situations where serious record keeping has not occurred. It was remarked that “When the wrong practice is consistent; it becomes normal” Coaching and mentoring Coaching and mentoring was conducted at visited health facilities by combined teams of national, regional and district coaches. At most sites, the visits took place less frequently than the monthly sessions that were planned. This occurred mainly because of limited resources. Although some district QI teams said they had made an effort to integrate QI coaching into routine support supervision, this had not been effective because of the limited amount of time they could spend on the full coaching process, as expected in QI. The site coaching process was clear with regard to expectations and documentation of the coaching and post-coaching actions required/planned. However, site QI team members stated that the QI process is less clear with respect to the coaching role of site teams, supporting other AIDS care staff, and applying QI to other services within the facility or other facilities. In districts where QI teams had been trained and initiated QI coaching, an effort was made to continue and document it. However, clear guidance was not provided on the need and provision for on-going coaching and mentoring of district QI teams beyond occasional invitations to learning sessions. In some districts QI members cited a need for further coaching that would include accreditation of existing and new ART sites, and the requirements and process to apply QI to other health services. Regional teams were clearly serving as mentors, working with national coaches from HCI and district QI teams. Coaching and mentoring was reported for the national core team, largely by working alongside HCI and HIVQUAL staff in supporting QI activities in the field, and in occasional learning opportunities abroad. We were told HCI provided QI mentoring support to national AIDS service partners, e.g., Inter-Religious Council of Uganda, through another USAID￾funded initiative called the AIDS Capacity Enhancement Program. Interviews with IRCU acknowledged that the QI support they received from HCI enabled them to develop a quality assurance strategy for their AIDS support programs. However, the strategy developed has not yet been implemented because their AIDS program funding has run out. They plan to use the same strategy in rolling out new programs funded since December 2009. Resource mobilization and provision The two projects enabled the QoC initiative to mobilize and empower people to deliver QI in the face of critical HRH shortages using many staff with rudimentary AIDS care skills. Major task shifting and role changes were supported by the QI initiative (e.g., nurses deployed as counsellors, and AIDS clients helping with service provision tasks). The projects also resulted in improved time management, thus enhancing service delivery efficiency at AIDS care clinics by modifying client flow systems. This was an element of focus in project support to sites, and an area that was acknowledged to have made a major impact at most facilities visited. The focus on quality enabled facilities and districts to identify and quantify needed resources (drugs, lab equipment/reagents, etc.) that would permit ‘minimum service/management’; and Formative Evaluation of Quality of Care Initiative by Ministry of Health |17 to lobby for support to put them in place. We were told that a minimal level of services was lacking at many facilities, and there was an urgent need for program support. Places where such program support was available enjoyed a clear advantage in receiving and responding to QI-related processes. For example, NUMAT has been a key contributor to AIDS service provision in Lira region, and this is regarded as a good platform for QI. Data, information and management The projects have supported capacity-building for quality assurance with regard to routine data, including data recording and processing, and have facilitated use of information for local decision making and action. With respect to managing and using QI data, the projects have supported the process of generating and adopting 46 national QI indicators and their dissemination for use by facilities. Most of these indicators have been addressed through QI collaboratives (see summary in Table 2 below). Table 2: Key collaboratives and indicators of focus Collaborative Indicators of focus Initiate ART for everyone who needs it 1, 7a, b, c, d, e, f, 28, 29 Retain in care everyone started on ART 12, 13, 18a, b, 19, 20, 22, 25, 26 Lead to good clinical outcomes 8 a1, a2, 14, 15, 17a, b, 23 Laboratory strengthening 5a, b , 6a, b, 9, 16, 24a, b District collaborative 21a, b TB-AIDS collaborative 4, 11, The key gap noted with respect to the national indicators concerned AIDS and reproductive health (e.g., indicators 2, 3a, b, c, 10a, b, 27a, b, and 30). Indicator tracking does not pick out some of the evident quality gaps at sites and district levels such as waiting times, stocks of ARVs and OI drugs, key lab supplies and CD4 access. Standard measures for the most important of these, collected across all facilities, would enhance an understanding of facility capacity even if they are not considered national indicators. Structural Changes At the national level, there has been a drawn-out process for finding an institutional home for QoC within the establishment. Project support has operated through Clinical Services and ACP, with a view to settle with Quality Assurance. Some of the gaps in QI implementation (HRH, drug supplies, laboratory/CD4, and communication) may be attributable to limited involvement by leaders in these areas. The projects have backed an effort to support establishment of sustainable strategic and technical leadership for QI. The Steering Committee has enabled QI to “take off” but requires a long-term institutional fit within the Quality Assurance Department. Technical leadership capacity for QI, intended to involve individuals/teams within and across service/disease programs, has developed so far among selected individuals in HQ and RRH, and mainly within AIDS programs. At the regional level, QI technical leadership has been developed through Regional Referral Hospitals. RRH staff were active in the process, co-opted on the national core team and district QIT, and used Formative Evaluation of Quality of Care Initiative by Ministry of Health |18 as a direct technical resource to site teams. However, we were told in several regions we visited that major ambiguities and a strained relationship continue to exist between regional staff and district health teams with regard to collaboration on technical support supervision of health services in general, and QI in particular. At the district level, QI is an integral element in managing the district’s response to AIDS.. We observed progress with respect to orienting and involving most DHT members to QI, and noted that including ART clinic heads in DQI teams provided a critical advantage. However, there was no clear role for the Health Sub-district in QI. At the site level, the key structural issues identified regarding QI included integrating expanded AIDS services into other operations and services at health facilities; work allocation and task shifting among staff; and the role of clients as service providers – expert clients; Treatment Support Teams, Network Support Agents; Case Managers (NACWOLA- in STAR-EC), etc. Conclusion Capacity-building is clearly central to the process of rolling out the QoC initiative. However, it has not been clearly defined in a framework that describes the key elements of capacity￾building and how they relate to each other. This may be a direct result of the approach adopted in providing project support to the QoC, which was largely evolutionary. The steps taken by the two projects in supporting QoC were different in a number of ways, which made it difficult to conduct aggregated analysis on their combined contribution during this evaluation. It was also apparent from the evaluation findings that a) linkages between the different elements of QI, and between QI and ART program development, were not always adequately addressed; and b) informed linkages between QI teams at different levels were constrained in several ways. Such issues can be detected and addressed in time in the context of a comprehensive QI capacity-building framework. Although multiple QI training sessions were conducted at all levels, data gathered during this evaluation was inadequate to provide a comprehensive understanding of the entire QoC training plan. The wide range of tools (both QI-specific and general AIDS service tools) was found useful at all levels. However, a number of respondents, especially at the facility level, were concerned that the data and information management requirements for QI introduced an added workload that was difficult to absorb into routine work schedules. This may partly explain the general trend in which QI data management began impressively, but gradually waned in frequency of measurement and reporting, quality of data, etc. It may also explain the limited effort we found to link QI data with routine health data captured and reported (e.g. through routine HMIS). It is apparent from the evaluation findings that sustained and effective QI coaching and mentoring is critical at all levels, yet difficult to achieve. The emerging constraints to this process include: a) inadequate funding for district and facility-level QI processes; b) limited integration of the QI support process into the existing mechanisms for technical and management support supervision (e.g., Area Teams and Regional Referral Hospitals); and c) limited decentralization of the QI support process, especially between the district and health Formative Evaluation of Quality of Care Initiative by Ministry of Health |19 Formative Evaluation of Quality of Care Initiative by Ministry of Health |20 sub-district levels. These issues need to be addressed in the planned scale-up of the HSSP III. 3.2.2 Integration and spread of QI approaches Some members of other departments such as Clinical Services, Quality Assurance, and AIDS Control Program were trained as part of the national core team of MoH. Some of them are on the CTT and the steering committee of the QoC initiative. We found that QI approaches spread to other service departments in five of the 24 facilities, mainly from the AIDS clinic to the MCH clinic (promotion of adherence) and from the AIDS clinic to the OPD and wards (screening and addressing TB/HIV co-infection). This spread has mainly occurred through staff rotation rather than training and skills transfer. However, no deliberate effort has been made by the QoC initiative to develop the capacity of the other departments in the MoH to integrate QI approaches. “The QoC initiative hasn’t had much influence on other departments in the MoH to implement the quality improvement initiative.” Key informant, QoC core technical team/Clinical Services MoH. Conclusion The QoC initiative has not been entirely effective in developing the capacity of other relevant departments to integrate QoC. The QoC initiative trained some staff members of the departments of clinical services, quality assurance and the AIDS Control Programme, but there has not been any strategic plan yet to integrate the quality initiative into other departments. This is mainly attributable to the fact that leadership for QoC has not been assumed by the Quality Assurance Department, which is mandated to address this within the existing health care system. The QI strategy has only spread to a limited extent to other service areas or departments within facilities. This has occurred because there was no intentional effort to do so; a few facilities that felt the need to do so went ahead and spread QI to other service areas or departments 3.2.3 Factors affecting the institutionalization of the QoC initiative Institutionalization2 was considered from the perspective of: 1) the internal enabling environment, including policy, leadership, core values and resources; 2) the structures necessary for institutionalization, which includes oversight, coordination, roles and responsibilities and accountability mechanisms; and 3) essential support functions, including capacity building, communication and motivation for high-quality work. All of these affect institutionalization in one way or another, as discussed below.                                                             2 According to the QAP/HCI documents reviewed, quality assurance will be institutionalized when it is formally and philosophically incorporated into the structure and functioning of a health system (or organization), consistently implemented, and supported by a culture of quality, as reflected in organizational values and policies that advocate quality care. Although the HSSP-II and the NHP did not explicitly make reference to Quality of Care, in 2005 the Quality Assurance Project (QAP) was implemented. This not only preceded the Healthcare Improvement Project (HCI) and the HIVQUAL project but served as the “de facto” MoH stance regarding QoC. Thus, in such a documented policy vacuum, the Director General for Health gave instructions to the staff to implement the HCI and HIVQUAL projects. Meanwhile, the two projects planned to support the MoH in developing national QI objectives and indicators, and gain their approval; and have QoC included in the HSSP-III and NHP-II. It was determined that although Regional and District QI teams were established, there was no clear policy on how the referral hospitals and districts would work together in ensuring that QoC was implemented. The MS of Lira stated that, “There is no policy on how the RRH and Lower Health Facilities would relate to each other as RRHs are under MoH while Lower Health Facilities are under local governments. Yet RRH should be responsible for support supervision to the district hospitals that in turn would be responsible for the other facilities. Currently, some district Health Facilities question the roles of the RRH except when executed in the form of RQI team”. QI was introduced to the other development partners at the operational facility level by QI teams with limited institutional/policy level support. This has had a bearing on the acceptance and harmonization of the QI approaches used by the partners and those approaches advocated by HCI/HIVQUAL. In general, development partners sign an MOU with the MoH, but this MOU is unclear with respect to the relationship among health facilities, QoC partners and other development partners. A case in point is Arua RRH, where it was determined that an MOU had to be signed between the hospital and MSF in order to harmonize the working relationship between MSF, which has local and international mandates, and Arua RRH, which only has a local mandate. It was also determined that health facilities did not have an internal policy on QoC. Only in the case of Lira RRH was QoC mentioned in the Hospital Mission displayed in the Administration office. However, even in that instance, QoC was not explicitly mentioned in the core values of the hospital. Instead, as expressed in Orum HC-IV, “QoC is perceived as a government policy for improving delivery of ART services because it was introduced to them through training that was approved, supported and implemented by MoH and health development partners”. In this instance, however, it was expressed in future terms, as the policy on QoC becomes explicit, caution should be taken in ensuring that the policy is accompanied by additional resources. Simply integrating QI into PHC funding would be useless since all DHOs observed that current PHC funding was already inadequate and extending it to QoC would not serve any purpose. Leadership in promoting QoC and ensuring that necessary support for it is provided at the national, regional, district and facility levels is critical to the institutionalization of QI in Uganda. In that regard, it was determined that at the national level, HIVQUAL would work closely with/through two MoH staff (in ACP) while HCI would support the position of the National Coordinator in MoH; and at the regional, district and facility levels, a QI team leader would be elected by teams at those levels. Formative Evaluation of Quality of Care Initiative by Ministry of Health |21 At most of the facilities, the Medical Superintendent and the administration are providing the necessary leadership and support for QI. For instance, the management of Arua Hospital has demonstrated a clear commitment to quality improvement, according to an MSF field officer. Similarly, the team interviewed at Namungona Hospital said it has received strong support from the facility’s administration, as well as encouragement to take QI to other services. Meanwhile, the MS of Lira Hospital has a good grasp of QoC from a structural perspective but is not informed about the details of process and outcomes of quality improvement at the facility. Since the DHO is responsible for all health-related matters in the district, it is assumed that he is also accountable for the activities of the district QI team and for providing the necessary leadership to the QI team. A review of the minutes of the district QI teams showed that the DHOs did not attend the teams’ meetings to provide the necessary leadership and guidance for QI. What’s more, the leadership role of the DHO is bound to increase as QI spreads beyond HIV care. In addition, concern was expressed that transfers of staff by DHOs were occurring without taking into account the impact of such administrative actions on QI. It was also noted that the creation of new districts posed a challenge to QoC activities because of (a) the establishment of a district QI team for each new district; (b) the identification of new members for replacement on the parent district QI team; and (c) the provision of training to new members of the QI teams. Although the projects have developed QI “toolkits,” these do not constitute an adequate replacement for training and practical hands on coaching. QI cannot be attained in any setting if the requisite human, material, and financial resources are not made available in appropriate quantity and quality, and on time. Accordingly, HCI and HIVQUAL planned to work closely with each district team to integrate QI activities in a way that could be supported on the basis of the resources made available by the district. Support has only been provided during the period of demonstration, wave 1, and wave 2, or during phase 1 and phase 2, when HCI has provided the necessary funds to support the QI activities of the QI teams at the different levels. It was envisioned that the districts would be able to conduct quarterly coaching visits using funds that they normally budget to conduct these visits. However, district budgets are severely limited and supplementary funds continue to be needed. The DHO of Arua summarized the situation by stating that, “The Indicative Planning Figures for this financial year are not any different from last year and it will therefore be very difficult to include QI activities in the already over constrained district budget” The cup is not full. At all of the facilities visited, accounts were given of how inadequate human resources were resulting in heavy staff workloads. The facilities have responded to this by training as many health facility workers as possible and rotating them throughout the HIV/AIDS clinics, but staff burn out also ensues. They also raised the issue of drug stock outs. While government policy is that ARVs should be available to all who are enrolled, some facilities experienced stock outs of ARVs from time to time. One of the QI requirements for HIV care is that CD4 counts for clients should be taken every six months. However, it was determined that CD4 services are generally inaccessible to most clients. This has occurred because machines at the health facilities are not functioning due to breakdowns and/or lack of proper maintenance and repair; a reduction in donor support for free services, resulting in a requirement of payments or an increase in payments; and generally unaffordable CD4 services by apparently un-regulated Private for Profit providers such as Cynapsis in Lira. Formative Evaluation of Quality of Care Initiative by Ministry of Health |22 In order to provide QoC oversight, a national QoC Steering Committee was established with support from HCI and HIVQUAL that was to meet quarterly. At the regional, district and facility levels, the objective was to institutionalize and sustain a culture of continuous improvement through establishment of quality improvement teams at the regional level and within the DHT, and at the facility level. Thus, HCI would build the capacity of teams to plan, manage, monitor and spread QI activities in HIV/AIDS in their areas of jurisdiction. The structures used to operationalize the QoC activities thus far have not been fully integrated into the MoH structures. Hence, there is an immediate need to regularize and integrate QI teams into the structures of the MoH and health facilities with clear terms of reference and composition. In this regard, the composition of the QI teams has been expanded over time. The expansion and reconstitution of the teams is likely to continue as QoC becomes not only institutionalized but also adopted by the entire health service delivery system in the country. Accordingly, the job descriptions of the office bearers included in the final composition of the QI teams should be defined and the appointment letters of the members accordingly revised to reflect this additional responsibility. We also found that the steering committee which is supposed to meet quarterly has not done so. Similarly, the QI teams have not been able to meet as regularly as stipulated. A review of the minutes of the QI team meetings showed there is poor follow-up of the issues discussed, suggesting that holding such meetings could be an academic exercise or a requirement of the HIC/HIVQUAL project. Indeed, efforts to establish the extent to which QI issues are presented to the various organs of the district tended to show that beyond the district QI team, there is no evidence that information on QI is presented to such district organs as the technical planning committee, and health secretary and council, let alone the facility management committees/boards. Without such transparency, it is difficult to hold leaders accountable so that they in turn will ensure that members of the QI teams and health facilities carry out QI activities. Since many health stakeholders are involved in implementing QoC, it is important that their respective roles and responsibilities be clearly defined and integrated into the routine job descriptions of key individuals. At the facility level, the HIVQUAL project strengthened and expanded the inclusion of Network Support Agents in site QI teams in order to improve community outreach and follow-up of patients. It has been determined that counselling plays a critical role in the provision of QI services. However, at the national level the specific roles of the nursing profession have not been adequately included in the design and delivery of AIDS services or in the official definitions of nursing responsibilities. Nurses often assume roles that are not traditionally considered core to the nursing professions in Uganda such as clinical consultations, dispensing and counselling. Consequently, the delivery of these services by nurses does not fit in with the nurses’ current job descriptions and is not included in their performance appraisals. This creates a disincentive for them to learn new tasks and perform them well. In addition, although palliative care was a key component of the early response to AIDS, its contribution has been “marginalized” in the era of ART. Thus, while palliative training and units still exist (but are not in active/prominent use in AIDS care), palliative care is not recognized as a key component of QI activities. Formative Evaluation of Quality of Care Initiative by Ministry of Health |23 Once QoC has been introduced, a mechanism is needed for continuously ensuring that the necessary capacity is available to carry out the QI activities. At the facility level, Peer Coaches were to be trained at graduated sites and tools shared with them to support peer coaching between sites. HCI would support the teams in developing systems for ensuring continuity of QI activities in their sites. It was apparent from the findings that not much thought had been put into capacity-building for institutionalization and sustainability. Most of what was done involved introducing, demonstrating and piloting the QI activities. The implementation of the QoC programme was affected by a number of factors, including that those trained often did not continue to serve in the facility, district or region where they were seconded for training due to transfers and creation of new districts. In addition, funding for on-going QI activities in the facility/district budgets is grossly inadequate. The institutionalization of QI will be affected if MoH’s responsibility to plan and build its capacity for implementing HSSP-III and NHP-II is not in place. In the case of HIV/AIDS, even making CD4 machines or viral load testing equipment available at regional referral hospitals is proving to be a challenge. When QI is extended to other health services, the demands for such machines will certainly increase, further aggravating the current situation. In the facilities, volunteers such as peer coaches, network support agents, peer educators, and elite clients were used in various ways to support the health workers with providing ART services in the context of QI. This resulted from an emergency push by the government to scale up ART access to those in need. It is unlikely that these volunteers will remain active; once left to the government, it will not be possible to provide them the token allowances they are usually given by development partners that collaborate with health facilities. To ensure that key stakeholders including policymakers and service providers are aware of QoC efforts, QI activities, results, outcomes, challenges and lessons learned need to be shared from time to time to facilitate strategic decision making and guide planning and implementation at different levels. Plans were made at the national level to achieve this by extracting and documenting tested improvement changes and technical outcomes, and by preparing and delivering presentations to conferences at the national, regional and international levels. In addition, a quarterly QoC newsletter was planned, as well regular feedback to key partners regarding pertinent issues in QI. For example, JMS/NMS would receive information on the logistics problems occurring at the facility level, and a registry of untrained staff in facilities would be provided to the MoH for enhancing their link with relevant trainings. At the national level, one publication on QI was issued in April 2007 and a global conference was held in Kampala in August 2010 that some regional and district QI teams attended. In addition, minutes were taken of the QI team meetings at all of the facilities visited. However, of all the facilities visited in Lira and Arua regions, only those in Orum HC-IV contained excellent information at the facilities that was displayed and seen by the team. Some of that information was also included in the district QI team files, but did not appear to have been publicized, used to encourage staff at Orum, or used as a good practice experience to motivate others. There was also a lack of experiential learning and information-sharing among QI participants in a given region; in some instances, information-sharing was also lacking at the district level. Formative Evaluation of Quality of Care Initiative by Ministry of Health |24 Conclusion Although some QoC implementation plans reviewed indicated that the QI process supported by the two projects since 2005 was considered to involve the institutionalization of quality, our evaluation found that such institutionalization was far from being achieved. What has clearly evolved is a commitment by MoH to extend QI beyond AIDS services to the entire health system. The inclusion of specific elements of QI in the HSSP III (still under development) may be a good pointer toward concerted institutionalization of QI over the next five years. A dominant theme in MoH strategies and policies reviewed in this evaluation is a commitment to develop and provide high-quality services. This may imply a commitment to include quality assurance and improvement considerations in the design, implementation and monitoring of such services. If such a commitment were to be realized, there would be little need for “add-on” quality improvement efforts such as those currently planned in the QoC model although specialized collaboratives to deal with complex system-wide problems such as stockouts will always be needed. However, it appears that the commitment in quality in policy and strategy documents have not yet been translated into concrete quality actions in service implementation and measurement. 3.2.4 Evidence of the sustainability of the QoC initiative Sustainability is viewed as the adoption of new ways of implementing quality improvement processes so that improved outcomes and the QI process become the norm. It also involves maintaining gains made and evolving, as required. The initiatives examined had an overall plan to institutionalize QI but this was hampered by inherent weaknesses at the time within the Quality Assurance Department of MoH. At the facility level, the original plan of “graduating” facilities when QI processes had been implemented for a year was determined to be erroneous. Furthermore, nothing has been compiled to guide what the “improvement keepers” need to do to successfully sustain the benefits of the improvement process. As a result, a gap exists in maintaining the gains made, especially after the targets have been met. There is a danger of undermining the gains made since the issues addressed in the current application of QI do not look at systems. In part, this is due to a focus on the clinic rather than the administrative leadership, and to the commitment of the facility or district in the case of district QITs. Consequently, there was very little administrative commitment with regard to the program. In fact, some districts either wholly or partially assigned the coordination of the QoCs to middle-level managers. Most facilities felt that the process could continue only when it was supported further, especially when the “low hanging fruits” at ART clinics had been harvested. There were no established or planned mechanisms to keep the process going or to facilitate “spill over” into management or other departments. The QI program was initiated in facilities to improve the quality of the rapidly scaled-up ART services. Along the way, the initiative worked to align the interventions within the health system Formative Evaluation of Quality of Care Initiative by Ministry of Health |25 hierarchy, but not within the overall health facility system where the ART clinics operate. This was observed within facilities that operate as outreach sites for ART programs. For example, in Kiswa HC III, each of the three outreach programs holds different QI meetings without any congruence. TASO outreach efforts in the catchment area of Busesa HC IV have no linkages to QI meetings. Therefore, although improvements in quality indicators and facility performance have been realized, these are quick successes that will not by themselves spread to other facilities. Facility managers and workers do not appear to receive any reward for improving quality, and fatigue has started to set in at some facilities that no longer hold regular QI meetings because of “other pressing commitments”. Proper problem analysis is the key to understanding the root and systemic causes of many of the problems but this has not been addressed adequately. In one facility visited, file cabinets were shifted from the maternity/antennal unit to the HIV clinic without due consideration of the negative impact this could have on the facility’s functions overall. This indicates that the impact on other supporting areas are not part of the QI plan within the facilities. The mission and vision of the processes have been applied to ART services rather than the facility system as a whole. All facilities visited had at least one staff member who was trained and still available, but all of the originally trained staff were available at only four facilities. Seven of the facilities could potentially sustain QI services because most trained staff were still available in the clinic and/or they had spread QI to other departments therefore enabling broad-based, long￾term planning for QI at the facility. Our discussions with facilities that appeared to have abandoned the system revealed that some project activities were viewed as a one-time effort to address specific issues until the facility “graduates”. The teams in Eastern region believed that either the next project (Star EC) would continue funding and supporting the process or it would terminate. QoC is seen as an isolated project with a start and an end date when funding and other resources come to an end. QI requires that someone with the right personality and willingness to assume a leadership role is selected to head the effort. In most facilities, the leadership is delegated to senior nursing staff. Many facilities have selected three to four manageable indicators that are easy to track. Some facilities have however been successful in tackling more than eight, at least in the short run. Communication has been a key to the QI approach. Most facilities viewed communication as displaying charts and graphs. Effective communication would enable each person to understand his or her role within the initiative and identify areas of resistance. Communication through feed forward/feedback is also crucial. Feed forward would require that districts are in a position to assume management roles in solving those problems within their sphere of control or influence. In one district visited, the problem of stock outs is left to districts to handle. However, the process of addressing this problem, manifested in a number Formative Evaluation of Quality of Care Initiative by Ministry of Health |26 of facilities, has not been subjected to a QI approach. The district QITs have not adequately understood how to apply QI in solving tough management problems. One site team lost leadership, interest and focus despite the fact that facility staff were aware of the benefits of a QI approach. They did not make a decision to stop QI implementation but it happened because the remaining managers did not know how to continue the QI process that could sustain some of the improvements made. Some facilities have a very large staff workload. One facility visited took the view that it had “no time” to hold QI meetings. We also observed that QI meetings were held outside the routine management meetings and organized according to funding projects. In most cases, many meetings and activities, especially analysis of data, occur in “overtime” at the end of a clinic day. Furthermore, more time is spent on reporting pressure than improvement. This can have a significantly negative impact in the long run since staff are sacrificing their personal time in some of the facilities visited. “With QI, the patient load has decreased but we have to spend more time in holding QI meetings.... “ “…we now rarely get many people attending the meetings because members are very busy” HC IV health worker Resource constraints were cited as a great challenge to the sustainability of a nationwide scale-up of the QoC initiative. Sustainability is an afterthought and is not built into the projects. The infrastructures needed for on-going sustainability have not been sufficiently prioritized. At the national level, the QoC initiative for HIV/AIDS has received significant attention only recently since the initial priority was to extend ART service coverage. The QI programs focused on high impact/immediate change in nationally designed indicators rather than on program sustainability in instituting PDSA cycles. This has created an impression of quick uptake at facilities but has involved limited institutionalization of support functions. A clear study did not exist to identify barriers to sustainability during the planning, start-up, and continuation phases. The implementation supervision coaching and mentoring support system that was adopted through the District QI Teams was started later and was made to fit within the projects. Indeed, some projects have not yet adopted this system. Although having a QI Team does not guarantee success, the approach of training a few at a time gives the staff an opportunity to implement what they have learned. However, training only those on the ART team elicits criticism by those who were not involved, resulting in a reluctance to implement any recommendations. Thus, despite immediate short-term needs, an approach emphasizing incremental QI achievements will be more effective in yielding sustainable improvements in health care quality at the facility, district and national level Formative Evaluation of Quality of Care Initiative by Ministry of Health |27 3.2.5 Key lessons identified from the design and implementation Our consideration of lessons learned with respect to capacity focused on MoH’s demonstrated ability to facilitate knowledge gained from QoC within the entire health sector; and to generate, package and apply lessons from the QoC process. In addition, we considered the presence of concrete lessons learned. These could include: a) experiential knowledge as mentioned by actors at different levels, but not yet captured and packaged in a “separate” product; b) actual documented lessons as evidenced in documents, videos, voice recordings, etc.; and c) fresh lessons arising out of analysis and documentation within this formative evaluation. The design of QoC provided an opportunity to generate and share lessons gained from the QI process through the learning sessions, collaborative team meetings at different levels, and the annual meeting/conference for regional teams and other core team members. The QI tools discovered at the site level included those entirely devoted to documenting/synthesizing lessons to be shared at various fora (e.g., Outcome Collaborative, synthesis guide of emerging best practices for sites, which was used at Nebbi and Kuluva) and those that contained elements focused on more general observations and notes on other identified effects in the documentation journal, and the Standard Format for sharing quality improvement intervention. “Both the most and least effective changes implemented are difficult to maintain/sustain because they are dependent or controlled by external factors. For example, constant adequate ARV stock was our most effective change implemented, but we cannot keep physically chasing after NMS to deliver, due to lack of resources (vehicle, driver, fuel).” District Hospital, Arua Region Although respondent interviews and documented evidence demonstrated many instances of lessons learned and shared at the site level and in learning sessions, no evidence was found of systematic collation of these lessons into a sharable document (within and outside the country), nor of follow-up to spread the lessons learned and their application. “Use of expert clients in ART clinic help to enforce good adherence to ARV and Septrin, especially as they talk with experience and give testimony in their lives, challenges faced and how they overcome them, in the process of taking ARVs.” PNFP Hospital, Arua region “Adherence meeting together with the PHAs helps to fight stigma among PHAs and restores hope as PHAs get to know their friends who are doing well and have clinically improved for over five years.” PNFP Hospital, Arua region HCI staff engaged in some research activities aimed at informing the needs of the QI teams. For example, a triage survey was conducted in FY08 that informed the effort to improve client flow as an integral element in QI efforts found at most facilities visited. Formative Evaluation of Quality of Care Initiative by Ministry of Health |28 The Uganda Quality of Care Program 2010 - 2014 Strategic Plan of July 15, 2009 was largely based on lessons learned from implementation of the QoC initiative on ART services. This is a good example of the application of lessons learned. Effective QI requires that a minimal caregiving process be in place, characterized in this study as ‘the cup being full’. The experience at most sites visited was that the cup was only half full, or even lower. The effects of limited resources were particularly felt in terms of maintaining initial gains. The need for a central and overarching framework for quality improvement is critical, especially in instances where multiple partners may be actively supporting service development and quality improvement, often by using a wide range of approaches and detailed tools. Data analysis and the use of local data in decision making is possible and makes a lot of difference at the facility level with great improvement in service provision. This warrants further discussions by policy makers, managers and providers. Another lesson learned is that in quality improvement it is important to connect and work with other teams both inside and outside a facility. The opportunity to work with other facilities in the learning sessions was very useful in promoting an exchange of lessons learned; but this was not sustained after graduation, and is not evident even in the new collaborative. Although found to be tedious by some staff, documentation at the facility level in a journal and the use of other tools constituted a critical QI practice. Declining commitment and sponsorship of top management, as occurred at Iganga Hospital, can be a devastating cause of poor performance. The pace of QI improvements depends on the capacity of the constrained resource. Failure to deal clearly with the interactions between structural constraints such as understaffing, lack of space and furniture and gross underfunding increases the risk of failure after the short-term. It was easier for NGO facilities to tackle many indicators at a time and for government facilities to address a few. The process of accreditation of facilities for ARTs also seemed skewed against FBO facilities. Only two districts had ever used QI to address accreditation There was little evidence that facilities were prioritizing and sequentially implementing improvement cycles. Many of the deeper problems in districts and facilities would require such an approach in order to produce more conspicuous successes. As a result, only relatively simple internal activities have changed instead of producing real solutions to the most frustrating systemic problems. We also noticed that improvement in data integrity and analysis at the ART clinics was key to determining gaps that the QI team needed to address. The introduction and use of revised ART patient records made it easier for the teams to identify problems. Though it would take a longer time, QoC efforts should first start with a data quality initiative since no Formative Evaluation of Quality of Care Initiative by Ministry of Health |29 improvement is possible without accurate measures. Such efforts are designed to establish and maintain consistent data definition, analysis and presentation rules so that facilities or departments can achieve a single version of the real data and save time searching for real problems. 3.3 Objective 3: QOC Support of Accreditation in Facilities Providing HIV Services 3.3.1 Accreditation support There was no deliberate support of accreditation of health facilities planned by the QoC initiative within the MoH. The accreditation of facilities was already in place and guided by the Uganda National ART policy when QoC was first developed. The accreditation is ideally supposed to be done by a team of experts selected by the MoH through the AIDS Control Programme (ACP) once the facility applies to the manager of the ACP. The requirements for accreditation include availability of trained health staff, availability of HCT services, availability of space, a functional laboratory ARV drug procurement and storage, functional HMIS and a follow-up and referral system/network with other providers. This process is supposed to be initiated, implemented, and supervised through the district health system. Accreditation was conducted in 2003 and has not been repeated. Moreover, no mechanism exists to regularly review the accredited facilities to ascertain that they are maintaining acceptable quality or standards. Although no clear documentation to prepare for accreditation is in place, the health care workers in the districts noted that the quality improvement activities helped some of the sites that had not been accredited to achieve accreditation after the facilities had improved the quality of services they provide. In Nebbi district for example, Goli, Zeu and Wadlai HCIII’s were doing quality improvement as part of the pre-accreditation process initiated by the district. They also noted that in a number of districts some health facilities have not been accredited “officially” yet but are offering ART services. The Private Not for Profit facilities under the Uganda Catholic Medical Bureau conduct annually a strict accreditation exercise, and those who default pay penalties. However, this is not connected to the QoC initiative or the MoH. Also, no mechanism currently exists for the QoC to directly support the MoH in the accreditation of public, private or NGO health facilities. 3.3.2 Factors facilitating and hindering accreditation Factors facilitating accreditation a) Availability of the minimum number of health workers required, which includes at least a medical doctor, laboratory technician, nurses and counsellors b) A laboratory for HIV testing and ability to conduct CD4 counts or proximity to another health facility that could conduct CD4 counts for patients Formative Evaluation of Quality of Care Initiative by Ministry of Health |30 c) Availability of health workers trained in ART HIV/AIDS services d) Availability of sufficient space for a clinic to operate and for drug storage e) Commitment and interest by the District AIDS Task Force (DATF) and the District AIDS Committee (DAC) f) A proactive District Health Team (DHT) and District ART coordination Committee Hindering Factors a) The high turnover rates of various cadres of trained health workers b) The uncoordinated transfer of health workers trained in particular skills c) Inadequate staffing of health workers who can assume ART service provision tasks d) Irregular availability of ARV drugs e) Health workers who do not know how to attain accreditation Discussion The QoC initiative has not been effective in supporting the MoH in the accreditation of private, public or PNFP facilities. Ever since the first accreditation exercise in 2003, regular annual maintenance accreditation is not being conducted except by the Uganda Catholic Medical Bureau for their facilities. A number of facilities are providing ART without being accredited. Some districts have used the QI strategy to help prepare some of their health facilities for accreditation. The districts are not mandated to accord accreditation to health facilities to provide ART services, but they can use the QI process to enable the facilities to attain accreditation. Since QI can be an effective element of the accreditation process, a collaborative effort between the Quality Assurance Control Programme Department and the ACP with support from the QoC initiative would be a desirable next step. Formative Evaluation of Quality of Care Initiative by Ministry of Health |31 4.RECOMMENDATIONS 4.1 National-level recommendations 4.1.1 Organization and structure a) After strengthening the Quality Assurance Department (QAD) in the MoH, all aspects of the QoC initiative should be housed within the QAD in order to facilitate planning and budgeting. b) The QoC needs to be integrated into the NHP II, HSSP III, M&E indicators and health system structures. c) Regular reporting of results should occur, and be aggregated across all areas. A small number of quality indicators to evaluate performance should be developed for national use while a somewhat larger number should be reported at the district level. Data on quality improvement efforts should be maintained at the facility for its use in seeking improvement. d) The position of focal person in the Quality Assurance Department should be strengthened with links to other departments. e) A technical working group composed of QI focal persons from other departments and based in the Quality Assurance Department should assume the roles of the CTT. 4.1.2 Policy f) Policy and guidelines should be developed to streamline the integration of quality improvement into other sectoral plans. g) A strategy should be designed to integrate the QoC initiative into other service areas at the health facility level. h) A strategy should be designed to ensure that district teams can use QI to solve their logistical and human resources problems. 4.1.3 Operations/practice i) A process should be developed for reporting the outcome of QI at all levels of the system, with aggregated national information made available to top leadership in the MoH and other stakeholders. j) QI should be considered a priority so that it can achieve the top management support it deserves. Rather than single departmental teams, an executive steering committee should oversee the deployment of the QoC, and ensure that facility level goals are set, resources are made available, and results are weighed against a well communicated facility QoC plan. k) A strategy should be designed to integrate the QoC initiative into other service areas at the health facility level. One or two high-volume service areas should be chosen and collaboratives developed to start them on QI. In addition, the organizational culture and value change should flow from top management to front-line staff. This should be done in a way that is clearly visible and consistent with the leadership structure. Formative Evaluation of Quality of Care Initiative by Ministry of Health |32 l) The annual conference on quality of care needs to maintained and institutionalized to facilitate a continuous sharing of experiences. m) A capacity-building plan should be developed for all levels that includes structured training and support. Pre-service training in quality of health care should be developed along with training institutions. In-service training should not only single out national quality targets but develop quality improvement skills among managers and develop awareness about reaching long- term quality goals. n) A comprehensive framework for QoC needs to be developed and shared among all key stakeholders. 4.2 Sub-national level recommendations 4.2.1 Regional level o) The regional quality teams should fit within the existing regional support supervision framework. p) Policies and other enabling measures should be adopted that assist the regional hospital to conduct its mandated activities. 4.2.2 District level q) The QI initiative should be integrated into the existing HIV/AIDS coordination and reporting structures at the district level, e.g., District AIDS Committee and District AIDS Team. r) QI processes should be developed along with collaboratives at the district level that deal with those administrative issues that are hindering program success, such as stock outs, human resources policies and a lack of CD4 testing availability. 4.2.3 Health facility level s) Administration and key facility leadership need to understand and own the benefits of change. Major differences exist at the facility level and the approach needs to take into account the circumstances of each level. t) Consideration should be given to providing incentives to facilities that meet goals and hold their gains. For example, providing modest financial incentives to successful teams has been used with success in other settings. Including QI results in personnel appraisals and public identification of the “best” facilities should also be considered. u) Since facility leaders (especially nursing officers) are instrumental in quality improvement, QI should be included in basic job descriptions so it becomes an integral part of supervision. v) Greater attention should be paid to sustainable QI methods. Implementation should be congruent with the target set by the facility and should strengthen rather than increase the burden on existing systems and structures at the facility that are already overstretched. Formative Evaluation of Quality of Care Initiative by Ministry of Health |33 ANNEXES Annex 1: Places Visited Region/District Facility/site Category Masaka region Masaka district Masaka RRH Public HCI Kitovu mobile PNFP HCI Kyazanga HCIV Public HCI TASO Masaka PNFP HIVQUAL Masaka District QIT Masaka Regional QIT Jinja Region Iganga/Jinja Iganga GH Public HCI Busesa HCIV Public HCI Jinja RRH Public HIVQUAL Jinja District QIT Buwenge HCIV Public HCI Iganga District QIT Jinja Regional QIT Mbale Region Tororo Tororo GH Public HCI St. Anthony PNFP HCI Rubongi GH Military/Public HCI Mukuju HCIV Public IDI/ID-CAP Lira Region Lira District Lira District QIT Lira Regional QIT Lira RRH Public HIVQUAL Orum HCIV Public HCI Barr HCIII Public HCI Amach HCIV Public District Oyam District Aber GH PNFP HIVQUAL Arua Region Arua Arua RRH Public HCI Arua Regional QIT Kuluva GH PNFP HCI Nebbi Nyapea GH PNFP HCI Nebbi District QIT Nebbi GH Public HCI Goli HCIII PNFP HCI Pakwach HCIV Public IDI/ID-CAP Fort Portal Region Kabarole Fort Portal RRH Public HIVQUAL Fort Portal Regional QIT Bukuuku HCIV Public HCI Rwimi HCIII Public HIVQUAL Kyenjojo Kyenjojo HCIV Public HCI Kyenjojo District QIT Mulago Region Kampala District Kiswa HCIII Public HCI Orthodox GH PNFP HCI Wakiso District Entebbe GH Public HCI Formative Evaluation of Quality of Care Initiative by Ministry of Health |34 Annex 2: Scope of Work Introduction A formative evaluation is proposed for the Quality of Care (QoC) initiative implemented by the Ministry of Health and supported by USG partners, Health Care Improvement (HCI) project and HIVQUAL. USAID/Uganda and CDC/Uganda will fund the proposed evaluation. The purpose of this evaluation is to determine whether QoC is on track to achieve its program objectives and to assess the program’s strengths and challenges in order to make programmatic and management improvements in USG support to the initiative. Findings and recommendations from this formative evaluation are anticipated to assist CDC, USAID and the Uganda PEPFAR Country Team to enhance programming in quality HIV care and treatment service delivery for adults and children. Background Since 2004, the Government of Uganda has sought to rapidly scale up antiretroviral therapy (ART) and related HIV/AIDS services. Rapid expansion of ART has proved to be an enormous challenge for the national health system due to high patient load, staff attrition and turnover, frequent stock outs of ARVs and other logistics, lack of patient monitoring tools and poor reporting, lack of partner coordination and other barriers to implementation. To address these issues and create a sustainable structure to ensure the quality of HIV/AIDS care, the Ministry of Health (MoH), in 2005, launched a Quality of Care (QoC) initiative in HIV/AIDS care. USAID/Uganda and CDC, through HCI project and HIVQUAL respectively, are supporting the QoC initiative to improve quality of comprehensive HIV/AIDS care and treatment service delivery for adults and children at national, regional, district and sub-district facilities. The overall objective of the QoC initiative is to develop the capacity of the Ministry of Health to improve and maintain the quality of the national comprehensive HIV/AIDS program as well as to support accreditation of public, private and NGO facilities providing HIV/AIDS services. The two main partners, HCI and HIVQUAL, use two different but complementary Quality improvement (QI) approaches. Both projects aim to support MoH to build a quality improvement structure that is integrated into all levels of the health care delivery system. HCI is a USAID field-support mechanism which aims to provide technical assistance in quality of health care with a focus on adapting modern quality improvement (QI) approaches to the needs of Uganda. HCI builds upon the successes of USAID’s Quality Assurance Projects (I, II and III) to adapt approaches such as Improvement Collaboratives, Continuous Quality Improvement, and applying QI methods to human resource issues. HCI’s QI methods address the processes of delivering health care using evidence-based guidelines as well as issues related to efficiency, coverage and accessibility. HCI’s goals are to improve the quality of HIV care provided to clients and to build a sustainable, government owned system for improving care at all levels of the health system. On the other hand, HIVQUAL is a capacity-building program for HIV-specific quality management supported through a partnership of the Uganda Ministry of Health, the U.S. Centers for Disease Control and Prevention (CDC), Global AIDS Program-Uganda, United Nations Children’s Fund (UNICEF), World Health Organization (WHO) and the HIVQUAL International consulting team. The Program facilitates the development of sustainable quality improvement activities through building capacity and capability for quality management. The overarching goal of the Program is to improve the quality of care provided to people living with HIV/AIDS in Uganda. The Program balances quality improvement and performance measurement while building a solid foundation of programmatic infrastructure. This approach emphasizes the development of systems and processes involving clinic staff and consumers with active support from agency leaders. These structural features are designed to be sustainable even when staff turnover is high or organizational affiliations change. The Program embraces the basic precepts of quality improvement: • Measurement: Collect and use data to improve care • Focus on the customer: Implement activities aimed at improving patient health • Emphasis on systems of care: Improve processes that link to desired outcomes • Involvement of participants: Encourage direct participation in teams by relevant staff. The program uses performance data to identify QI needs and improvement priorities/strategies while monitoring progress over time. Formative Evaluation of Quality of Care Initiative by Ministry of Health |35 Currently QoC falls under two departments; National Disease Control (HIVQUAL) and Clinical services (HCI) but both closely work with Aids Control Program (ACP). At National level QoC has built the capacity of the Ministry of Health to implement Quality Improvement activities at national, regional and site levels through training of technical officers mainly from ACP, Clinical Services Department and Quality Assurance Department. At the provider level, QoC initiative assumes a supervisory and technical supportive role, building capacity and capability for quality improvement among providers through training and integrated support supervision visits, targeted coaching and mentoring as well as development and implementation of national performance measures. The QoC is supported by a Steering/Advisory Committee and a core technical team. The role of the core technical team, comprised of MoH technical staff and partners, is to provide expertise in all related QI areas, develop standardized materials as well as provide training, coaching and mentoring to ministry departments, regional QI teams, districts and facilities. The Steering/Advisory committee, with representative membership from Directorates, Departments and major partners in QI, provides general oversight, policy guidance and prioritization of quality improvement activities. Purpose of evaluation: The purpose of this evaluation is to determine whether QoC is on track to achieve its program objectives and to assess the program’s strengths and challenges in order to make programmatic and management improvements in MoH partner support to the initiative. The evaluation will make recommendations for strengthening institutionalization of QI in HIV/AIDS care and treatment services as well as extract lessons learned and best practices for the benefit of other MoH programs and/or for future programming. This formative evaluation is anticipated to assist MoH and its partners to enhance programming in quality HIV care and treatment service delivery for adults and children. Key evaluation questions: Objective 1: To improve the quality of comprehensive HIV/AIDS care and treatment service delivery for adults and children at national, regional, district and sub-district facilities. 1. To what extent is the QoC initiative effective in improving quality of HIV/AIDS services at the national, regional, district, sub-district and facility level? 2. What have been key factors facilitating or hindering improvements in HIV/AIDS services? 3. To what extent is the QoC initiative effective in collaborating and coordinating with other partners implementing QI activities at the national, regional, district, sub-district and facility level? Objective 2: To develop the capacity of the Ministry of Health to improve and maintain the quality of the national comprehensive HIV/AIDS program. 1. How effective have partners been in building capacity in the MoH to improve and maintain the quality of HIV care and treatment services at national, regional and district levels? 2. To what extent has the QoC initiative developed the capacity of other relevant departments in MoH to integrate QI approaches? 3. What have been key factors facilitating or hindering the institutionalization of quality improvement activities in the different levels of the MoH? 4. What evidence is there that the project will be sustained in the long-term? 5. What are the key lessons learned from the design and implementation of the QoC initiative? Objective 3: To support accreditation of public, private and PNFP facilities providing HIV/AIDS services. 1. How effective has QoC initiative been in supporting MoH in the accreditation of facilities for ART? 2. What have been key factors facilitating or hindering the support for accreditation of public, private and NGO facilities providing HIV/AIDS services? Methodology: USAID/Uganda and CDC/Uganda will fund the proposed evaluation. The evaluation team will be required to propose a clear methodology to answer all the evaluation questions. The methodology should encompass a sampling approach of supported health facilities to review in depth a reasonable number of concrete QI interventions within facilities, their sustainability and performance. Team Member Composition and Skills: It is essential that all team members understand the context of HIV/AIDS in Uganda. The team should number no more than eight persons who, collectively, possess the skills and experience below: 1. HIV/AIDS programming in Africa, with an added advantage of significant exposure to PEPFAR. Formative Evaluation of Quality of Care Initiative by Ministry of Health |36 Formative Evaluation of Quality of Care Initiative by Ministry of Health |37 2. Adequate knowledge and experience in Continuous Quality Improvement approaches. An understanding of the improvement collaborative methodology would be an added advantage. 3. Capacity building for public health service delivery. 4. Monitoring and evaluation. 5. Knowledge of Uganda’s decentralized health system. Deliverables: 1. Inception report to be reviewed by QoC core technical team, USAID, CDC and partners: a. Methodology including: i. People to be interviewed and interview plan ii. Districts and facilities to be visited and data collection plan b. A detailed work plan including timelines c. Data collection tools and plans for pre-testing and revisions Note: together with the evaluation team, key concepts and indicators to measure the evaluation questions will be defined. 2. First Draft Evaluation report submitted for review by QoC core technical team, USAID, CDC, WHO, UNICEF and selected key stakeholders: The content should cover all the main elements of the report including major findings, conclusions, lessons learned, and relevant annexes. 3. Oral debriefing to QoC Steering Committee, USAID, CDC, WHO, UNICEF and selected key stakeholders to discuss key findings, conclusions and recommendations prior to submission of final draft report. 4. Final Draft evaluation report submitted in the agreed-upon format and incorporating comments from QoC Steering Committee and core technical team, USAID, CDC, WHO, UNICEF and selected key stakeholders. Draft report copies will be submitted to QoC, USAID, CDC, WHO and UNICEF prior to the departure of the evaluation team leader. 5. Final evaluation report incorporating feedback from QoC core technical team, USAID, CDC, WHO, UNICEF and selected key stakeholders. The final report should be submitted by the team leader within one week of receiving comments from the QoC and should not exceed 30 pages, excluding the executive summary and annexes. Timelines: The evaluation is planned for April 2010 and will require approximately 30 working days of effort: • Review of documents, preliminary interviews and development of evaluation plan 5 days • Tool pretesting and revision 2 days • Field work • • 10 days • Data analysis, oral debriefs and drafting report 10 days • Revision of final report 3 days Annex 3: Facility-Level Findings Masaka RRH Kyazanga HCIV Masaka TASO Iganga GH HCIV Busesa Jinja RRH Buwenge HCIV Tororo GH St. Anthony Rubongi GH Mukuju HCIV Lira RRH Orum HCIV Barr HCIII Aber GH Arua RRH Kuluva GH Nyapea GH Nebbi GH Fort Portal RRH Bukuuku HCIV HCIII Rwimi Kyenjojo HCIV Orthodox GH Has staff trained in QI in the past 1 year N N N N Y N N Y Y N N N N Y Y Y Y Y Y N N Y N Y Availability of QoC guidelines Y Y N N Y Y Y Y Y N Y Y Y Y Y Y Y Y Y Y Y Y Y Y Documented improvement projects at facility N Y Y N Y Y Y Y Y N Y Y Y N Y Y Y Y Y Y Y N Y Y QoC job aids, reminders, etc, posted in service areas N Y N N N N N Y Y N Y N Y N Y Y Y Y Y Y Y N N N Presence of a QIC/Team at facility N Y N N Y Y Y Y Y N Y Y Y Y Y Y Y Y Y Y Y N Y Y Minutes of previous QIC meetings available N/ A Y N/ A N Y Y Y Y Y N Y Y Y NA N Y Y Y Y Y Y N N N Check for availability of workplans related to QoC at facility N Y N N Y Y Y Y Y N Y N N N N Y Y Y Y Y Y N N N Reports of improvement projects N Y Y N Y Y Y Y Y N Y N Y NA N Y Y Y Y Y Y N N Y Does this facility collaborate with other facilities in QI N Y N N Y Y Y Y Y N Y N N NA N Y, Y Y Y Y N Y N Y Are QI techniques used in any other service areas? N N N N N N N Y N N/ A N N Y NA N N Y Y N N N N N Y Which ones? N/ A N/ A N/ A N/ A N/ A N/ A N/ A Male ward N/ A N/ A N/ A RH RH General Ward N/ A N/ A N/ A N/ A EPI, OPD ANC How many QoC trained providers are on the staff of the health facility 5 5 2 1 6 12 8 8 8 0 6 4 2 1 3 7 9 8 6 2 8 6 How many QoC trained providers are present on day of visit 2 4 1 1 6 3 8 6 8 0 2 3 2 1 Nil 6 6 4 6 2 3 4 Is there on-going training and coaching on QI N Y N N N N N N N N/ A N N Y N Y Y Y Y Y Y N Y Y Conducting of internal coaching for of QI activities within the facility N Y N N N N N N/A N/ A N/ A N/ A N Y N Y, Y Y Y Y Y Y N Y Y Facility visited in the last 3 months by external QI coaches N Y N N Y N Y Y Y N Y N Y Y N Y Y Y Y N Y N Y N Does this facility receive written feedback from the external QI coaches? N N N N N N N N N N N Y Y Y N N N Y Y N N N N N Does facility collect information related to QI activities at facility N Y Y N Y Y Y Y Y N/ A Y Y Y N Y Y Y Y Y Y Y N Y Y Formative Evaluation of Quality of Care Initiative by Ministry of Health |38 Formative Evaluation of Quality of Care Initiative by Ministry of Health |39 Reports/charts related to use of information N Y Y N/ A Y Y Y Y Y N/ A Y N Y Y Y Y Y Y Y Y N Y N Is this information transmitted to other levels N Y Y N Y Y Y Y Y N/ A Y Y Y Y Y, Y Y Y Y Y N Y Y Have staff at this facility participated at QI meeting outside this facility N Y N N Y Y Y Y Y N/ A Y N Y N N Y Y Y Y Y N N Y Y Have QI interventions spread to other HIV treatment sites N N N N/ A N N N N N N/ A N N N N N N N Y N N N N Y Have QI initiatives been adopted by other departments N Y N N/ A N N N Y N N/ A N N Y N N Y Y Y N N N N N Annex 4: Bibliography 1. AIDS Control Programme, Ministry of Health, Uganda, Performance Report 2007/08, October 2008. 2. Draft Uganda National Health Sector Strategic Plan III (HSSP III) 3. Health Care Improvement Project, Annual work-plans 2008. 4. Health Care Improvement Project, Annual work-plans 2009. 5. Health Care Improvement Project, Annual work-plans 2010. 6. Health Care Improvement Project, quarterly reports March 2008 to December 2009. 7. Health Sector, Ministry of Health Uganda HIV and AIDS strategic Plan 2007-2010. 8. HIVQUAL- Uganda (MoH), Aggregated data report 2006-2009. 9. HIVQUAL-Uganda, Evaluation Summary March 2010. 10. Quality Assurance Project, Ministry of Health Uganda, Quality Assurance for Health workers in Uganda, Manual of Quality Improvement Methods 1996. 11. Uganda AIDS Commission, Report on implementation of National HIV and AIDS Strategic Plan. 12. Uganda National Health Sector Strategic Plan II (HSSP II) Formative Evaluation of Quality of Care Initiative by Ministry of Health |40