DECEMBER 2009 This report was prepared by University Research Co., LLC, for review by the United States Agency for International Development (USAID) and was authored by Lynne Miller Franco, Lani Marquez, Kim Ethier, Zohra Balsara, and Whitney Isenhower. The USAID Health Care Improvement Project is made possible by the generous support of the American people through USAID. Results of Collaborative Improvement: Effects on Health Outcomes and Compliance with Evidence-based Standards in 27 Applications in 12 Countries COLLABORATIVE EVALUATION SERIES DECEMBER 2009 Lynne Miller Franco Lani Marquez Kim Ethier Zohra Balsara Whitney Isenhower Results of Collaborative Improvement: Effects on Health Outcomes and Compliance with Evidence-based Standards in 27 Applications in 12 Countries C O L L A B O R A T I V E E V A L U A T I O N S E R I E S DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acknowledgements This paper was written at the request and guidance of James R. Heiby, Medical Officer at the Global Bureau for Health at the United States Agency for International Development (USAID) for the purpose of summarizing current evidence of the effectiveness of collaborative improvement in improving health outcomes and compliance with evidenced-based standards. It summarizes the efforts of over 1,300 teams of health care providers in 12 countries. The authors would like to acknowledge the dedication of these teams to improving the quality of services they provide to their patients. The work described in this paper also reflects the efforts of coaches and supervisors in the Ministries of Health and by QAP and HCI country teams who have made it possible to achieve such results through their very dedicated and capable technical support: • Benin – Mandy Rose, PhillipeTousougbe, Marthe Agathe, in collaboration with the Ministry of Health staff in Aplahoue Dogbo Djakotome District • Benin – AguimaTankoano,AndreYebadokpo, Eleonore Rabalahasa, Gaston Kekin, Franz Simeon, and others, in collaboration with the Ministry of Health in Zou-Collines Department • Bolivia – Jorge Hermida, Jenny Romero, Luisa Mendizábal, and Rosemary Chavez, in collaboration with the Ministry of Health and John Snow International • Ecuador – Jorge Hermida, Patricio Ayabaca, Bernarda Salas, Maria Elena Robalino, Luis F.Viera, LuisVaca, Maria L. Freire, and others, in collaboration with the Ministry of Health at the central level and in the 12 provinces • Guatemala – Rodrigo Bustamante, Elena Hurtado, Leonel Gómez, Héctor Chaclán and Mélida Chaguaceda, and others, in collaboration with the Ministry of Health • Honduras – Norma Aly, Gladis Castellanos, Mariela Fuentes, Jose Ochoa and others, in collaboration with the Secretariat of Health • Nicaragua – Oscar Nuñez, Ivonne Gomez,Yudy Wong and others, in collaboration with the Ministry of Health • Niger – Maina Boucar, Zakari Saley, Sabou Djibrina and others, in collaboration with the Ministry of Public Health • Russia – M. Rashad Massoud,Anna Korotkova, Olga Chernobrovkina,Victor Melnikov, Kim Ethier, andVictor Boguslavsky, in collaboration with the Federal Public Health Institute in Moscow • Rwanda – Rachel Jean-Baptiste, Maina Boucar,Apolline Uwayitu, Sabou Djibrina, and others, in collaboration with the Ministry of Health • Tanzania – Raz Stevenson, Festus Kalokola, Stephen Kinoti, Davis Rumisha, Faridah Mgunda, Stephen Hobokela, Elizabeth Hizza, and others, in collaboration with the Ministry of Health • Uganda – Rachel Jean-Baptiste,Anthony Musisi Kyayise, Stephen Kinoti, Ibrahim Kirunda,Augustin Muhwezi, BensonTumwesigye,Aldo Burua, Edmund Pachuto and others, in collaboration with the Ministry of Health • Vietnam – LienThuThi Nguyen, Linh Nguyen, and Hien Le, in collaboration with the NationalTB Program and Ministry of Health in Thai Binh Province We also want to express our appreciation to M. Rashad Massoud and James Heiby for their thorough review of drafts of this paper and thank Barbara N.Turner andTisnaVeldhuijzen van Zanten for their contributions to discussions on the links between quality improvement and health systems strengthening. This analysis of evidence on the capacity of collaborative improvement to strengthen the performance of health systems was supported by the American people through the United States Agency for International Development (USAID) and its Bureau for Global Health, Office of Health, Infectious Diseases and Nutrition.The study was carried out by staff of University Research Co., LLC (URC), under the USAID Health Care Improvement (HCI) Project, which is managed by URC under the terms of Contract No. GHN-I-01-07-00003-00. URC’s subcontractors for the HCI Project include EnCompass LLC, Family Health International, Initiatives Inc., Johns Hopkins University Center for Communication Programs, and Management Systems International. Recommended citation: Franco LM, L Marquez, K Ethier, Z Balsara, and W Isenhower. 2009. Results of Collaborative Improvement: Effects on Health Outcomes and Compliance with Evidence-based Standards in 27 Applications in 12 Countries. Collaborative Evaluation Series. Published by the USAID Health Care Improvement Project. Bethesda, MD: University Research Co., LLC (URC). Results of Collaborative Improvement in 12 Countries i Table of contents List of figures.................................................................................................................................................................... ii List of boxes..................................................................................................................................................................... ii List of tables..................................................................................................................................................................... ii Abbreviations................................................................................................................................................................... ii Executive summary........................................................................................................................................................ iii 1. Introduction.................................................................................................................................................................1 The challenge: Delivering effective health care...........................................................................................................................1 How does modern quality improvement help us improve outcomes? .......................................................................1 Recent developments in implementing quality improvement at scale .........................................................................2 2. The improvement collaborative and its adaptation in USAID-assisted countries, 1998–2008.................3 3. Methods: Analyzing the evidence on effectiveness of the improvement collaborative...............................5 4. Findings: Evidence of effectiveness of collaborative improvement to achieve and maintain high levels of care................................................................................................................7 Were significant improvements achieved in the quality of care and outcomes?....................................................7 How quickly were improvements achieved? ............................................................................................................................10 Were gains in the quality of care maintained?.........................................................................................................................11 5. Discussion and conclusions....................................................................................................................................12 Level of improvement achieved.......................................................................................................................................................12 Speed of improvement.........................................................................................................................................................................13 Maintaining gains over time ................................................................................................................................................................14 Conclusions..................................................................................................................................................................................................14 6. Future directions and implications for health systems strengthening ..........................................................14 The next frontier in quality improvement: Achieving even higher levels of quality, more rapidly, at larger scale ..........................................................................................................................................14 Implications for the role of modern improvement in the health systems-strengthening agenda...............15 Appendix 1: List of 54 USAID-funded collaboratives: 1998–2008.....................................................................17 Appendix 2: List of 27 collaboratives for which data were available and analyzed: Topics and scale .........19 Appendix 3: Indicators included in the analysis of the 135 time series charts assessed in the 27 collaboratives...............................................................................................................................21 Appendix 4: Time series charts included in the analysis, by topic area (n=135).............................................25 ii Results of Collaborative Improvement in 12 Countries List of figures Figure 1: Inclusion and exclusion of collaboratives in analysis of the evidence base......................................................................6 Figure 2: Relative increase in percentage of patients receiving quality care, by topic area (MNCH and FP, HIV andTB, other) and by baseline value.................................................9 Figure 3: Examples of speed of improvement in Uganda and Rwanda..........................................................................................10 Figure 4: Speed of improvement: Mean time in months to achieve 80% (n=118 time series charts) compliance with standards for MNCH and FP and HIV andTB collaboratives.....................................................................11 Figure 5: Speed of improvement: Mean number of months to reach 80% and 90%, all time series charts (all topics).............11 List of boxes Box 1: An example of the application of modern quality improvement to operational problems in HIV care in Uganda....................................................................3 Box 2: A collaborative improves antenatal care in Ecuador...........4 Box 3: Indicators of success of collaborative improvement...........7 Box 4: Improvement in health outcomes due to improvement efforts in Tver Oblast, Russia.............................................7 Box 5: Achieving and maintaining results for essential obstetric care, Niger.............................................................................................8 List of tables Table 1: Collaboratives by year of initiation.............................................5 Table 2: Collaboratives by topic and region ............................................5 Table 3: Mean (range) of absolute increases in quality, by topic (MNCH and FP, HIV andTB, and other) and by baseline level .............................................................................................8 Table 4: Percentage (number) of time series charts reaching 80% and 90% levels...........................................................................9 Table 5: Overall success of collaboratives: Number (%) achieving 80% or 90% for their indicators..............10 Table 6: Mean (range) number of months a time series chart showed maintenance of 80% or higher.......................12 Abbreviations ACT Artemisinin Combination Therapy AH Arterial hypertension AIDS Acquired immunodeficiency syndrome AMTSL Active management of the third stage of labor ANC Antenatal care ART Antiretroviral therapy ARV Antiretroviral CQI Continuous quality improvement DOTS Directly observed treatment, short course ENC Essential newborn care EOC Essential obstetric care EONC Essential obstetric and newborn care FP Family planning HCI USAID Health Care Improvement Project HIV Human immunodeficiency virus HR Human resources IDU Intravenous drug user IHI Institute for Healthcare Improvement MCH Maternal and child health MDG Millennium Development Goal MNCH Maternal, newborn, and child health NRDS Neonatal respiratory distress syndrome PHI Pediatric hospital improvement PIH Pregnancy-induced hypertension PISAF Projet Intégré de Santé Familiale (Integrated Family Health Project, Benin) PLWHA Persons living with HIV/AIDS PMTCT Prevention of mother-to-child transmission of HIV QAP Quality Assurance Project QI Quality improvement SD Standard deviation STI Sexually transmitted infection TASC Technical Assistance Support Contract TB Tuberculosis URC University Research Co., LLC US United States USAID United States Agency for International Development VCT Voluntary counseling and testing for HIV WHO World Health Organization Results of Collaborative Improvement in 12 Countries iii Executive summary Introduction This paper summarizes evidence from 12 low- and middle-income countries on the effectiveness of a modern quality improvement (QI) approach, the improvement collaborative, in ensuring that proven interventions are implemented consistently and at scale, in order to improve health outcomes. The collaborative approach was designed by the Institute for Healthcare Improvement (IHI) in the United States (US) to produce rapid, significant improvements in a targeted area of health care. The paper was commissioned by the U.S. Agency for International Development (USAID) and analyzes the results achieved by over 1,300 teams of health care providers who participated in 27 collaboratives supported by USAID during 1998-2008. Data analyzed cover 81 measures of compliance with standards and outcomes for maternal, newborn and child health, HIV/AIDS care, family planning, and malaria and tuberculosis diagnosis and treatment. Modern QI uses team work, a focus on client needs, and an understanding of processes and systems of care to test changes to processes and systems to achieve better outcomes. Collaborative improvement organizes QI beyond the scale of an individual team of health care providers, mobilizing multiple teams to identify changes that lead to better outcomes and facilitating sharing of ideas and learning across these teams. With many sites involved, a collaborative can simultaneously test a variety of changes, identify the most effective ones, and efficiently spread them across all participating sites. Based on highly successful applications of collaborative improvement in the US, Europe, and Australia, USAID began investing in the early 2000s in the widespread application of the improvement collaborative approach in developing countries. This paper analyzes the results of those efforts to answer the question, Does collaborative improvement consistently achieve rapid improvements in health care in developing country settings and at scale? Methods Between 1998 and 2008, USAID funded 54 collaboratives in 14 low- and middle-income countries. These collaboratives generally varied both in duration, from 12–36 months, and in size, from smaller “demonstration” collaboratives that sought to come up with a set of effective changes through the work of a few demonstration sites, to larger “spread” collaboratives that sought to scale up changes and best practices that had already been tested in demonstration sites. The number of participating sites in these collaboratives ranged from 3 to 127 (excluding one collaborative with 442 sites). Data produced in collaboratives are self-reported measurements made by participating teams. In the USAID-funded collaboratives, teams were made up largely of Ministry of Health personnel. Validation studies conducted in Niger and Ecuador indicate that self-reported data in collaboratives did not vary significantly from measurements made by external auditors. For this study, we analyzed data from collaboratives that met two criteria: 1) availability of at least 12 months of consecutive data on quality indicators and 2) used indicators measured as a percentage of the patient population. Data from 27 of the 54 collaboratives met these criteria; those excluded were a combination of collaboratives with non-percentage indicators (33% of those excluded), older collaboratives for which no data or incomplete data were available (44%), and collaboratives with less than 12 months of data available (22%). Each of the 27 studied collaboratives monitored multiple indicators (range: 4–22). For purposes of this analysis, we focused exclusively on indicators of quality of care provided to patients and relevant outcomes. Across the 27 collaboratives, we analyzed an average 3.75 indicators per collaborative (range: 1–7). Because some indicators were measured by more than one group of teams (teams either in different collaboratives or in different cohorts or “waves” in the same collaborative), data for the 81 indicators generated 135 time series charts, the standard method for displaying data on compliance with standards over time and a fundamental tool in health care improvement. A list of the 81 indicators and the 135 time series charts analyzed are included in the appendices of this report. Findings Were significant improvements achieved in quality of care and outcomes? The evidence from the 27 improvement collaboratives demonstrated that large increases in compliance with health care standards and in some cases, in health outcomes, were achieved across multiple care areas (maternal and neonatal health, malaria, TB, and HIV/AIDS), regardless of the baseline level of quality. Of the 135 time series charts, 88% attained levels of at least 80% compliance, and 76% reached at least 90%, even though more than half had baseline levels at 50% or below. Across the collaboratives, time series charts showed average increases of 51.9 percentage points (SD = 28.0, range 0–100%), regardless of baseline level and topic area. The figure on p. iv shows the variation in average absolute improvement for various baseline levels. The average relative or percentage increase was 210% iv Results of Collaborative Improvement in 12 Countries (SD = 350%; range 0–2400%).Almost two-thirds of the collaboratives produced consistently high levels of quality across all indicators tracked (all at levels over 80%). How quickly were improvements achieved? The evidence from these 27 improvement collaboratives on speed of improvement supports the hypothesis that collaboratives can achieve rapid results. Across clinical focus areas, collaboratives with indicators starting at performance levels below 50% reached levels of 80% in an average of 13 months, while those starting at perfomance levels above 50% reached levels of 80% in six months. These results were generated from collaboratives averaging 34 sites each, indicating the capacity to achieve speed at appreciable scale. Transferring learning from demonstration sites to teams that joined the improvement collaborative Mean absolute improvement in compliance with standards over baseline (135 time series charts) 0 10 20 30 40 50 60 70 80 90 100 baseline 0-25% baseline 26-50% baseline 51-75% baseline >75% % compliance with standards Minimum baseline level Absolute improvement n=52 n=32 n=33 n=16 of care, are needed to ensure that all patients receive quality services. Conclusion The evidence from 27 applications of collaborative improvement in 12 USAID-assisted countries shows that this approach raised compliance with standards of care to over 80% in most cases (in 88% of the time series charts analyzed) and to over 90% levels for 76% of the charts analyzed, regardless of initial levels. Improvement collaboratives achieved these results within about a year and generally sustained these levels over periods of observation averaging 12 to 21 months. The strength of a health system is measured in its ability to deliver good health outcomes. By achieving significant, sustained improvements in compliance with standards and outcomes, collaborative improvement is a viable tool for health systems strengthening in developing countries. in subsequent expansion waves appears to have doubled the speed of improvements: Expansion teams generally achieved performance levels of 80% in about half the time the original sites had. Even larger gains in speed of improvement occurred when spread was done with deliberate synthesis of learning. Interestingly, the time required to raise average performance from 80% to 90% was often substantial, taking from 34% to 140% longer than the time needed to bring performance from baseline to 80%. Were gains in quality of care maintained? While achieving levels of 80% or 90% performance is laudable, also important is maintaining those gains in quality over time. The evidence on maintaining quality gains indicates that these collaboratives did sustain quality levels of at least 80% for an average of 13.4 consecutive months, representing 72% of the months for which data were available. In some cases, sustained levels continued for four years. It should be noted that few collaboratives achieved reliable processes (i.e., stable performance levels of 95% or higher): The health systems in most of the countries that hosted these collaboratives have underlying weaknesses—such as significant staff movement, lack of critical supplies at the national level, and other system-wide constraints—that thwart health facilities’ effor ts to achieve and maintain gains in quality. This suggests that other strategies, such as those drawing on reliability science to error-proof processes Results of Collaborative Improvement in 12 Countries 1 1. Introduction The challenge: Delivering effective health care Our biggest challenge in achieving the Millennium Development Goals is ensuring that people receive effective health care delivered according to internationally accepted standards. In many countries, technology and trained staff exist, but health system flaws prevent services from being delivered correctly. For example, every year more than 500,000 women die in pregnancy or childbirth, with postpartum hemorrhage causing over a third of these deaths. Research has demonstrated that the risk of postpartum hemorrhage can be reduced by over 50% with use of an approach known as “active management of the third stage of labor” (AMTSL), yet many women in low-resource settings worldwide do not receive this life-saving care. Even where providers have the knowledge and skills, many health systems are not capable of delivering this evidence-based intervention to every woman who should receive it. Different parts of the health system do not work together to ensure that what is known to be effective is delivered to every patient every time. Among efforts to strengthen health systems so they can achieve the Millennium Development Goals is the application of modern quality improvement. This paper summarizes the evidence from low- and middle-income countries on the effectiveness of quality improvement (QI) in ensuring proven interventions are implemented consistently and at scale, thus contributing to reaching the Millennium Development Goals. Based on extensive field results, we argue that QI is an essential component of any health system-strengthening strategy because it measurably improves health system processes and therefore health systems results. How does modern quality improvement help us improve outcomes? The United States Agency for International Development (USAID) has defined health systems strengthening as “a continuous process of implementing changes in policies and management arrangements within the health sector in order to ensure it delivers the right volume and distribution of services, through effective organizations and processes, to meet the people’s needs in a fair, responsive, effective and efficient manner.”3 A health system, like any system, has inputs (resources), processes (activities), and outcomes (results). The hallmark of a strong health system is its ability to provide quality health care, as measured by reductions in morbidity and mortality, efficiency of provision, and patient satisfaction. Health systems are complex systems, with many interacting inputs and processes. The World Health Organization (WHO) framework for understanding health systems includes six building blocks: 1) service delivery; 2) health workforce; 3) medical products, vaccines, and technologies; 4) health financing; 5) health information system; and 6) leadership and governance. Each building block is an important health system input that, in its current state of functioning, contributes positively or negatively to the outcomes of that system. The field of modern QI provides effective strategies, methods, and approaches to address weaknesses in these building blocks and, more importantly, the interaction between these blocks, with the aim of improving the quality of care and, ultimately, improved health outcomes. “Health care delivery comprises a complex set of activities. Donors provide training and other technical support, but health systems offer many opportunities for additional often dramatic improvement. More than 50 assessments conducted in recent years show that the quality of care provided in large-scale programs falls short of evidence-based standards, while studies of health service organizations show high levels of inefficiencies.” From http://www.usaid.gov/our_work/global_health/hs/ techareas/quality.html (accessed 7/17/09) 1 World Health Organization. 2007. Everybody’s business: Strengthening health systems to improve health outcomes:WHO’s framework for action. Geneva, Switzerland:WHO. 2 Committee on Quality of Care inAmerica, Institute of Medicine. 2001. Crossing the Quality Chasm:A New Health System for the 21st Century.Washington,DC:NationalAcademy Press. 3 http://www.usaid.gov/our_work/global_health/hs/ (accessed 7/17/09). “The reality is straightforward. The power of existing interventions is not matched by the power of health systems to deliver them to those in greatest need, in a comprehensive way, and at an adequate scale.” 1 – Margaret Chan, Director General of the World Health Organization “Between the health care we have and the care we can have lies not only a gap, but a chasm.”2 – Committee on Quality of Care in America, Institute of Medicine 2 Results of Collaborative Improvement in 12 Countries Evidence-based guidelines developed at national and international levels offer guidance on what needs to be done to provide cost￾effective care. Failure to adhere to evidence-based guidelines is associated with negative outcomes.4 However, even if a provider has the knowledge and skills to implement evidenced-based care, he or she may not easily integrate that knowledge into the services he or she provides for many reasons. Those reasons may include: the task is not in his or her job description, care is not organized in a way that enables implementation of the evidence-based practice, needed resources are not routinely available, supervisors or colleagues fail to provide support for the evidence-based practice, and counter-incentives (time, fees) inhibit him or her from exercising that knowledge. Modern quality improvement—often referred to as “continuous quality improvement” (CQI) because of its emphasis on organizing quality teams that continually seek improvement in care—focuses on redesigning processes to ensure that health care delivery is effective, efficient, equitable, and responsive to clients.5 Using QI methods, health care providers can identify quality gaps between desired practices (what the evidence indicates are best care practices to positively affect patient outcomes) and the actual practice at a health care facility. QI approaches mobilize front-line health workers and managers, give them a framework to solve problems, and build on their insights and efforts. 4 Walker JA, Ashley DEC, Hayes RJ. 1988. The quality of care is related to death rates: Hospital inpatient management of infants with acute gastroenteritis. American Journal of Public Health. 78:149-52; Grimshaw JM and Russell IT. 1993. Effect of clinical guidelines on medical practice:A systematic review of rigorous evidence. Lancet. 342: 1317-22. 5 For further discussion of modern quality improvement methods, see Massoud R, Askov K, Reinke J, Franco LM, Bornstein T, Knebel E, Macaulay C. 2001. A Modern Paradigm for Improving Healthcare Quality. QA Monograph Series 1(1). Published for USAID by the Quality Assurance Project. Bethesda, MD: Center for Human Services. 6 Langley GJ,Nolan KM,NolanTW,Norman CL, Provost LP. 1999. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: Josey-Bass. The Model for Improvement6 (shown at left) frames improvement efforts for health care providers working in teams by focusing on client needs and using data to implement and test changes to the care system. The model starts with three questions: 1) What are we trying to accomplish? 2) How will we know if that change is an improvement? 3) What changes can we make that will result in an improvement? Using the answers as their guide, teams test and implement changes in the care delivery system to improve results. Box 1 shows the work of one team in Uganda that used quality improvement approaches as suggested in the Model for Improvement. Such approaches test both changes in standards of care and in processes of service delivery. Through an iterative process, teams of those directly involved in the processes of care identify the most robust, efficient, and effective changes that lead to consistently high quality services. Teams then work to incorporate these changes in their everyday practice to become the routine or “default” way of providing care. QI approaches can be applied to any process or system, be it service delivery, logistics, human resource management, health financing, etc. The power of quality improvement is its focus on making iterative changes in the processes that make up health systems and on consistently measuring the results of these changes on quality of care and health outcomes. Making changes in how specific tasks are performed or how care is organized and tracking the effects (positive or negative) of these changes allows health care workers to determine what changes lead to improvements in compliance with standards and outcomes. When improvement activities at the point of service delivery are accompanied by efforts at the health system level to reinforce and sustain the changes, improvements can be integrated into health policy for long-term effects. Recent developments in implementing quality improvement at scale As a leader in the evolution of modern quality improvement in health care in developed countries, the United States-based Institute for Healthcare Improvement (IHI) developed, in 1995, a new variant called the improvement collaborative to increase Model for Improvement What are we trying to accomplish? How will we know that a change is an improvement? What changes can we make that will result in improvement? Act Plan Study Do Results of Collaborative Improvement in 12 Countries 3 7 Institute for Healthcare Improvement. 2003. The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement. Innovation Series. Cambridge, MA: Institute for Healthcare Improvement. 8 For further discussion of this approach to spread, seeWorld Health Organization and Institute for Healthcare Improvement. 2004. An Approach to Rapid Scale-up Using HIV/ AIDS Treatment and Care as an Example. Geneva, Switzerland: World Health Organization. Box 1: An example of the application of modern quality improvement to operational problems in HIV care in Uganda A quality improvement team of front-line health care providers at the Kiyunga Health Center in Uganda set out to increase the percentage of women seen in PMTCT/antenatal care clinics who tested positive for HIV and who subsequently enrolled in general HIV care. Using the Model for Improvement, the teams developed and tested several changes – using peer escorts, cross-checking daily in the patient registers, and ongoing counseling on disclosure – to encourage women to enroll in general HIV care.The example shows the poster the team presented of the process they used, the changes they implemented, and the results they achieved in the form of a time series chart, which plots successive measurements of an indicator over time. the pace and uptake of improvements in health care.7 IHI applied the improvement collaborative approach to multiple clinical areas in the United States, with significant results. The improvement collaborative was designed to overcome obstacles to consistent application of evidence-based processes by bringing together a number of teams (or health care facilities or ”sites”) to work on rapidly achieving significant improvements in processes, quality, and efficiency of a specific technical area. USAID has had a long-standing interest in modern quality improvement, investing since the early 1990s in the application of modern QI approaches. In addition to other projects that included quality as one of many components, USAID has continuously funded, since 1991, a central effort to adapt modern QI approaches for USAID-assisted countries: the Quality Assurance Project (QAP) I (1991–96), II (1996–2002), III (2002–2008), and the USAID Health Care Improvement Project (HCI) (2007–present). In 1998, QAP supported the first adaptation of the improvement collaborative approach in Russia, and achieved significant improvements in a few sites. This was followed in 2000 by a spread of these improvements through a series of “waves” of new sites taking up the tested changes and achieving improved outcomes on a large scale.8 By 2003, USAID began supporting the widespread application of the approach in order to achieve significant improvements at scale in USAID-assisted countries. Between 1998 and 2008, USAID funded applications of collaborative improvement in 14 low- and middle-income countries: QAP and the USAID Health Care Improvement Project initiated 48 collaboratives, and another six were supported through USAID-funded bilateral efforts, addressing topics as diverse as essential obstetric and newborn care, prevention of mother-to-child transmission of HIV (PMTCT), and human resource management. In 2009, HCI initiated 27 new collaboratives, including ones in two countries that had not previously had a USAID-supported collaborative. Through 2009, USAID thus had supported 81applications of the improvement collaborative approach in 16 countries—a substantial body of experience that has given us a solid basis for the feasibility of the approach and a rationale for evaluating the effectiveness of collaboratives in USAID-assisted countries. 2. The improvement collaborative and its adaptation in USAID-assisted countries, 1998–2008 An improvement collaborative, whether implemented in the United States or in USAID-assisted countries, engages multiple sites to share and learn from each other through a facilitated process of applying modern quality improvement methods towards achieving a common aim. As opposed to independent problem solving and improvement work carried 4 Results of Collaborative Improvement in 12 Countries out by individual sites, improvement collaboratives leverage the collective power of many sites working simultaneously on the same topic or problem. With many sites involved (each engaging its own providers and patients in the process), a collaborative can rapidly test a variety of changes, identify the most effective ones (“best practices”), and share them across participating sites. This approach also facilitates scale-up of best practices that have been tested for robustness in a range of environments. By facilitating the rapid spread of effective changes, the collaborative approach strengthens the health system on a broader scale. An improvement collaborative integrates modern QI principles and methods (team work, process analysis, testing and implementing changes, client satisfaction) with many elements of traditional public health programming that have long been the foundation of USAID assistance: standards, training, new equipment and technology, job aids, and supervision. While traditional programming has tended to focus on inputs, QI is With USAID technical suppor t, Ecuador’s Ministry of Health (MOH) initiated an improvement collaborative in 2003 to address essential obstetric care. The collaborative started with 8 sites and within the first year grew to 55, all working on improving antenatal and delivery care. Quality improvement teams were formed and trained to monitor indicators of quality and test changes to ensure that evidence-based practices were routinely implemented. In about a year, the percentage of prenatal visits conducted according to standards began to stabilize near 80%. This first wave of implementation was followed in 2005 by a second wave of sites that joined the collaborative and had the advantage of the learning that had been generated by the first wave. These second-wave sites received less support from external technical advisors and more from the MOH. In 2007, without external technical support, MOH staff initiated another wave of improvement activities in 10 new sites. As the lines show, using learning from the previous wave, each successive wave achieved rapid scale-up of best practices.By the end of 2007,all 86 hospitals in 12 provinces were reporting data and showing consistent results, with most women receiving care according to standards. In 2007, the MOH scaled up the AMTSL component to all remaining provinces, and as of this writing (2009), the Ministry is expanding the entire essential obstetric and neonatal package to five more provinces with its own funding. Box 2: A collaborative improves antenatal care in Ecuador 0 20 40 60 80 100 2003 2004 Percentage 2005 2006 2007 2008 2009 Ecuador: Percent of pregnant women receiving prenatal care according to norms Successive waves of 85 hospitals in 12 provinces joining collaborative improvement Wave 1: 55 facilities in 2003–2004 Wave 2: 21 facilities in 2005 Wave 3: 10 facilities in 2007 focused on making changes in processes of care, often to more effectively utilize those inputs for the achievement of outcomes. Collaboratives go one step further by organizing multiple teams to work together intensively for a short period of time to identify and share amongst each other effective changes for achieving a common objective or outcome. As a modern QI approach, collaborative improvement provides added value to strategies that have traditionally focused on providing the inputs and structures for quality. Box 2 provides an example of an improvement collaborative in Ecuador that made significant improvements in essential obstetric care on a large scale. The central innovation of the collaborative improvement approach is the organized shared learning across multiple sites at the service delivery level, supported by the larger health system, which promotes rapid uptake and spread of high impact interventions. As it has generally been adapted in the context of USAID efforts,9 collaborative improvement intercedes at both the macro and micro levels of the system, through a 9 USAID Health Care Improvement Project. 2008. The Improvement Collaborative: An Approach to Rapidly Improve Health Care and Scale Up Quality Services. Published by the USAID Health Care Improvement Project. Bethesda, MD: University Research Co., LLC (URC). Results of Collaborative Improvement in 12 Countries 5 Year Number Percent 1998–2002 9 18% 2003 9 17% 2004 7 13% 2005 2 4% 2006 5 9% 2007 11 20% 2008 11 20% TOTAL 54 100% Table 1: Collaboratives by year of initiation Topic Number Percent Essential obstetric and newborn care 17 31% Child health 10 19% Family planning 2 4% HIV/AIDS 11 20% Tuberculosis 4 7% Other (e.g., malaria, hypertension, community financing) 10 19% TOTAL 54 100% Region Number Percent Africa 19 35% Latin America 17 31% Eastern Europe/ Eurasia 17 31% Asia and Near East 1 2% TOTAL 54 100% Table 2: Collaboratives by topic and region product of this step is an enhanced implementation package that will guide the scale-up of the improved care practices and operational changes needed to support them to other parts of the health care system. During the final phase, Spread and Institutionalization, the macro level works to organize organizational mechanisms and processes for spreading quality improvement and better-care practices to additional service delivery sites and for building quality improvement activities into routine work at the service delivery and support levels. This paper focuses on summarizing the evidence on collaborative improvement as an approach to strengthening the quality of care provided by health systems. USAID supported implementation of 54 improvement collaboratives between 1998 and 2008 in 14 countries (Table 1), addressing various care areas in four world regions (Table 2). Appendix 1 provides more details on these collaboratives. The USAID experience with collaborative improvement extends for more than a 10-year period, during which time the approach itself has evolved. USAID-funded collaboratives varied in duration and size. These 54 collaboratives lasted from 6 months to three years or more. The number of sites participating in a demonstration (i.e., initial) phase ranged from three to 127. Some collaboratives expanded either by successive waves (without an explicit activity to synthesize and consolidate learning) while others expanded through a deliberate spread phase. Expansions in either case ranged from 10 to 442 sites. Some collaboratives focused on specific geographic regions within a country (as requested by USAID or the host government) or on specific types of facilities. 3. Methods: Analyzing the evidence on effectiveness of the improvement collaborative This section discusses the methods used in this paper to analyze the results of collaboratives as a mechanism to achieve significant improvements in the quality of care rapidly and consistently over time. While 54 USAID-funded improvement collaboratives were implemented through 2008, our analysis included only those collaboratives for which: • Adequate data were available:We required that at least 12 months of consecutive data on quality indicators be available for analysis so that trends could be ascertained. • Indicators were suitable for comparative analysis:We examined only indicators that are measured as a percentage of the patient population so that comparisons could be made. series of four major steps. During improvement collaborative Preparation, stakeholders at central (macro) and decentralized (micro) levels of the health system identify improvement aims (“What are we trying to accomplish?”) and indicators (“How will we know our change is an improvement?”); identify the evidence-based, better-care practices all providers should be implementing to achieve desired results (those changes known to yield improvement); and develop the structures, tools, capacity building, and support mechanisms to help sites in implementing the package of better practices. During Implementation (also known as the Demonstration Phase), collaborative activities focus on the micro level of the health system (the point of service delivery), with quality improvement teams at participating sites applying modern QI methods and sharing changes tested and results with other sites. The third step, Synthesis and Consolidation, centers on considering how what is learned during the collaborative can be summarized, simplified, and packaged into a set of the most effective changes (i.e., changes tested and shown to yield improvements) emerging from the work of its teams. The 6 Results of Collaborative Improvement in 12 Countries Figure 1 shows the results from applying these criteria, resulting in 27 collaboratives available for analysis.Appendix 2 has a list of the 27 collaboratives analyzed, including topics and scale of efforts. Within these 27 collaboratives were 36 groups of teams, as several collaboratives included waves of teams (as exemplified by Ecuador, in Box 2). These various groupings of teams were analyzed separately in several of the analyses presented below. The number of teams contributing data to this analysis totals 1338, with an average of 50 teams per collaborative.11 All the improvement collaboratives monitored multiple indicators (range 4 to 22), which included some varying mix of outcome, process and input indicators. For this analysis, we focused exclusively on indicators of quality of care provided to patients and relevant outcomes.Across the 27 improvement collaboratives, we analyzed an average 3.75 indicators per collaborative (range 1–7), for a total of 135 time series charts,12 representing 81 different indicators. If collaboratives included three waves (or groupings of teams), a single indicator would generate three times series charts, one for each group of maternal health indicators, 21% child health indicators,15 18% HIV care and treatment, 15% newborn care, 7% malaria, 6% family planning, 4% tuberculosis, and 3% primary care. Appendix 3 provides a list of all the indicators of quality of care and outcomes included in this analysis, as well as the countries and number of total teams reporting on each indicator. Appendix 4 presents all 135 time series charts analyzed, organized by care area and country, and provides the number of sites participating in that collaborative wave. Where a downward trend in the level of the indicator signals improvement (e.g., patients lost to follow-up), this is noted. Otherwise, charts showing a decline or plateauing of the indicator signal a failure of improvement activities, except for values approaching 100%, where little or no further improvement is possible. Indicator data in improvement collaboratives are generated from self-assessment of results by facility-based QI teams. While many pitfalls exist for the accuracy and validity of self-reported data, efforts were made to validate team-based data. Steps taken included review of data collected and records with facility QI teams during coaching visits by Ministry of Health and project staff; spots checks of a sample of records to validate 10 The nine collaboratives which used non-percentage indicators were mainly HIV/AIDS collaboratives in Russia whose indicators were primarily absolute number of patients, rather than percentage of patients. Of the 12 collaboratives for which adequate data were not available, several were early collaboratives with no possibility of obtaining data, spread collaboratives run by local Ministries of Health where we were not able to obtain data in a timely manner for the analysis, or data sets that did not allow analysis as needed. 11 The Russia spread collaborative for arterial hypertension was exceptionally large with 442 teams. Removing this collaborative from the calculation leaves a total of 896 teams, with an average of 34 teams per collaborative. 12 Time series charts show changes in indicator measurements over time and allow QI teams to see the effects of changes they are testing (both Box 1 and Box 2 have examples of time series charts). 13 Some indicators were measured in more than one collaborative. 14 For example, in most of the maternal and newborn collaboratives, process indicators were calculated as the percentage of women for whom all standards were adhered to. Other indictors focused on compliance with a single task, such as assessing HIV-positive patients forTB. 15 Note that 1 pediatric HIV and 4 pediatric malaria indicators were classified as child health indicators because they were part of a broader child health collaborative. Figure 1: Inclusion and exclusion of collaboratives in analysis of the evidence base10 2 newer collaboratives excluded because 12 months of data were not yet available 4 collaboratives excluded because they were terminated before 12 months of data were collected 12 collaboratives excluded because source data were not available at project headquarters 9 collaboratives excluded because indicators were expressed not as percentages but as numbers REMAINING FOR ANALYSIS: 27 COLLABORATIVES IN 12 COUNTRIES 54 COLLABORATIVES IN 14 COUNTRIES, 1998–2008 teams.13 Of the 135 time series charts, 82% focused on indicators of process quality and 18% on patient outcome indicators (health status or behavior), and about half of the process indicators covered the range of actions needed to achieve evidence-based care.14 Indicators of quality serve as a proxy for outcomes and generally focused on the percentage of patients for whom providers complied with evidence￾based standards, for example, the percentage of women with a vaginal delivery in the maternity who received the three key tasks of AMTSL, or percentage of children with fever attending the health center who were treated according to standards. Of the 135 time series charts, 26% measured Results of Collaborative Improvement in 12 Countries 7 In 1998, 5 sites participated in a collaborative to improve care for newborns with respiratory distress syndrome. By 2000, the changes developed by the demonstration phase of the collaborative had been spread to all 42 oblast hospitals. By 2003, these improved outcomes were stabilizing. Further monitoring of outcomes demonstrated even more significant declines in early neonatal mortality, neonatal mortality, and infant mortality from 2006 onwards. Box 4: Improvement in health outcomes due to improvement efforts in Tver Oblast, Russia 0 5 10 15 20 25 1998 2000 2002 2004 2006 2008 Deaths per 1,000 live births Early Neonatal Mortality Neonatal Mortality Infant Mortality 19.5 13.4 10.3 17.2 6.3 9.6 14.4 8.4 6.0 11.9 6.6 5.4 10.3 5.7 4.5 7.6 3.9 2.8 Box 3: Indicators of success of collaborative improvement • Mean absolute increase in quality from baseline to highest level (for 135 time series charts) • Mean relative increase in quality from baseline to highest level (for 135 time series charts) • Percentage of time series charts achieving 80% and 90% levels of quality (for 135 time series charts) • Percentage of collaboratives achieving 80% and 90% across all their time series charts (for 38 groupings of teams in 27 collaboratives) • Mean number of months to achieve 80% and 90% levels of quality (for the 118 and 102 time series charts, respectively) • Mean number of months time series charts showed indicators that consecutively stayed over 80% (for 118 time series charts reaching 80%) • Mean interval (range) with stabilized level of quality (for 87 time series charts reaching a stabilized level) repor ted data; and review of data received by project offices for inconsistencies and anomalies, with follow-up visits to sites with questionable data. Recent evidence from studies examining validity of self-assessment results indicate that these results do not vary substantially from results obtained through external audit.16 Collaborative results are most evocative when displayed as a time series chart. We developed a series of generic “indicators of success”that could be analyzed across 27 collaboratives, 38 groupings of teams, and their selected indicators. These “collaborative success” indicators, defined in Box 3, seek to facilitate understanding of the trends emerging from collaboratives’ effor ts to improve care and to characterize the collaboratives’ ability to achieve rapid, significant improvement that can be sustained (and is therefore worth spreading).17 In many cases, our analysis examined results pooled across all collaboratives. It focused on magnitude of quality performance (results) achieved, speed of improvement, and maintaining gains over time. While ideally the desired level of provider compliance with evidence-based guidelines is 100%, for the purposes of analyzing this complex data set, we selected 80% and 90% compliance with guidelines as common benchmarks to describe improvement over baseline values. 4. Findings: Evidence on the level of effectiveness of collaborative improvement to achieve and maintain high levels of care This section presents the results for 27 improvement collaboratives and their 135 selected time series charts with 16 Preliminary results from a comparison of self-monitoring results and external audit of identical clinical records from a set of 12 facilities in Ecuador (2007) indicate general high levels of percent agreement (and moderate to substantial agreement as determined by the Kappa statistic) in individual indicators. In Niger (2009), a comparison of results across self-monitoring and external audit data indicate only about a 5% difference in overall results, calculated across 20 sites for six comparison periods and three indicators. 17 Several indicators were designed to measure success through downward trends, rather than upward trends, such as loss to follow-up for HIV and postpartum hemorrhage rate. The values for these indicators were reversed to facilitate our analysis. 8 Results of Collaborative Improvement in 12 Countries Table 3: Mean (range) of absolute increases in quality, by topic (MNCH and FP, HIV and TB, other) and by baseline level Collaborative topic Starting at ≤ 25% Starting at 26–50% Starting at 51–75% Starting at > 75% TOTAL # time series charts MNCH and FP 80% (23–100%) 51% (23–72%) 33% (14–47%) 11% (6–24%) 92 HIV andTB 65% (24–95%) 49% (0–67%) 34% (9–48%) 8% (0–21%) 30 Other 69% (13–100%) 60% (56–65%) 30% (28–31%) 21% (21–21%) 13 ALL TOPICS 77% 52% 33% 10% 135 TOTAL # 52 35 32 16 135 Measured for all time series charts (across collaboratives) (Absolute increase = highest percentage minus initial percentage) Note: MNCH is maternal, newborn, and child health; FP is family planning;TB is tuberculosis. these baseline levels, either in absolute terms (as the change in percentage points from the baseline percentage and the highest percentage) or in relative terms as the percentage increase from baseline to highest level. Both the absolute and relative levels of improvement are dependent on the baseline level, and lower baseline levels leave room for greater improvement. For this analysis, each of the 135 time series charts was analyzed separately. Across the 27 improvement collaboratives, time series charts showed average increases of 51.9 percentage points (SD = 28.0, range 0–100%), regardless of starting level and topic area. Box 5: Achieving and maintaining results for essential obstetric care, Niger 0 0.5 1.0 1.5 2.0 2.5 2006 2007 2008 Postpartum hemorrhage rate (percent) 0 20 40 60 80 100 Percent births covered by AMTSL Births covered by AMTSL Post-partum hemorrhage Active management of the third stage of labor and postpartum hemorrhage rates in targeted facilities Niger, January2006–December 2008 Teams at 39 referral and peripheral maternities in Niger worked in a collaborative to improve delivery and immediate postpartum care. AMTSL was introduced and operational changes implemented to ensure that AMTSL was systematically performed for each woman delivering in the participating sites. These 39 teams achieved significant improvements in the quality of care (as measured by a change from basically 0 to over 98% of women delivering receiving AMTSL). This consistent application of the new standard (represented by the blue line) was associated with a concomitant significant drop in postpartum hemorrhage among these same women (green line). These results were consistently maintained for over two years with little variation. respect to collaboratives’ ability to rapidly achieve significant, sustainable results. The level of quality of care is measured by indicators of process (compliance with standards) or outcomes (changes in patient health status or behaviors) as a percentage of the patients seeking care. Were significant improvements achieved in the quality of care and outcomes? All the collaboratives started measuring their indicators before changes were implemented. Box 4 and Box 5 present specific examples of time series charts measuring improvements in health outcomes in collaborative areas. Box 4 presents significant decreases in three mortality indicators as a result of work on respiratory distress syndrome in one region of Russia. Box 5 shows significant declines in postpartum hemorrhage rates associated with increased compliance with AMTSL standards in seven regions in Niger.Across the 27 collaboratives, improvements were measured by changes from Results of Collaborative Improvement in 12 Countries 9 The average relative or percentage increase was 210% (SD = 350%; range 0–2400%).Table 3 presents the data for the numeric difference between baseline and highest levels achieved, disaggregated by baseline level and by general topic area, while Figure 2 presents the same for the relative increase. As expected, increases are larger for time series charts with lower baseline levels. Nevertheless, in both absolute and relative measures, increases in performance achieved by collaboratives for indicators starting at lower levels brought them in line with standard measures of success, achieving an average highest value of 90.2% (SD = 15.6, median 96%). A significant difference in baselines should also be noted: In the 27 improvement collaboratives, 39% of the 135 time series charts had starting values of 25% or lower, and two-thirds Table 4: Percentage (number) of time series charts reaching 80% and 90% levels Collaborative topic Reached 80% Reached 90% # of time series charts MNCH and FP 90% (83) 75% (69) 92 HIV andTB 83% (25) 77% (23) 30 Other 85% (11) 77% (10) 13 ALL TOPICS 88% (119) 76% (102) 135 Measured for all time series charts (across collaboratives) Figure 2: Relative increase in percentage of patients receiving quality care, by topic area (MNCH and FP, HIV and TB, other) and by baseline value Measured for all time series charts (across collaboratives) (Relative increase = [highest percentage minus baseline percentage]/baseline percentage) 0% 100% 200% 300% 400% 500% 600% Starting at ≤ 25% Starting at 26-50% Starting at 51-75% Starting at >75% Baseline values Percentage increase in the level of quality of care provided to patients MNCH and FP HIV and TB Other of them started with less than 50% of patients receiving care according to standards. Such low levels of compliance indicate little if any standardization of processes at the beginning of the collaboratives. Achievement of levels of 80% and subsequently 90% by the collaboratives shows significant improvements in compliance with guidelines, a central measure of the quality of care. Table 4 presents the percentage of time series charts that reached 80% or 90% during the life of the collaborative. As the improvement collaboratives generally had more than one quality indicator, we examined how well each was able to standardize the processes of care (that is, achieve levels of 80%) across their indicators (i.e., across the clinical area they were working on), as well as move toward higher levels of reliable care.Table 5 assesses the 38 groupings of teams in the 27 collaboratives in four ways, whether: • All result indicators being monitoring reached a level of 80%, • At least three-quarters of their indicators achieved 80%, • At least one indicator achieved 90%, and • All indicators achieved 90% or higher. The mean number of results indicators (those included in this analysis) per collaborative team grouping was 3.5, with a range of 1–6 indicators.18 18 For example, the Ecuador Essential Obstetric Care Collaborative (Box 2) measured five quality of care indicators: quality of prenatal care, use of the partograph, use of AMTSL, postpartum care, and newborn care. 10 Results of Collaborative Improvement in 12 Countries Table 5 indicates that almost two-thirds of the improvement collaboratives achieved high levels of quality consistently across the indicators they were measuring (all over 80%) and that most collaboratives achieving 80% in at least 75% of their indicators also Table 5: Overall success of collaboratives: Number (%) achieving 80% or 90% for their indicators Topic # collaboratives achieving ≥ 80% for: # collaboratives achieving ≥ 90% for: Total # waves of teams of all assessed collaboratives All indicators ≥ 75% of indicators At least 1 indicator All indicators MNCH and FP 17 (71%) 18 (75%) 22 (92%) 10 (42%) 24 HIV andTB 5 (56%) 6 (67%) 9 100%) 5 (56%) 9 Other 3 (60%) 3 (60%) 4 (80%) 2 (40%) 5 ALL TOPICS 25 (66%) 27 (71%) 35 (92%) 17 (45%) 38 Figure 3: Examples of speed of improvement in Uganda and Rwanda 0 20 40 60 80 100 2005 2006 Percent of HIV+ Patients Asessed for Active TB 2007 2008 Uganda: Screening for active TB in HIV-positive patients Wave 1 (57 sites): 17,000 HIV– patients/month Wave 2 (37 sites): 5,000 HIV+ patients/month Wave 3 (33 sites): 5,000 HIV+ patients/month 0 20 40 60 80 100 2003 2004 Percent of HIV+ Patients Asessed for Active TB 2005 2006 Rwanda: Prevention of mother-to-child transmission of HIV: Percentage of partners tested Wave 1 (16 sites) Wave 2 (20 sites) achieved it for all indicators. Such findings indicate improvement across a group of care activities, and not just a single indicator; this fact is important in the sense that most patients require a bundle of services, and quality must be present in all parts for the patient to experience better outcomes. However, moving to higher levels of performance (≥ 90%) was more difficult, with only 45% of collaboratives being able to achieve this level for all indicators. The table indicates that collaboratives are making significant improvements in the quality of care provided to patients through collaboratives, but work remains to ensure reliable processes for all patients. How quickly were improvements achieved? Working together across teams on a common aim facilitates rapid achievements in improvement in quality of care through sharing of effective changes and results. In addition, the learning generated about what changes lead to improvement should help teams that follow the initial teams, enabling the new teams to achieve improvements more quickly than the initial teams had done. Some of the collaborative groupings analyzed were waves of teams (as discussed in Box 2), where a first wave began working on specific aims, and subsequent waves joined the efforts and learned from what had been accomplished by the initial teams. In many cases, these subsequent teams benefited from informal sharing of results and effective changes, rather than from deliberate, systematic synthesis and dissemination of learning from earlier teams. Informal spread of best practices from initial teams can occur during coaching visits or interactions between sites of different waves through group discussions or presentations during learning sessions. Figure 3 provides additional examples of how successive waves of teams achieved improvements more quickly than did initial teams. Note that the rise in the subsequent wave lines is steeper than in the initial wave lines, indicating more rapid improvement. Figure 4 presents the results of our analyses of time series charts reaching 80% (across the 38 groupings or waves of teams within the 27 improvement collaboratives), showings the mean time (in months) from the beginning of the collaborative until indicators reached 80%. These results are disaggregated into three groups: a) first wave or demonstration sites, b) subsequent waves, and c) sites participating in a formal spread collaborative with deliberate synthesis of learning and dissemination of findings. (Improvement collaboratives addressing topics other than MNCH, FP, HIV orTB were dropped from this analysis because the number of time series charts was too small to allow such disaggregation.) For time series charts with baseline levels ≤ 50%, it took an average of 12–13 months for average performance to increase from baseline to 80%. For those starting above 50%, mean time Results of Collaborative Improvement in 12 Countries 11 was about six months. These results were generally comparable between MNCH and FP time series charts and HIV andTB time series charts. While subsequent waves of teams with informal spread do achieve results more quickly than the original teams, for HIV andTB collaboratives even larger differences occurred when spread was done with deliberate synthesis of learning, at least for Figure 5: Speed of improvement: Mean number of months to reach 80% and 90%, all time series charts (all topics) Indicators reaching 80% (n=118) Indicators reaching 90% (n=102) 0 2 4 6 8 10 12 14 16 18 20 Demo Spread Demo Subsequent wave Spread Starting ≤ 50% Starting > 50% Mean number of months to reach specic level of quality Subsequent wave that reached those levels, combined across MNCH, FP, HIV,TB, and other topic areas. The extra time and effort needed to go from 80% to 90% is not insignificant, taking from 34% to 140% longer than the time needed to reach 80%. This represents an additional 2–7 months on average. Were gains in the quality of care maintained? Being able to reach levels of 80% or 90% is laudable and important, but if the changes made to achieve that level are not sustainable, the quality of care will likely decline over time. While some degree of variation is inherent in every process, the goal of quality improvement is to both increase the level of quality of care and decrease the variation in quality across patients and over time. The individual time series charts presented in Appendix 4 point to an array of variation patterns, with some charts showing a line that plateaus and remains relatively flat (stable process with little variation), while others display lines that continue to “ping Figure 4: Speed of improvement: Mean time in months to achieve 80% (n=118 time series charts) compliance with standards for MNCH and FP and HIV and TB collaboratives 0 2 4 6 8 10 12 14 16 Demo Spread Demo Spread Starting ≤50% Starting > 50% Months to acheive level of 80% MNCH and FP indicators reaching 80% Subsequent wave Subsequent wave HIV and TB indicators reaching 80% time series charts with low baseline levels. While the number of time series charts analyzed for the spread group is relatively small, additional data should be available in the next 12 months from sites currently participating in deliberate spread processes. Figure 5 presents a comparison of average time to reach 80% and 90% levels for the 118 and 102 time series charts, respectively, 12 Results of Collaborative Improvement in 12 Countries pong.” While significant variation (indicated by a “ping-pong” pattern) in a process may be due to small sample sizes in the measurement of performance, it may also indicate an unstable process for which quality performance has not been built into the care delivery system. It is common for temporary “stresses” to a process—such as inducing staff to work harder or temporarily providing additional resources—to produce jumps in performance, but when these efforts subside, performance declines to previous levels. For improvements in performance to be permanent (i.e., sustainable), processes and systems must be redesigned in ways that enable teams to consistently produce better results. In the analysis presented below, we sought to examine the ability of process improvements to consistently maintain the level of quality above 80% over time with minimal variation (indicating a stable, effective process). We analyzed improvement maintenance in two ways: 1) maintaining indicator measurements above 80% after having reached that level and 2) the stability of quality (that is, the variation in the percentage level once it stabilizes). For the 135 time series charts, data were collected, on average, for 29 months from the baseline. For the 118 time series charts that reached 80%, data were available for 20 months, on average, from the time the indicator reached 80%. Table 6 presents results on the time duration for which gains were maintained, showing the number of consecutive months that results remained above 80% once that level was reached. For our analysis, we were strict in applying the “consecutive months” standard: If an indicator dipped below 80% even once, we assessed maintenance of gains as having ended. While the improvement collaboratives studied did not always maintain performance at or above 80%, they did accomplish continuous performance at that level and above that averaged 13.4 months.This represents an average of 72% (median=86%) of months since reaching 80%, regardless of how long data were available. If changes leading to improvement become integrated into the daily routine (the “default” practice), one would expect only small variations in results once a new, stable level had been reached, as seen in the AMTSL indicator in Box 5. We analyzed data on the 87 time series charts that plateaued at 80% or higher to determine the degree of variation between the highest and lowest values in each stabilized level. The mean interval (range) between lowest and highest value once the levels plateaued was 11 percentage points, with significant disparity across clinical areas: 18.3 for MNCH and FP, 14.5 for HIV andTB, and 6.5 for other. Examples of this kind of variation can be seen in some of the time series charts in Appendix 4. While some time series charts displayed wide variation, others were stable. Box 5 discusses the outcomes resulting from sustained levels of consistent (i.e., minimal variation) provision of AMTSL in the facilities participating in the Niger collaborative: a significant and consistent drop in postpartum hemorrhage rates. 5. Discussion and conclusions Our findings highlight the capacity and potential of modern quality improvement to enable rapid and significant gains in health care service quality for high impact interventions. The evidence also points to the ability of improvement collaboratives to maintain these improvements over time. Level of improvement achieved The evidence from the 27 improvement collaboratives demonstrates that, through testing and sharing changes, large increases in the level of performance (compliance with standards as well as health outcomes) are feasible across various topic areas (maternal and neonatal health, child health, family planning, malaria,TB, and HIV/AIDS), regardless of the baseline level of quality: 88% of time series charts showed levels of at least 80% compliance, and 76% reached at least 90%, even though more than half started at 50% or below. The low baseline values are common findings in published assessments for most low- and middle-income county settings, where, even if up-to-date standards exist, they are not routinely implemented. Collaborative improvement focuses on testing changes in care systems needed to ensure routine implementation of evidence-based practices for every patient. Some examples of effective changes tested in the improvement collaboratives analyzed here include: • Reorganizing call schedules to ensure reliable presence of skilled personnel, • Organizing pre-filled syringes of Oxytocin on ice packs in the delivery room, • Enlisting lay counselors to coach patients in producing sputum samples, • Creating new information flows between different clinical services, Table 6: Mean (range) number of months a time series chart showed maintenance of 80% or higher Topic Mean # (range) of months consistently above 80% Average # months of data after reaching 80% Total # of months of data # of time series charts MNCHandFP 14.2 (0–57) 20.9 31 82 HIV andTB 13.2 (2–42) 17.6 24 25 Other 8.8 (1–21) 12.5 23 11 ALL TOPICS 13.4 (0–57) 19.5 29 118 Results of Collaborative Improvement in 12 Countries 13 • Using monthly phone calls to satellite clinics to check on patients with a history of poor adherence to treatment regimens, • Providing public feedback on performance, • Integrating newborn and postpartum care for the mother, • Task-shifting for counseling of HIV-positive patients, and • Sending samples rather than patients to labs for testing purposes. Some indicators measure elements that are harder to improve than others. For example, some indicators that did not reach 80% relied on patient behavior change (e.g., percentage of par tners taking an HIV test) rather than provider behavior change; some measured compliance with standards that were not yet well accepted (e.g.,AMTSL in Ecuador); and some involved complex clinical tasks that were new to providers (e.g., pediatric hospital improvement in Niger). Experiences in improvement around the world have shown that a certain level of performance can be achieved through a few key actions oriented to ensuring that what needs to be done is done: basic standardization of processes, awareness raising and skill building, and feedback mechanisms. These actions go beyond simply providing inputs (training, supplies) and focus on clarifying and advocating for standards to be implemented, testing changes in organization that lead to a basic effective process becoming the “standard”, and using feedback through monitoring and analysis of indicator data. This first basic improved level of performance seems to plateau at about 80% compliance with standards worldwide and is reflected in the experiences described here. Several collaboratives moved beyond this basic level, however; doing so involves a shift in thinking, to determine: 1) what additional changes in the process are needed to address those situations in which care is not provided in accordance with standards, and 2) how to ensure that these new, improved processes become the normal way of doing things. Box 5 offers an example of such efforts to make consistently high quality care the routine. Speed of improvement The evidence on speed of improvement corroborates the hypothesis that improvement collaboratives can achieve rapid results – across all types of collaboratives, with time series charts starting below 50% reaching 80% on average in 13 months, while those starting above 50% reached 80% in six months. These results on “speed” (and other aspects) were generated from an average of 27 sites per collaborative, indicating an ability to achieve speed at some level of scale as well. 19 NolanT, Resar R, Haraden C, Griffin FA. 2006. Improving the reliability of health care. White Paper. Boston, MA: Institute for Healthcare Improvement. Subsequent wave teams offer the most intriguing possibilities in terms of increasing the speed of improvements, be they part of an informal or a deliberate spread process. The evidence indicates that the sharing of ideas from wave to wave speeds improvement: Subsequent teams generally achieved results at levels of 80% in about 50% less time than the original site teams had done. Follow-up questions include how many waves could a single region support and maintain both the speed and levels of quality we have seen in countries like Ecuador. In many of these collaboratives, “spread” was begun, at the request of host-country governments and donors, before the “demonstration” phase had been completed: There was pressure to increase the geographic coverage. This resulted in some “spread collaboratives” repeating the demonstration collaborative’s activities rather than building on them.The number of examples of deliberate spread is still small in this dataset (although several more examples will be available in the next 12 months), but these preliminary results suggest that deliberate spread (with synthesis and consolidation of learning packaged for new sites) may produce even more reductions in the time needed to achieve results near 80% or higher. A key strategy for this increased speed is using champions from the demonstration phase and equipping them to support quality improvement in new sites because they bring knowledge, experience, lessons learned, and enthusiasm to the task. The time needed to make an improvement to 90% is not insignificant, sometimes requiring 35–140% more time than to reach 80%. This finding is consistent with reliability science19 and highlights the need for efforts, beyond those focused on creating a process in which most situations will result in things done right, to organizational changes that allow one to catch and correct where things might be done wrong before they happen, so that all patients receive quality services. Perhaps the most pressing challenge for any improvement approach is its ability to take learning about what changes in processes are needed to increase and then maintain quality of care to achieve similar or better results at scale. Collaborative improvement can create the conditions for such spread, as demonstrated by the more rapid improvement in new waves of teams. These results may be even more dramatic in cases where the demonstration phase was completed at a point of productive learning and where results were appropriately synthesized for sharing. In many of these collaboratives, “spread” was begun, at the request of host-country governments and donors, before the “demonstration” phase had been completed, due to pressure to increase geographic coverage. This resulted in some “spread 14 Results of Collaborative Improvement in 12 Countries collaboratives”repeating the demonstration collaborative’s activities rather than building on them. The number of examples of deliberate spread is still small in this dataset (although several more examples will be available in the next 12 months), but these preliminary results suggest that deliberate spread (with synthesis and consolidation of learning packaged for new sites) may produce even more reductions in the time needed to achieve results near 80% or higher. While not represented in the data set analyzed here, key questions which are being tested in current implementation experience include: what role can web sites and documents play in the spread of improvement to new areas? To what extent do new teams need QI skills to accompany information about what will help them consistently achieve high quality care for all their patients? Experience to date already indicates that a key strategy for this increased speed is using champions from the demonstration phase and equipping them to support quality improvement in new sites because they bring knowledge, experience, lessons learned, and enthusiasm to the task. Maintaining gains over time The evidence on maintaining gains in quality over time indicates that the achievements discussed above are not ephemeral: The achievement of quality at levels of at least 80% has been sustained for an average of 13 consecutive months during at least 19 months of subsequent data collection. In some cases, sustained levels were maintained for as long as four years. Reliable processes (indicated by reaching levels of 95% or higher)20 are not yet in place in most collaboratives, and variation continued even when the indicator had plateaued. The health systems in most of the countries where these collaboratives occurred have weaknesses – such as significant staff movement, lack of critical supplies at the national level, and other system-wide aspects – that affect health facilities’ ability to achieve and maintain gains in quality. Care and support processes need to be designed with these factors in mind: error-proofing is required at both local and higher levels of the system. Conclusions Experiences of implementing the improvement collaborative approach have evolved over the last ten years. This paper presents evidence across 27 improvement collaboratives in 12 countries that indicates that collaborative improvement: • Raised 88% of the indicators of quality of care studied to levels of at least 80%, and in 76% of cases, above 90%, regardless of initial levels; • Reached these levels of performance within an average of about a year; and • Maintained consistent care at an improved level for an average of 13.4 consecutive months. Not every collaborative achieved these results, but a majority did. Improvement collaboratives included in our analysis were implemented over the period 1998–2008, and it is wor thwhile bearing in mind that the collaborative approach itself evolved over that period, as our understanding grew about the process of implementing collaborative improvement in resource￾constrained settings. Our results indicate that modern QI has the capacity to strengthen health systems’ ability to delivery quality care to a majority of patients/clients. Qualitative data from collaborative evaluations21 provide some insight into why modern QI improvement can achieve these results. These data indicate that participating in a collaborative 1) fosters the engagement of health staff in the process of improvement and 2) helps them understand how processes work and how to make operational changes to achieve quality for every patient. Furthermore, sharing changes and results motivates teams to work hard and produce good results. Maintaining changes over time is facilitated when higher levels of the health system routinely check on results, recognize good work, and integrate quality indicators into national monitoring systems. Such support from higher levels also builds expectations for quality improvement into job descriptions and performance evaluations and creates mechanisms to motivate the application of QI. We conclude that collaborative improvement is one effective strategy to ensure the sustainability of results and institutionalization of improvement. 6. Future directions and implications for health systems strengthening Since modern quality improvement approaches are capable of producing better patient outcomes and improved care processes, they contribute to strengthening health systems’ ability to improve results. Our analysis of 135 time series charts measuring quality of care and outcomes across 27 collaboratives reveals other areas for work that would likely bring these results to higher and more consistent levels and possibly even more rapidly at larger scales. The next frontier in quality improvement: Achieving even higher levels of quality, more rapidly, at larger scale Improving outcomes through redesign of care processes to reduce errors and assure quality for all patients: Quality 20 The level of 95% equals a level of reliability of 10-2 for a bundle of measures. Ibid. 21 CatsambasTT, Franco LM, Gutmann M, Knebel E, Hill P, LinY-S. 2008. Evaluating health care collaboratives:The experience of the Quality Assurance Project. Collaborative Evaluation Series. Bethesda, MD: USAID Health Care Improvement Project, University Research Co., LLC. Results of Collaborative Improvement in 12 Countries 15 improvement efforts in most of these collaboratives focused on integrating standards into the way health workers perform their tasks. Several of the newer collaboratives, some not yet analyzed due to as yet insufficient data, focus on “reliable design”: redesigning care processes to ensure consistently high levels of quality care and results22, moving from 80% to 90%, and then to 95%. Such redesign involves addressing process issues to reduce defects, compensate for the limits of human ability, and take into account situational factors that affect outcomes for individual patients. More importantly, once processes are redesigned, they must be “hardwired” into the existing system to ensure that they are maintained despite changes in personnel. In health care, we are striving to achieve high levels of reliable care (95% or higher), not only on single tasks, but on groups of tasks or interventions that can together positively affect outcomes. This kind of reliability has been demonstrated in the Niger AMTSL example (shown in Box 4): levels of compliance with AMTSL’s three key elements (Oxytocin one minute after delivery, controlled cord traction, and uterine massage) reached more than 98% of women delivering in health facilities, and these results were maintained over time. With this level of reliability, we see significant improvement in post-partum hemorrhage. These results are consistent with the IHI experiences in the United States, i.e., with the ventilator-associated pneumonia improvements in the “Saving 100,000 Lives” campaign.23 Work on institutionalization is now generating discussions on how to ensure that the frequent transfer of personnel (an extremely common practice among health systems using clinical civil service personnel) disrupts neither the quality of care to patients nor the continued focus on QI. Accelerating improvement at scale: Many of the analyzed collaboratives have since moved into deliberate spread phases, including systematically gathering, synthesizing, and consolidating information on what changes worked best across teams and led to actual improvements; communication of what was learned by the initial teams to new sites; and engaging experienced implementers from the initial sites to serve as “champions” of the new practices who coach and support new teams. Results of this type of spread will be available in the next 6–12 months for careful analysis. The USAID Health Care Improvement Project is also conducting a series of small research projects on the spread of learning: within a collaborative, from demonstration to spread sites, and from one country to another. This learning is not limited to the organizational changes that lead to consistent care provided according to evidence-based standards, but also includes increased capacity among Ministries of Health and other local implementing partners to manage collaboratives, adapt QI to local conditions, and provide technical support to sites. Implications for the role of modern improvement in the health systems-strengthening agenda Our discussion of findings shows that modern quality improvement approaches have strengthened health systems, often rapidly and usually on a sustained basis. The strength of a health system is measured in its ability to deliver good health outcomes. Modern improvement redesigns care systems to achieve these outcomes. It is not the only approach needed for health systems strengthening, but it offers one effective way to address the weaknesses in the functioning of health systems. Long-term sustainability will require other health system interventions to ensure logistics and stable, competent human resources, as well as accountability for results. However, the USAID-funded experiences reported here strongly recommend a place for quality improvement in the basket of strategies to strengthen health systems in low- and middle-income countries. Quality improvement has demonstrated its ability to achieve significant, sustained improvements in care and outcomes well beyond the demonstration phase and at scale. Modern quality improvement’s focus on making changes in processes to change outcomes are as applicable to other health systems processes as they are to service delivery. The USAID Health Care Improvement Project is currently applying quality improvement approaches in several countries to improve human resource management, community-based services, social services for vulnerable children affected by HIV, and other non-clinical areas. Achieving the Millennium Development health goals will require that people receive effective health care delivered according to locally adapted, internationally accepted standards. The obstacles go beyond inputs, technology and training, because health systems have process flaws that prevent effective, efficient and equitable service delivery. The evidence presented here demonstrates the potential that quality improvement approaches can have to achieve the targets of both health systems strengthening and the Millennium Development Goals. QI should be considered an essential approach for strengthening health systems. 22 Op cit., Nolan et al. 2006. IHI proposes a three-tiered strategy for designing reliable care systems: 1. Prevent failure (takes steps to prevent breakdowns in procedures); 2. Identify and mitigate failure: Identify failure promptly when it occurs and intercede before harm is caused, or mitigate the harm caused by failures that are not detected and intercepted; and 3. Redesign the standards process based on the critical failures identified. 23 Berwick DM, Calkins DR, McCannon CJ, Hackbarth AD. 2006. The 100,000 Lives Campaign: Setting a goal and a deadline for improving health care quality. Journal of the American Medical Association. 295(3):324-327. 16 Results of Collaborative Improvement in 12 Countries Results of Collaborative Improvement in 12 Countries 17 Appendix 1: List of 54 USAID-funded collaboratives: 1998–2008 (Note:The collaboratives listed in bold have data included in the analysis in this paper) Collaborative Topic Time Period Project Benin EONC (demonstration) Essential obstetric and newborn care 2/05–9/08 QAP III/HCI Benin EOC Essential obstetric care 5/08–present PISAF Benin malaria Malaria 9/08–present PISAF Benin mutual health organizations Community financing of health care 6/08–present PISAF Bolivia DOTS (demonstration) Tuberculosis 1/07–12/08 HCI Ecuador EOC Essential obstetric and newborn care 8/03–12/07 QAP III Ecuador complications Essential obstetric and newborn care 10/06–12/07 HCI Ecuador spread AMTSL Essential obstetric and newborn care 5/07–12/07 QAP III Eritrea PHI Child health 7/03–9/05 QAP III Eritrea EONC Essential obstetric and newborn care 7/04–10/05 QAP III Guatemala PHI Child health 6/04–12/04 QAP III Guatemala EONC basic (San Marcos demonstration) Essential obstetric and newborn care 8/07–9/08 TASC Guatemala EONC community Essential obstetric and newborn care 6/08–present TASC/HCI Guatemala EONC basic (spread to 8 health areas) Essential obstetric and newborn care 12/08–present HCI Guatemala STI Sexually transmitted diseases 11/08-9/09 TASC Honduras EOC and child health (5 demonstration regions) Essential obstetric and newborn care 11/03–present QAP III/HCI Honduras EOC replication (6 new regions) Essential obstetric and newborn care 1/07–present HCI Honduras IMCI/diarrhea/ pneumonia (La Paz region) Child health 12/08–present HCI Malawi PHI Child health 6/04–12/05 QAP III Nicaragua PHI Child health 10/03–12/07 QAP III Nicaragua EOC Essential obstetric and newborn care 9/03–11/08 HCI NicaraguaVCT-FP integration HIV 2/06–12/07 QAP III NicaraguaVCT-STI integration HIV 4/08–present HCI Nicaragua newborn hand hygiene Infection prevention 3/08–present HCI Niger PHI Child health 10/03–8/07 QAP III Niger AMTSL/essential newborn care Essential obstetric and newborn care 1/06–12/08 HCI Niger eclampsia Essential obstetric and newborn care 4/08–12/08 HCI Russia NRDS (demonstration Tver) Child health 11/98–6/00 QAP II Russia PIH (demonstration Tver) Essential obstetric and newborn care 11/98–6/00 QAP II 18 Results of Collaborative Improvement in 12 Countries Collaborative Topic Time Period Project Russia AH (demonstration Tula) Primary health care 11/98–6/00 QAP II Russia NRDS (spread Tver) Child health 5/00–6/02 QAP II Russia PIH (spread Tver) Essential obstetric and newborn care 5/00–6/02 QAP II Russia AH (spread Tula) Primary health care 5/00–6/02 QAP II Russia (“Phase III” national) MCH Maternal and child health 6/02–12/04 QAP III Russia (“Phase III” national) PIH/NRDS Essential obstetric and newborn care 6/02–12/04 QAP III Russia (“Phase III” national) cardiovascular disease Primary health care 6/02–12/04 QAP III Russia HIV/AIDS treatment, care and support HIV 11/04–12/06 QAP III Russia FP for PLWHA Family planning 3/06–9/07 QAP III Russia ART for intravenous drug users HIV 5/07–present HCI Russia ART (Orenburg spread) HIV 5/07–9/08 HCI Russia ART (St. Petersburg spread) HIV 3/07–present HCI Russia social services HIV-positive women HIV 3/07–present HCI RussiaTB-HIV (Orenburg spread) Tuberculosis 5/07–9/08 HCI RussiaTB-HIV (St. Petersburg spread) Tuberculosis 3/07–present HCI Rwanda PMTCT/VCT HIV 7/03–8/06 QAP III Rwanda Malaria Malaria 6/03–8/06 QAP III Rwanda ART HIV 7/04–8/06 QAP III Tanzania infection prevention Infection prevention 4/03–6/04 QAP III Tanzania PHI/pediatric AIDS Child health 10/04–6/08 QAP III Tanzania FP Family planning 10/04–7/06 QAP III Tanzania PHI/pediatric AIDS (spread) Child health 12/05–6/08 QAP III Tanzania ART/PMTCT (Tanga/AIDS Relief) HIV 5/08–present HCI Uganda ART/pediatric HIV/TB HIV 1/06–9/09 QAP III/HCI Vietnam TB-HIV Tuberculosis 4/07–present QAP III/ HCI Results of Collaborative Improvement in 12 Countries 19 Appendix 2: List of 27 collaboratives for which data were available and analyzed: Topics and scale Country Topic/Sponsoring Project Dates # of Sites in Collaborative Scale/Proportion of Sites/Areas in Country Benin EONC/QAP 2/05–9/08 10 facilities in 1 district 1 of 34 districts Benin EOC/PISAF 5/08–present 17 facilities in 6 districts 6 of 34 districts Benin Malaria/PISAF 9/08–present 15 facilities in 6 districts 6 of 34 districts Bolivia DOTS (demonstration)/QAP 1/07–12/08 114 facilities in 16 municipalities 16 of 169 municipalities Ecuador EOC(wave 1)/QAP EOC (wave 2)/QAP EOC (wave 3)/QAP 8/03–12/07 1/05–12/07 1/07–12/07 55 facilities in 48 districts 21 facilities in 18 districts 10 facilities in 9 districts 75 of 169 districts (130 of which have facilities that offer obstetrical care) Ecuador Obstetric complications/QAP 10/06–12/07 6 hospitals in 6 provinces 6 of 22 provincial hospitals Guatemala Basic EONC (San Marcos demonstration)/Calidad en Salud 8/07–09/08 22 sites in 1 health area 1 of 29 health areas Guatemala Community EONC (San Marcos)/ Calidad en Salud Highland communities Lowland communities 9/07–present 6/08–present 22 of 22 districts 14 facilities in 11 districts 15 facilities in 11 districts 1 of 29 health areas Honduras EOC and child health demonstration in 5 original health regions/QAP and HCI 11/03–9/09 All 127 facilities in 5 regions (5 hospitals, 14 maternal clinics, 108 health centers) 5 out of 20 regions in the country Nicaragua PHI/QAP 10/03–12/07 19 of 21 hospitals; 97 of 175 health centers; 16 of 17 health areas (SILAIS) 19 out of 21 hospitals (90%); 97 out of 175 health centers (55%); 16 out of 17 SILAIS (94%) Niger PHI (wave 1)/QAP PHI (wave 2)/QAP 10/03–8/07 3/05–8/07 14 hospitals 15 hospitals 70% of first referral hospitals (32/46) in 7 of 8 regions Niger EONC:AMTSL/ENC/QAP and HCI (hospitals) EONC wave 2 (maternities) 1/06–12/08 1/07–12/08 29 hospitals in 7 regions 10 out of 139 peripheral maternities 68% of maternity hospitals (28/41) 7% of peripheral maternities Niger EONC: eclampsia/HCI 4/08–12/08 11 reference hospitals; 20 peripheral maternities 27% of maternity hospitals (11/41) 14% of peripheral maternities Russia NRDS (demonstration)Tver)/QAP 3/98–3/00 5 sites 1 of 89 oblasts 20 Results of Collaborative Improvement in 12 Countries Country Topic/Sponsoring Project Dates # of Sites in Collaborative Scale/Proportion of Sites/Areas in Country Russia NRDS (spread inTver)/QAP 3/00–6/02 42 sites Entire oblast health system (1/89 oblasts) Russia PIH (demonstration inTver)/QAP 3/98–3/00 3 sites 1 of 89 oblasts Russia PIH (spread inTver)/QAP 3/00–6/02 40 sites Entire oblast health system (1/89 oblasts) Russia AH (demonstration inTula)/QAP 3/98–3/00 5 sites 1 of 89 oblasts Russia AH (spread inTula)/QAP 3/00–6/02 442 sites Entire oblast health system (1/89 oblasts) Rwanda PMTCT/VCT/QAP: wave 1 PMTCT/VCT/QAP: wave 2 7/03–8/06 17 original sites 19 expansion sites 4 hospitals (12% of district hospitals) 32 health centers (9% of health centers) Rwanda ART/QAP 7/04–8/06 30 sites 5 hospitals (15% of district hospitals) 25 health centers (7% of health centers in the country) Rwanda Malaria/QAP: wave 1 Malaria/QAP: wave 2 6/03–8/06 6/05–8/06 3 district hospitals and 20 health centers in 4 districts 31 health centers in 4 districts 3 hospitals (9% of district hospitals) 20 health centers (6% of health centers in the country) 31 health centers (9% of health centers in the country) Tanzania PHI and pediatric AIDS 10/04–6/08 5 hospitals in 3 regions 3 regions of 26 Tanzania Family planning 10/04–7/06 9 facilities in all 3 districts in 1 region (capital) 3% of districts (3/122) Tanzania ART/PMTCT:Tanga AIDS Relief 5/08–Present 1 out of 26 regions 8 facilities (the regional hospital, all 5 district hospitals, and 2 health centers) 4% of mainland regions (1/26) Uganda ART, PMTCT, TB: demonstration ART, PMTCT, TB: wave 1 ART, PMTCT, TB: wave 2 1/06–6/08 1/07–12/08 1/08–9/09 57 sites 32 sites 31 sites Total now at 120 of 336 accredited ART sites (36%), in 11 of 12 regions and 71 of 80 districts Vietnam TB-HIV 4/07–present All 13 hospitals in all 8 districts of Thai Binh Province 1 out of 63 provinces (2%) Abbreviations: AH Ar terial hyper tension;AIDS Acquired immunodeficiency syndrome;AMTSL Active management of the third stage of labor;ART Antiretroviral therapy; DOTS Directly observed treatment, short course; ENC Essential newborn care; EOC Essential obstetric care; EONC Essential obstetric and newborn care; FP Family planning; HIV Human immunodeficiency virus; HR Human resources; IDU Intravenous drug user; MNCH Maternal, newborn and child health, NRDS Neonatal respiratory distress syndrome; PHI Pediatric hospital improvement; PIH Pregnancy-induced hyper tension; PMTCT Prevention of mother-to-child transmission of HIV; STI Sexually transmitted infection;TBTuberculosis;VCTVoluntary counseling and HIV testing. Results of Collaborative Improvement in 12 Countries 21 Appendix 3: Indicators included in the analysis of the 135 time series charts assessed in the 27 collaboratives Indicators Countries (# of sites reporting) MATERNAL, NEWBORN, AND CHILD HEALTH Newborn health Percent of newborns with a temperature above 36.5o Celsius Benin (10) Percent of newborns immediately breast fed Benin (10) Niger (39) Percent of newborns receiving immediate newborn care according to standards Benin (17) Ecuador (86) Niger (39) Proportion of newborns for whom at least 80% of surveillance standards were respected Benin (17) Percent of newborns examined, registered in Perinatal Clinical History according to standards Honduras (19) Percent of newborns diagnosed with neonatal sepsis managed according to standards Honduras (47) Percent of newborns with temperature taken Niger (39) Number of neonates arriving to the center with hypothermia Russia (5) Neonatal mortality in the first week of life Russia (5) Neonates with respiratory distress who died in the first week of life Russia (42) Mortality of newborns due to respiratory distress Russia (42) Child health Percent of children under 2 receiving consultations and correctly registered in monitoring list Honduras (108) Percent of children under 5 who were correctly evaluated according to Integrated Management of Childhood Illness (IMCI) guidelines Honduras (108) Percent of children under 5 diagnosed with pneumonia managed according to standards Honduras (108) Niger (14) Percent of children under 5 diagnosed with dehydration and managed according to standards Honduras (108) Niger (14) Percent of children with danger signs receiving care according to standards Nicaragua (13) Niger (14) Percent of children with severe pneumonia diagnosis receiving care according to standards Nicaragua (13) Tanzania (5) Percent of children with bronchial obstruction who received care according to standards Nicaragua (13) Percent of children with diarrhea and dehydration receiving care according to standards Nicaragua (13) Percent of children cared for in an integrated manner Nicaragua (13) 22 Results of Collaborative Improvement in 12 Countries Indicators Countries (# of sites reporting) Percent of children 0–59 months coming for consultations hospital who were triaged upon arrival Niger (29) Tanzania (5) Percent of analyzed emergency triage and treatment (ETAT) forms for children 0–9 months with 80% compliance with ETAT norms Niger (29) Percent of analyzed charts for children with dehydration with 80% compliance with case management norms Niger (29) Percent of analyzed charts for children with pneumonia with 80% compliance with case management norms Niger (29) Average percent of standards for acute malnutrition recuperation complied with Niger (15) Children with malaria managed at hospital according to standards Niger (15) Tanzania (5) Percent of analyzed charts of children with malaria for which 80% of case management norms were complied with Niger (29) Percent of HIV-positive children managed according to HIV case management guidelines Tanzania (5) Percent of children managed according to case management guidelines (combined HIV, malaria and pneumonia) Tanzania (5) Maternal health Percent of women receiving counseling about birth preparedness Benin (10) Percent of deliveries for which the three elements of active management of the third stage of labor (AMTSL) were applied according to standards Benin (17) Niger (39) Percent of women receiving antenatal care according to standards Ecuador (86) Percent of deliveries in which the partogram was used properly Ecuador (86) Guatemala (29) Honduras (126) Percent of deliveries with Oxytocin administered within a minute of delivery Ecuador (86) Honduras (126) Percent of deliveries which received postpartum care according to standards Ecuador (86) Percent of women with eclampsia who were managed according to standards Ecuador (6) Percent of women with hemorrhage who were managed according to standards Ecuador (6) Percent of women with sepsis who were managed according to standards Ecuador (6) Pregnant women mentioning ≥ 3 danger signs during pregnancy, delivery and postpartum Guatemala (14) Pregnant women who have an emergency plan Guatemala (14) Pregnant women who mentioned at least 3 danger signs in the newborn Guatemala (29) Percent of deliveries for which at least 80% of the post-partum surveillance standards were respected Benin (17) Niger (39) Percent of deliveries resulting in hemorrhage Benin (17) Niger (39) Results of Collaborative Improvement in 12 Countries 23 Indicators Countries (# of sites reporting) Percent of pre-eclampsia and eclampsia case management standards complied with Niger (31) Percent of women with pregnancy-induced hypertension delivered this month Russia (40) Hospitalizations due to pregnancy-induced hypertension Russia (3) HIV/AIDS Treatment and care for HIV-positive patients Percent of patients on ARVs who adhere to treatment at 95% Rwanda (11) Uganda (120) Percent of patients on antiretrovirals (ARVs) with medical records completely filled out Rwanda (10) Percent of HIV-positive patients on ART who are lost to follow-up per month Tanzania (6) Percent of HIV-positive patients assessed for tuberculosis (TB) Uganda (120) Percent of patients on ARVs with clinical improvement Uganda (44) Prevention of mother -to -child transmission Percent of women tested for HIV who returned for their results Rwanda (36) Percent of women testing positive for HIV whose partners are also tested Rwanda (36) Percent of children born to HIV-positive mothers who were tested at 15–18 months Rwanda (36) Percent of pregnant women in antenatal care (ANC) who tested positive for HIV and enrolled at care and treatment centers (CTC) Tanzania (6) Uganda (120) Pediatric HIV/AIDS care Percent of HIV-exposed children under 18 months receiving daily Cotrimoxazole per month Tanzania (6) Percent of HIV-exposed infants receiving ARV prophylaxis Tanzania (6) MALARIA Performance score for use of rapid diagnostic tests for malaria at health centers Benin (15) Percent of children under 5 with fever whose weight was registered in the consultation chart Benin (15) Percent of children under 5 with fever correctly treated with Artemisinin combination therapy (ACT) Benin (15) Percent of adult fever patients with a positive rapid diagnostic test correctly treated with ACT Benin (15) Percent of children managed at hospital according to malaria case management standards Rwanda (3) Percent of children with fever taken to the health center within 24 hours Rwanda (48) Percent of children managed at health centers according to malaria case management standards Rwanda (48) 24 Results of Collaborative Improvement in 12 Countries Indicators Countries (# of sites reporting) TUBERCULOSIS New pulmonaryTB cases detected over those programmed for each quarter Bolivia (114) Patients who receive directly observed treatment (DOTS) for TB Bolivia (114) Cure rate among pulmonary TB cases programmed to end their treatment in the quarter Bolivia (114) Percent of all samples sent to the lab that contained saliva Bolivia (114) Percent ofTB patients counseled on HIV Vietnam (13) Percent ofTB patients tested for HIV Vietnam (13) FAMILY PLANNING Percent of users of post-obstetric events who received family planning counseling Honduras (108) Percent of family planning clients weighed Tanzania (9) Percent of family planning clients screened for high blood pressure Tanzania (9) Percent of family planning clients screened for thyroid, lumps Tanzania (9) Percent of family planning clients with urine tested Tanzania (9) Percent of family planning clients with abdominal exam Tanzania (9) Percent of family planning clients with vaginal exam Tanzania (9) Percent of family planning clients with date for next visit registered Tanzania (9) OTHER Percent of population diagnosed with arterial hypertension Russia (5) Hospitalization due to arterial hypertension Russia (442) Percent of diagnosed cases experiencing hypertensive crises Russia (442) Results of Collaborative Improvement in 12 Countries 25 Appendix 4: Time series charts included in the analysis, by topic area (n=135) Maternal, newborn, and child health .........................................................................................................................27 HIV ...................................................................................................................................................................................40 Malaria .............................................................................................................................................................................44 Tuberculosis....................................................................................................................................................................46 Family planning...............................................................................................................................................................47 Other...............................................................................................................................................................................48 Results of Collaborative Improvement in 12 Countries 27 MATERNAL, NEWBORN, AND CHILD HEALTH 0 20 40 60 80 100 2007 Percentage Benin I - Newborns immediately breast fed (n=10 sites) 0 20 40 60 80 100 2007 2008 Percentage Benin I - Newborns with a temperature above 36.5o C (n=10 sites) 0 20 40 60 80 100 2008 Percentage Benin I – Women receiving counseling about birth preparedness (n=10 sites) 0 6 12 18 24 30 2008 2009 Percentage Benin II – Deliveries resulting in hemorrhage (Downward trend signals improvement) (n=17 sites) 0 20 40 60 80 100 2008 Percentage 2009 Benin II - Newborns receiving immediate newborn care according to standards (n=17 sites) 0 20 40 60 80 100 2008 Percentage 2009 Benin II - Deliveries with all three AMTSL elements were applied (n=17 sites) 0 20 40 60 80 100 2008 Percentage 2009 Benin II - Deliveries with at least 80% of the post-partum surveillance standards respected (n=17 sites) 0 20 40 60 80 100 2008 Percentage 2009 Benin II - Newborns for whom at least 80% of surveillance standards were respected (n=17 sites) 28 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 2009 Ecuador - Women receiving antenatal care according to standards [Wave 2] (n=21 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Ecuador - Women receiving antenatal care according to standards [Wave 3] (n=10 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 2007 2008 2009 Ecuador - Deliveries with proper use of partogram [Wave 1] (n=55 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 2007 2008 2009 Ecuador - Women receiving antenatal care according to standards [Wave 1] (n=55 sites) Results of Collaborative Improvement in 12 Countries 29 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 2009 Ecuador - Deliveries with proper use of partogram [Wave 2] (n=21 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Ecuador - Deliveries with proper use of partogram [Wave 3] (n=10 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 2007 2008 2009 Ecuador – Deliveries with Oxytocin administered within a minute of delivery [Wave 1] (n=55 sites) 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 2009 Ecuador – Deliveries with Oxytocin administered within a minute of delivery [Wave 2] (n=21 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Ecuador – Deliveries with Oxytocin administered within a minute of delivery [Wave 3] (n=10 sites) 30 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 2007 2008 2009 Ecuador - Deliveries which received postpartum care according to standards [Wave 1] (n=55 sites) 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 2009 Ecuador - Deliveries which received postpartum care according to standards [Wave 2] (n=21 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Ecuador - Deliveries which received postpartum care according to standards [Wave 3] (n=10 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 2007 2008 2009 Ecuador - Newborns receiving immediate newborn care in accordance with standards [Wave 1] (n=55 sites) Results of Collaborative Improvement in 12 Countries 31 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 2009 Ecuador - Newborns receiving immediate newborn care in accordance with standards [Wave 2] (n=21 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Ecuador - Newborns receiving immediate newborn care in accordance with standards [Wave 3] (n=10 sites) 0 20 40 60 80 100 2007 Percentage Ecuador - Women with eclampsia managed according to standards (n=6 sites) 0 20 40 60 80 100 2007 Percentage Ecuador - Women with sepsis managed according to standards (n=6 sites) 0 20 40 60 80 100 2007 Percentage Ecuador - Women with hemorrhage managed according to standards (n=6 sites) 0 20 40 60 80 100 2007 Percentage 2008 2009 Guatemala - Pregnant women mentioning ≥ 3 danger signs during pregnancy, delivery and postpartum (n=22 sites) 0 20 40 60 80 100 2007 Percentage 2008 2009 Guatemala - Pregnant women with an emergency plan (n=11 sites) 32 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 2007 Percentage 2008 Guatemala - Deliveries with proper use of partogram (n=22 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Deliveries with proper use of partogram (n=19 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras – Deliveries with Oxytocin administered within a minute of delivery (n=19 sites) 0 20 40 60 80 100 2007 Percentage 2008 2009 Guatemala - Pregnant women mentioning ≥ 3 newborn danger signs [Wave 1] (n=11 sites) 0 20 40 60 80 100 2008 Percentage 2009 Guatemala - Pregnant women mentioning ≥ 3 newborn danger signs [Wave 2] (n=11 sites) Results of Collaborative Improvement in 12 Countries 33 0 20 40 60 80 100 2008 Percentage Honduras - Newborns diagnosed with neonatal sepsis and managed according to standards (n=47 sites)* 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Newborns examined, registered in Perinatal Clinical History according to standards (n=19 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Children under 2 receiving consultations and correctly registered in monitoring list (n=108 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Children under 5 correctly evaluated according to Integrated Management of Childhood Illness (IMCI) guidelines (n=108 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Children under 5 diagnosed with pneumonia managed according to standards (n=108 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Children under 5 diagnosed with dehydration and managed according to standards (n=108 sites) * includes data from 28 sites in the 6 expansion regions 34 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 2004 Percentage 2005 2006 Nicaragua - Children with danger signs receiving care according to standards (n=13 sites) 0 20 40 60 80 100 2004 Percentage 2005 2006 2007 2008 2009 Nicaragua - Children with severe pneumonia diagnosis receiving care according to standards (n=13 sites) 0 20 40 60 80 100 2006 Percentage Nicaragua - Children with bronchial obstruction who received care according to standards (n=13 sites) 0 20 40 60 80 100 2004 Percentage 2005 2006 2007 2008 2009 Nicaragua - Children with diarrhea and dehydration receiving care according to standards (n=13 sites) 0 20 40 60 80 100 2004 Percentage Nicaragua - Children cared for in an integrated manner (n=13 sites) 0 20 40 60 80 100 2007 Percentage 2008 Niger - Deliveries with at least 80% of the post-partum surveillance standards respected [Wave 1] (n=29 sites) 0 20 40 60 80 100 2007 Percentage 2008 Niger - Deliveries with at least 80% of the post-partum surveillance standards respected [Wave 2] (n=10 sites) Results of Collaborative Improvement in 12 Countries 35 0 20 40 60 80 100 2006 Percentage 2007 2008 Niger - Deliveries with all three AMTSL elements were applied [Wave 1] (n=29 sites) 0 20 40 60 80 100 2007 Percentage 2008 Niger - Deliveries with all three AMTSL elements were applied [Wave 2] (n=10 sites) 0 4 8 12 16 20 2006 Percentage 2007 2008 Niger – Deliveries resulting in hemorrhage (Downward trend signals improvement) [Wave 1] (n=29 sites) 0 4 8 12 16 20 2007 Percentage 2008 Niger – Deliveries resulting in hemorrhage (Downward trend signals improvement) [Wave 2] (n=10 sites) 0 20 40 60 80 100 2006 Percentage 2007 2008 Niger - Newborns receiving immediate newborn care according to standards [Wave 1] (n=29 sites) 0 20 40 60 80 100 2007 Percentage 2008 Niger - Newborns receiving immediate newborn care according to standards [Wave 2] (n=10 sites) 0 20 40 60 80 100 2006 Percentage 2007 2008 Niger – Newborns immediately breast fed [Wave 1] (n=29 sites) 0 20 40 60 80 100 2007 Percentage 2008 Niger – Newborns immediately breast fed [Wave 2] (n=10 sites) 0 20 40 60 80 100 2006 Percentage 2007 2008 Niger - Newborns with temperature taken [Wave 1] (n=29 sites) 36 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 2003 Percentage 2004 2005 Niger – Children with danger signs receiving care according to standards (n=14 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Analyzed emergency triage and treatment (ETAT) forms for children 0–9 months with 80% compliance with ETAT norms [Wave 1] (n=14 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Analyzed emergency triage and treatment (ETAT) forms for children 0–9 months with 80% compliance with ETAT norms [Wave 2] (n=15 sites) 0 20 40 60 80 100 2003 Percentage 2004 2005 Niger - Children under 5 diagnosed with dehydration and managed according to standards (n=14 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Analyzed charts for children with dehydration with 80% compliance with case management norms [Wave 1] (n=14 sites) 0 20 40 60 80 100 2007 Percentage 2008 Niger - Newborns with temperature taken [Wave 2] (n=10 sites) 0 20 40 60 80 100 2003 Percentage 2004 2005 Niger - Children 0–59 months coming for consultations hospital who were triaged upon arrival [Wave 1] (n=14 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Children 0–59 months coming for consultations hospital who were triaged upon arrival [Wave 2] (n=15 sites) 0 20 40 60 80 100 2008 Percentage Niger - Pre-eclampsia and eclampsia case management standards complied with (n=31 sites) Results of Collaborative Improvement in 12 Countries 37 0 20 40 60 80 100 2003 Percentage 2004 2005 Niger - Children under 5 diagnosed with pneumonia managed according to standards (n=14 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Analyzed charts for children with pneumonia with 80% compliance with case management norms [Wave 1] (n=14 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Analyzed charts for children with pneumonia with 80% compliance with case management norms [Wave 2] (n=15 sites) 0 20 40 60 80 100 2006 Percentage 2007 2008 Niger - Average percent of standards for acute malnutrition recuperation complied with (n=15 sites) 0 20 40 60 80 100 2006 Percentage 2007 Niger - Analyzed charts for children with dehydration with 80% compliance with case management norms [Wave 2] (n=15 sites) 0 20 40 60 80 100 Percentage 2004 2005 0 20 40 60 80 100 Percentage 2006 2007 Niger - Children with malaria managed at hospital according to standards [Wave 1] (n=15 sites) Niger - Analyzed charts of children with malaria for which 80% of case management norms were complied with [Wave 1] (n=14 sites) 0 20 40 60 80 100 Percentage 2006 2007 Niger - Analyzed charts of children with malaria for which 80% of case management norms were complied with [Wave 2] (n=15 sites) 38 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 2000 Percentage 2001 2002 Russia - Women with pregnancy-induced hypertension delivering in month – (Downward trend signals improvement) [Wave 2] (n=40 sites) 0 20 40 60 80 100 1999 Percentage 2000 2001 Russia - Neonates arriving to the Neonatal Referral Center with hypothermia (Downward trend signals improvement) (n=5 sites) 0 6 12 18 24 30 1999 Percentage 2000 2001 Russia - Neonatal mortality due to respiratory distress in the first week of life (Downward trend signals improvement) (n=5 sites) 0 20 40 60 80 100 2000 Percentage 2001 2002 Russia - Neonates with respiratory distress who died in the first week of life (Downward trend signals improvement) (n=42 sites) 0 20 40 60 80 100 1999 Percentage 2000 2001 Russia - Women with pregnancy-induced hypertension delivering in month (Downward trend signals improvement) [Wave 1] (n=3 sites) 0 70 140 210 280 350 1998 1999 2000 2001 Percentage Russia - Number of hospitalizations due to pregnancy-induced hypertension over total deliveries (Downward trend signals improvement) (n=3 sites) Results of Collaborative Improvement in 12 Countries 39 0 20 40 60 80 100 2000 Percentage 2001 2002 2003 Russia - Newborns with respiratory distress who died in maternity hospitals (Downward trend signals improvement) (n=42 sites) 0 20 40 60 80 100 2006 Percentage Tanzania - Children managed according to case management guidelines (combined HIV, malaria and pneumonia) (n=5 sites) 0 20 40 60 80 100 2006 Percentage Tanzania - Children under 5 diagnosed with severe pneumonia managed according to standards (n=5 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 Tanzania - Children 0–59 months coming for consultations hospital who were triaged upon arrival (n=5 sites) 0 20 40 60 80 100 Percentage 2006 2007 Tanzania - HIV-positive children managed according to HIV case management guidelines (n=5 sites) 0 20 40 60 80 100 Percentage 2006 2007 Tanzania - Children with malaria managed at hospital according to standards (n=5 sites) 40 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 Percentage 2004 2005 2006 Rwanda - ARV patients who adhere to treatment at 95% (n=11 sites) HIV 0 20 40 60 80 100 Percentage 2005 2006 Rwanda - ARV patients with medical records completely filled out (n=10 sites) Rwanda - Women tested for HIV who returned for their results [Wave 1] (n=19 sites) 0 20 40 60 80 100 Percentage 2005 2006 Rwanda - Women tested for HIV who returned for their results [Wave 2] (n=17 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 Rwanda – Women testing positive for HIV whose partners are also tested [Wave 1] (n=19 sites) 0 20 40 60 80 100 Percentage 2005 2006 Rwanda – Women testing positive for HIV whose partners are also tested [Wave 2] (n=17 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 Results of Collaborative Improvement in 12 Countries 41 0 20 40 60 80 100 Percentage 2005 2006 Rwanda - children (born to HIV-positive mothers) tested at 15–18 months [Wave 1] (n=19 sites) 0 20 40 60 80 100 Percentage 2005 2006 Rwanda - children (born to HIV-positive mothers) tested at 15–18 months [Wave 2] (n=17 sites) 0 20 40 60 80 100 Percentage 2008 2009 Tanzania - Pregnant women in ANC who tested positive for HIV and enrolled in CTC (n=6 sites) 0 20 40 60 80 100 Percentage 2008 2009 Tanzania - HIV-exposed children under 18 months receiving daily Cotrimoxazole per month (n=6 sites) 0 4 8 12 16 20 Percentage 2008 2009 Tanzania - HIV-positive patients on ART lost to follow-up per month (Downward trend signals improvement) (n=6 sites) 0 20 40 60 80 100 Percentage 2008 2009 Tanzania - HIV-exposed infants receiving ARV prophylaxis (n=6 sites) 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 2009 Uganda - HIV-positive patients assessed for TB [Demonstration] (n=57 sites) 42 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 Percentage 2008 Uganda - Pregnant women in ANC who tested positive for HIV and enrolled in CTC [Wave 2] (n=31 sites) 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 Uganda - ARV patients who adhere to treatment at 95% [Demonstration] (n=57 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Uganda - HIV-positive patients assessed for TB [Wave 2] (n=31 sites) 0 20 40 60 80 100 Percentage 2006 2007 2008 2009 Uganda - HIV-positive patients assessed for TB [Wave 1] (n=32 sites) 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 Uganda - Pregnant women in ANC who tested positive for HIV and enrolled in CTC [Demonstration] (n=57 sites) 0 20 40 60 80 100 Percentage 2007 2008 Uganda - Pregnant women in ANC who tested positive for HIV and enrolled in CTC [Wave 1] (n=32 sites) Results of Collaborative Improvement in 12 Countries 43 0 20 40 60 80 100 Percentage 2008 Uganda - ARV patients who adhere to treatment at 95% [Wave 2] (n=31 sites) 0 20 40 60 80 100 Percentage 2006 2007 2008 Uganda - ARV patients who adhere to treatment at 95% [Wave 1] (n=32 sites) 0 20 40 60 80 100 Percentage 2005 2006 2007 2008 Uganda – ARV patients with clinical improvement [Demonstration] (n=7 sites) 0 20 40 60 80 100 Percentage 2006 2007 2008 Uganda – ARV patients with clinical improvement [Wave 1] (n=12 sites) 0 20 40 60 80 100 Percentage 2007 2008 2009 Uganda – ARV patients with clinical improvement [Wave 2] (n=25 sites) 44 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 2008 2009 Percentage 0 20 40 60 80 100 2008 2009 Percentage 0 20 40 60 80 100 2008 2009 Percentage Benin - Patients whose RDT was performed correctly (n=15 sites) Benin - Children under 5 with fever whose weight was registered (n=15 sites) Benin - Children under 5 with fever treated correctly (n=15 sites) 0 20 40 60 80 100 2008 2009 Percentage Benin - Adults with a positive RDT treated correctly (n=15 sites) MALARIA 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 Rwanda - Children under 5 with fever taken to health center within 24 hours of onset [Wave 1] (n=20 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 Rwanda - Children under 5 with fever treated in compliance with standards at health center [Wave 1] (n=20 sites) 0 20 40 60 80 100 Percentage 2005 2006 Rwanda - Children under 5 with fever taken to health center within 24 hours of onset [Wave 2] (n=28 sites) Results of Collaborative Improvement in 12 Countries 45 0 20 40 60 80 100 Percentage 2005 2006 Rwanda - Children under 5 with fever treated in compliance with standards at health center [Wave 2] (n=28 sites) 0 20 40 60 80 100 Percentage 2003 2004 2005 2006 Rwanda - Children under 5 with fever treated in compliance with standards at hospital level (n=3 sites) 46 Results of Collaborative Improvement in 12 Countries 0 20 40 60 80 100 Percentage 2006 2007 2008 2009 Vietnam - TB patients counseled on HIV (n=13 sites) 0 20 40 60 80 100 Percentage 2006 2007 2008 2009 Vietnam - TB patients tested for HIV (n=13 sites) 0 20 40 60 80 100 2007 2008 Percentage 0 20 40 60 80 100 2007 2008 Percentage Bolivia - New pulmonary TB cases de￾tected over expected (n=114 sites) Bolivia - TB patients receiving DOTS (n=114 sites) 0 20 40 60 80 100 2007 2008 Percentage Bolivia - Cure rate among TB cases programmed to end treatment (n=114 sites) 0 10 20 30 40 50 2007 2008 Percentage Bolivia - Sputum samples with saliva (Downward trend signals improvement) (n=114 sites) TUBERCULOSIS Results of Collaborative Improvement in 12 Countries 47 FAMILY PLANNING 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients screened for high blood pressure (n=9 sites) 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients with urine tested (n=9 sites) 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients weighed (n=9 sites) 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients screened for thyroid, lumps (n=9 sites) 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients with abdominal exam (n=9 sites) 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients with vaginal exam (n=9 sites) 0 20 40 60 80 100 Percentage 2005 2006 Tanzania - Family planning clients with date for next visit registered (n=9 sites) 0 20 40 60 80 100 2005 Percentage 2006 2007 2008 Honduras - Users of post-obstetric care who received family planning counseling (n=108 sites) 48 Results of Collaborative Improvement in 12 Countries OTHER 0 2 4 6 8 10 Percentage 1999 2000 2001 Russia - Hospitalization due to arterial hypertension of those diagnosed with arterial hypertension (Downward trend signals improvement) [Wave 1] (n=5 sites) 0 2 4 6 8 10 Percentage 2001 2002 Russia - Hospitalization due to arterial hypertension of those diagnosed with arterial hypertension (Downward trend signals improvement) [Wave 2] (n=442 sites) 0 4 8 12 16 20 1999 2000 2001 Percentage Russia - Persons diagnosed with arterial hypertension as percentage of population (n=5 sites) 0 2 4 6 8 10 Percentage 2001 2002 Russia - Hypertensive crises in those diagnosed with arterial hypertension (Downward trend signals improvement) [Wave 2] (n=442 sites) USAID HEALTH CARE IMPROVEMENT PROJECT University Research Co., LLC 7200 Wisconsin Avenue, Suite 600 Bethesda, MD 20814 Tel: (301) 654-8338 Fax: (301) 941-8427 www.hciproject.org