User login
The Learning Hospital System
In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.
Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.
Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.
THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM
Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]
The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]
Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.
THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS
The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]
Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]
AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE
Translating Science to Evidence in Sepsis
Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]
Translating Evidence to Care in Sepsis at KPNC
In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).
| Time Period | Event Summary |
|---|---|
| |
| 2007 | Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes. |
| Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009). | |
| Spring 2008 | Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes. |
| May 2008 | Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement. |
| Summer 2008 | Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots. |
| Fall 2008 | Pilot intervention deployed at 2 medical centers. |
| November 2008 | First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy. |
| November 2008 | All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians. |
| January 2009 | Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value. |
| Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure. | |
| March 2009 | Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits. |
| August 2009 | Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development. |
| November 2009 | Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality. |
| January 2010 | Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest. |
| 2010 | Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation. |
| Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics. | |
| April 2011 | Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock. |
| May 2012 | Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins. |
| February 2013 | Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements. |
| May 2013 | Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality |
| March 2014 | Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners. |
| October 2014 | Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients. |
| October 2015 | Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines. |
Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.
The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]
CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS
Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?
Confirming Concordance in the Impact of Sepsis Nationally
The heightened identification of sepsis patients through program implementation revealed that the impact of sepsis on hospital mortality was greater than originally estimated; based on improved patient identification, sepsis now accounted for upward of 1 in 2 hospital deaths.[25] This sobering statistic confirmed that the investments in standardizing best sepsis care following the mortality diagnostic were critical. However, were similar estimates of sepsis‐attributable mortality consistent outside of the KPNC system? To study this, we examined US hospitalizations occurring across >1000 hospitals and involving >6 million hospital stays to estimate corresponding prevalence.[25] In this national sample, sepsis contributed to as many as half of hospital deaths in the United States in 2010, lending strong support to ongoing international and state‐based efforts to improve sepsis care. These studies also paved the way to use these data drawn from our large sepsis population to inform updated international consensus definitions for sepsis and septic shock.[26, 27, 28]
Identifying New Avenues for Reducing the Toll of Sepsis
A major concern of sepsis program leaders was the prior findings that sepsis hospitalizations among Medicare beneficiaries were associated with substantial new cognitive and functional disability.[29] This lingering toll of sepsis had been termed a hidden public health disaster.[30] To further understand the posthospital impact of sepsis and to begin investigating new avenues to reduce this impact, a cohort of patients was followed for 1 year following sepsis hospitalization.[31] Over that period, nearly half of sepsis survivors were rehospitalized. When compared with their presepsis levels of healthcare utilization, middle‐aged and elderly sepsis patients experienced a 3‐fold increase in their days using facility‐based care. Subsequent studies in other populations outside of KPNC have confirmed these concerning findings, resulting in new efforts to address postsepsis survivorship care.[32, 33]
Phenotyping New Targets for Standardized Sepsis Care
At its outset, the sepsis improvement program applied the best available evidence to treat patients with the most severe forms of sepsisseptic shock. However, once the initial implementation phase had succeeded, clinicians and operational leaders quickly realized from the emerging data that there was a far larger group of sepsis patients for whom treatment guidelines were poorly defined.[25, 34, 35] These were severe sepsis patients with so‐called intermediate lactate values between 2 mmol/L and 4 mmol/L; they comprised a substantial proportion of all sepsis patients dying in the hospital. Using data generated from the routine care of sepsis patients treated across 21 hospitals, the sepsis leadership group was able to rapidly assemble a cohort of intermediate lactate sepsis patients up to 20‐ to 100‐fold larger than that reported in prior studies and evaluate their outcomes.[34, 35]
The data used to evaluate these intermediate lactate sepsis patients now spurred a new implementation program in 2013 for a group of patients in whom there was essentially no existing evidence to guide care. Rapidly implemented within a mature sepsis performance improvement program, evaluations at the 6‐month and 1‐year intervals demonstrated significant decreases in mortality.[36] Importantly, to allay the justified concerns of clinicians, these evaluations also clearly showed no evidence of harm from more aggressive fluid resuscitation (eg, increased transfer to intensive care, increased rates of mechanical ventilation). Again, driven by clinician input, subgroup analyses further revealed that the implementation program was only associated with reduced mortality in patients who could be at risk for iatrogenic fluid overload (ie, those with a history of congestive heart failure or chronic kidney disease).[36] Spurred by these provocative findings, operational and clinical leaders are currently considering how to guide future care in these patients, especially with the emerging use of noninvasive methods to quantify patients' fluid responsiveness.
PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM
The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.
Building a Digital Infrastructure for Utilizing Granular Hospital Data
As described above, current transitions between the nodes of the learning framework are rife with missed opportunities. Perhaps one of the most glaring is the inability to use highly granular data already collected within the electronic health record (eg, trajectories and trends across vital signs or laboratory results, large‐scale medication administration records to evaluate multidrug interactions). An essential starting point for continuous learning is investing in the digital infrastructure to improve the use of data beyond traditional claims (administrative dataadmission source codes, disposition codes, diagnoses, and procedures). As shown in Table 2, the first key step is incorporating laboratory data into the quality assessment/emmprovement process. In addition, using these data to automate severity of illness and risk adjustment metrics fosters use of similar comparison cohorts across time or disease types.[18, 37, 38, 39, 40]
| Data Type | Contents | Degree of Difficulty in Accessing | Degree of Difficulty in Analyzing |
|---|---|---|---|
| Administrative | Traditional claims data, diagnostic or procedural codes | Low | Low to moderate |
| Standard cohort profiling | Limited instances of vitals signs, laboratory, diagnostic testing, or treatment data | Low to moderate | Low to moderate |
| Metrics reporting for care improvement | Standard cohort identification, aggregated achievement of treatment targets, scorecard dissemination | Moderate | Moderate |
| Advanced cohort profiling | Time series of physiologic data, inpatient triage and treatment data within short temporal intervals | Moderate to high | High |
| Research‐grade discovery | Data with breadth (representative sample size) and depth (highly granular physiologic and treatment data) | High | Very high |
| Patient‐reported outcomes | Quality of life, functional and cognitive disability | Very high | High |
Employing Novel Methods to Address the Limitations of Using Real‐World Data
The rapid digitization of medicine through the use of electronic medical records offers tremendous opportunities to facilitate continuous learning. However, these opportunities are accompanied by important limitations.[41] Data collected as a byproduct of real‐world care can be vulnerable to many forms of bias and confounding, potentially clouding the validity and robustness of corresponding analytic results. Fortunately, advanced methods including causal inference are now used routinely to address some limitations.[42] In the context of a learning healthcare system, other opportunities for improved study design including cluster randomized trials or stepped wedge implementation can also be employed to preserve the statistical rigor of subsequent analyses.[43] Finally, emerging methods employing randomization through the electronic medical record alongside adaptive trial design offer great potential to increase the efficiency of continuous learning.[44]
Evaluating the Hospital as a Single System
Advances in contemporary hospital care require seamless transitions of patient care, screening strategies, and therapeutic approaches across multiple hospital domains and with diverse providers; these interventions also need to happen rapidly. Many traditional approaches to inpatient care have taken a bottom‐up approach (eg, studying a specific disease within a specific hospital ward like the intensive care unit) that have proven useful but may limit generalizability when applied to a real‐world hospital operating with Pareto optimality (ie, the trade‐off scenario where new resource allocation to 1 area also requires resource withdrawal from another area). In certain cases, an empiric approach, without initial preference for any specific ward or disease, can aid decision making by hospital operational and clinical leaders by providing a global picture of impact and value.
Focusing on Early Detection in Hospital Settings as Secondary Prevention
Once patients have been admitted to the hospital, a race against the clock begins. Each additional hour of hospitalization increases the risks of iatrogenic injury or medical harm manifested by immobility, disorientation and delirium, nosocomial infections, or medication errors, among others. In this context, detection systems that use granular hospital data to focus on the earliest detection of risk can aid critical approaches to secondary prevention (Although the hospitalization for sepsis cannot be avoided, careful attention to mobility can limit the risk of developing delirium. In turn, preventing delirium can limit the risk of new functional disability).
Contextualizing Hospital Care Within a Longitudinal Trajectory
Although we described the benefit of hospital episodes having well‐demarcated beginning and ending points, it remains essential to recognize that the harms associated with hospitalization extend well beyond discharge. In this context, hospitalizations can serve as waypoints in patients' health trajectories as well as an opportunity to achieve patient‐centered care including discussing and aligning goals of care with actual care provision. Furthermore, although we have seen steady declines in hospital mortality over time, it is highly likely that we will reach a nadir in mortality where additional metrics of hospital outcomes will need to include postdischarge events like readmission, long‐term mortality, quality of life, and the prevention of disability or decline.
CONCLUSION
Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.
Disclosures
As part of our agreement with the Gordon and Betty Moore Foundation, we made a commitment to disseminate our findings in articles such as this one. However, the Foundation and its staff played no role in how we actually structured our articles, nor did they review or preapprove any of the manuscripts submitted as part of the dissemination component. None of the authors has any conflicts of interest to declare of relevance to this work, which was funded by a combination of funding from the Gordon and Betty Moore Foundation, The Permanente Medical Group, Inc., and Kaiser Foundation Hospitals, Inc. VXL was supported by NIH K23GM112018.
- Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press; 2012.
- , , , et al. Toward a science of learning systems: a research agenda for the high‐functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43–50.
- National Center for Health Statistics. Health, United States, 2014: With Special Feature on Adults Aged 55–64. Hyattsville, MD; 2015.
- , . Critical care medicine in the United States 2000‐2005: an analysis of bed numbers, occupancy rates, payer mix, and costs. Crit Care Med. 2010;38(1):65–71.
- , , , . Trends and variation in end‐of‐life care for medicare beneficiaries with severe chronic illness. A report of the Dartmouth Atlas Project. Lebanon, NH: The Dartmouth Institute for Health Policy and Clinical Practice; 2011.
- , , , et al. Change in end‐of‐life care for Medicare beneficiaries: site of death, place of care, and health care transitions in 2000, 2005, and 2009. JAMA. 2013;309(5):470–477.
- , , . Finding the missing link for big biomedical data. JAMA. 2014;311(24):2479–2480.
- . Code red and blue—safely limiting health care's GDP footprint. N Engl J Med. 2013;368(1):1–3.
- , . The inevitable application of big data to health care. JAMA. 2013;309(13):1351–1352.
- , . What is citizen science?—a scientometric meta‐analysis. PLoS One. 2016;11(1):e0147152.
- , , . Biomedical ontologies: a functional perspective. Brief Bioinform. 2008;9(1):75–90.
- . Rapid learning: a breakthrough agenda. Health Aff (Millwood). 2014;33(7):1155–1162.
- , , , et al. Are regional variations in end‐of‐life care intensity explained by patient preferences?: a study of the US Medicare population. Med Care. 2007;45(5):386–393.
- , . Extreme markup: the fifty US hospitals with the highest charge‐to‐cost ratios. Health Aff (Millwood). 2015;34(6):922–928.
- , , , . The price ain't right? Hospital prices and health spending on the privately insured. Health Care Pricing Project website. Available at: http://www.healthcarepricingproject.org/sites/default/files/pricing_variation_manuscript_0.pdf. Accessed February 15, 2016
- . A new, evidence‐based estimate of patient harms associated with hospital care. J Patient Saf. 2013;9(3):122–128.
- , , . Hospitalization‐associated disability: “she was probably able to ambulate, but I'm not sure”. JAMA. 2011;306(16):1782–1793.
- , , , , . Risk‐adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5):446–453.
- . New definitions for sepsis and septic shock: continuing evolution but with much still to be done. JAMA. 2016;315(8):757–759.
- , . Severe sepsis and septic shock. N Engl J Med. 2013;369(21):2063.
- , , . Sepsis‐induced immunosuppression: from cellular dysfunctions to immunotherapy. Nat Rev Immunol. 2013;13(12):862–874.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
- , , , et al. Kaiser Permanente's performance improvement system, part 3: multisite improvements in care for patients with sepsis. Jt Comm J Qual Patient Saf. 2011;37(11):483–493.
- , , , , . Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91(2):354–394.
- , , , et al. Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA. 2014;312(1):90–92.
- , , , et al. Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):762–774.
- , , , et al. Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):775–787.
- , , , et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):801–810.
- , , , . Long‐term cognitive impairment and functional disability among survivors of severe sepsis. JAMA. 2010;304(16):1787–1794.
- . The lingering consequences of sepsis: a hidden public health disaster? JAMA. 2010;304(16):1833–1834.
- , , , , , . Hospital readmission and healthcare utilization following sepsis in community settings. J Hosp Med. 2014;9(8):502–507.
- , , , , . Increased 1‐year healthcare use in survivors of severe sepsis. Am J Respir Crit Care Med. 2014;190(1):62–69.
- , , , et al. Post‐acute care use and hospital readmission after sepsis. Ann Am Thorac Soc. 2015;12(6):904–913.
- , , , , . Fluid volume, lactate values, and mortality in sepsis patients with intermediate lactate values. Ann Am Thorac Soc. 2013;10(5):466–473.
- , , . Prognosis of emergency department patients with suspected infection and intermediate lactate levels: a systematic review. J Crit Care. 2014;29(3):334–339.
- , , , et al. Multicenter implementation of a treatment bundle for sepsis patients with intermediate lactate values. Am J Respir Crit Care Med. 2016;193(11):1264–1270.
- , , , et al. Risk adjusting community‐acquired pneumonia hospital outcomes using automated databases. Am J Manag Care. 2008;14(3):158–166.
- , , , , , . Intra‐hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS). J Hosp Med. 2011;6(2):74–80.
- , , , , , . Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7(5):388–395.
- , , , , . An electronic Simplified Acute Physiology Score‐based risk adjustment score for critical illness in an integrated healthcare system. Crit Care Med. 2013;41(1):41–48.
- . Learning from big health care data. N Engl J Med. 2014;370(23):2161–2163.
- , . Getting the methods right—the foundation of patient‐centered outcomes research. N Engl J Med. 2012;367(9):787–790.
- , , , , . The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.
- . Fusing randomized trials with big data: the key to self‐learning health care systems? JAMA. 2015;314(8):767–768.
In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.
Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.
Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.
THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM
Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]
The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]
Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.
THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS
The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]
Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]
AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE
Translating Science to Evidence in Sepsis
Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]
Translating Evidence to Care in Sepsis at KPNC
In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).
| Time Period | Event Summary |
|---|---|
| |
| 2007 | Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes. |
| Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009). | |
| Spring 2008 | Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes. |
| May 2008 | Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement. |
| Summer 2008 | Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots. |
| Fall 2008 | Pilot intervention deployed at 2 medical centers. |
| November 2008 | First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy. |
| November 2008 | All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians. |
| January 2009 | Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value. |
| Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure. | |
| March 2009 | Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits. |
| August 2009 | Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development. |
| November 2009 | Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality. |
| January 2010 | Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest. |
| 2010 | Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation. |
| Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics. | |
| April 2011 | Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock. |
| May 2012 | Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins. |
| February 2013 | Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements. |
| May 2013 | Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality |
| March 2014 | Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners. |
| October 2014 | Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients. |
| October 2015 | Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines. |
Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.
The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]
CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS
Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?
Confirming Concordance in the Impact of Sepsis Nationally
The heightened identification of sepsis patients through program implementation revealed that the impact of sepsis on hospital mortality was greater than originally estimated; based on improved patient identification, sepsis now accounted for upward of 1 in 2 hospital deaths.[25] This sobering statistic confirmed that the investments in standardizing best sepsis care following the mortality diagnostic were critical. However, were similar estimates of sepsis‐attributable mortality consistent outside of the KPNC system? To study this, we examined US hospitalizations occurring across >1000 hospitals and involving >6 million hospital stays to estimate corresponding prevalence.[25] In this national sample, sepsis contributed to as many as half of hospital deaths in the United States in 2010, lending strong support to ongoing international and state‐based efforts to improve sepsis care. These studies also paved the way to use these data drawn from our large sepsis population to inform updated international consensus definitions for sepsis and septic shock.[26, 27, 28]
Identifying New Avenues for Reducing the Toll of Sepsis
A major concern of sepsis program leaders was the prior findings that sepsis hospitalizations among Medicare beneficiaries were associated with substantial new cognitive and functional disability.[29] This lingering toll of sepsis had been termed a hidden public health disaster.[30] To further understand the posthospital impact of sepsis and to begin investigating new avenues to reduce this impact, a cohort of patients was followed for 1 year following sepsis hospitalization.[31] Over that period, nearly half of sepsis survivors were rehospitalized. When compared with their presepsis levels of healthcare utilization, middle‐aged and elderly sepsis patients experienced a 3‐fold increase in their days using facility‐based care. Subsequent studies in other populations outside of KPNC have confirmed these concerning findings, resulting in new efforts to address postsepsis survivorship care.[32, 33]
Phenotyping New Targets for Standardized Sepsis Care
At its outset, the sepsis improvement program applied the best available evidence to treat patients with the most severe forms of sepsisseptic shock. However, once the initial implementation phase had succeeded, clinicians and operational leaders quickly realized from the emerging data that there was a far larger group of sepsis patients for whom treatment guidelines were poorly defined.[25, 34, 35] These were severe sepsis patients with so‐called intermediate lactate values between 2 mmol/L and 4 mmol/L; they comprised a substantial proportion of all sepsis patients dying in the hospital. Using data generated from the routine care of sepsis patients treated across 21 hospitals, the sepsis leadership group was able to rapidly assemble a cohort of intermediate lactate sepsis patients up to 20‐ to 100‐fold larger than that reported in prior studies and evaluate their outcomes.[34, 35]
The data used to evaluate these intermediate lactate sepsis patients now spurred a new implementation program in 2013 for a group of patients in whom there was essentially no existing evidence to guide care. Rapidly implemented within a mature sepsis performance improvement program, evaluations at the 6‐month and 1‐year intervals demonstrated significant decreases in mortality.[36] Importantly, to allay the justified concerns of clinicians, these evaluations also clearly showed no evidence of harm from more aggressive fluid resuscitation (eg, increased transfer to intensive care, increased rates of mechanical ventilation). Again, driven by clinician input, subgroup analyses further revealed that the implementation program was only associated with reduced mortality in patients who could be at risk for iatrogenic fluid overload (ie, those with a history of congestive heart failure or chronic kidney disease).[36] Spurred by these provocative findings, operational and clinical leaders are currently considering how to guide future care in these patients, especially with the emerging use of noninvasive methods to quantify patients' fluid responsiveness.
PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM
The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.
Building a Digital Infrastructure for Utilizing Granular Hospital Data
As described above, current transitions between the nodes of the learning framework are rife with missed opportunities. Perhaps one of the most glaring is the inability to use highly granular data already collected within the electronic health record (eg, trajectories and trends across vital signs or laboratory results, large‐scale medication administration records to evaluate multidrug interactions). An essential starting point for continuous learning is investing in the digital infrastructure to improve the use of data beyond traditional claims (administrative dataadmission source codes, disposition codes, diagnoses, and procedures). As shown in Table 2, the first key step is incorporating laboratory data into the quality assessment/emmprovement process. In addition, using these data to automate severity of illness and risk adjustment metrics fosters use of similar comparison cohorts across time or disease types.[18, 37, 38, 39, 40]
| Data Type | Contents | Degree of Difficulty in Accessing | Degree of Difficulty in Analyzing |
|---|---|---|---|
| Administrative | Traditional claims data, diagnostic or procedural codes | Low | Low to moderate |
| Standard cohort profiling | Limited instances of vitals signs, laboratory, diagnostic testing, or treatment data | Low to moderate | Low to moderate |
| Metrics reporting for care improvement | Standard cohort identification, aggregated achievement of treatment targets, scorecard dissemination | Moderate | Moderate |
| Advanced cohort profiling | Time series of physiologic data, inpatient triage and treatment data within short temporal intervals | Moderate to high | High |
| Research‐grade discovery | Data with breadth (representative sample size) and depth (highly granular physiologic and treatment data) | High | Very high |
| Patient‐reported outcomes | Quality of life, functional and cognitive disability | Very high | High |
Employing Novel Methods to Address the Limitations of Using Real‐World Data
The rapid digitization of medicine through the use of electronic medical records offers tremendous opportunities to facilitate continuous learning. However, these opportunities are accompanied by important limitations.[41] Data collected as a byproduct of real‐world care can be vulnerable to many forms of bias and confounding, potentially clouding the validity and robustness of corresponding analytic results. Fortunately, advanced methods including causal inference are now used routinely to address some limitations.[42] In the context of a learning healthcare system, other opportunities for improved study design including cluster randomized trials or stepped wedge implementation can also be employed to preserve the statistical rigor of subsequent analyses.[43] Finally, emerging methods employing randomization through the electronic medical record alongside adaptive trial design offer great potential to increase the efficiency of continuous learning.[44]
Evaluating the Hospital as a Single System
Advances in contemporary hospital care require seamless transitions of patient care, screening strategies, and therapeutic approaches across multiple hospital domains and with diverse providers; these interventions also need to happen rapidly. Many traditional approaches to inpatient care have taken a bottom‐up approach (eg, studying a specific disease within a specific hospital ward like the intensive care unit) that have proven useful but may limit generalizability when applied to a real‐world hospital operating with Pareto optimality (ie, the trade‐off scenario where new resource allocation to 1 area also requires resource withdrawal from another area). In certain cases, an empiric approach, without initial preference for any specific ward or disease, can aid decision making by hospital operational and clinical leaders by providing a global picture of impact and value.
Focusing on Early Detection in Hospital Settings as Secondary Prevention
Once patients have been admitted to the hospital, a race against the clock begins. Each additional hour of hospitalization increases the risks of iatrogenic injury or medical harm manifested by immobility, disorientation and delirium, nosocomial infections, or medication errors, among others. In this context, detection systems that use granular hospital data to focus on the earliest detection of risk can aid critical approaches to secondary prevention (Although the hospitalization for sepsis cannot be avoided, careful attention to mobility can limit the risk of developing delirium. In turn, preventing delirium can limit the risk of new functional disability).
Contextualizing Hospital Care Within a Longitudinal Trajectory
Although we described the benefit of hospital episodes having well‐demarcated beginning and ending points, it remains essential to recognize that the harms associated with hospitalization extend well beyond discharge. In this context, hospitalizations can serve as waypoints in patients' health trajectories as well as an opportunity to achieve patient‐centered care including discussing and aligning goals of care with actual care provision. Furthermore, although we have seen steady declines in hospital mortality over time, it is highly likely that we will reach a nadir in mortality where additional metrics of hospital outcomes will need to include postdischarge events like readmission, long‐term mortality, quality of life, and the prevention of disability or decline.
CONCLUSION
Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.
Disclosures
As part of our agreement with the Gordon and Betty Moore Foundation, we made a commitment to disseminate our findings in articles such as this one. However, the Foundation and its staff played no role in how we actually structured our articles, nor did they review or preapprove any of the manuscripts submitted as part of the dissemination component. None of the authors has any conflicts of interest to declare of relevance to this work, which was funded by a combination of funding from the Gordon and Betty Moore Foundation, The Permanente Medical Group, Inc., and Kaiser Foundation Hospitals, Inc. VXL was supported by NIH K23GM112018.
In the landmark Best Care at Lower Cost report, the Institute of Medicine presents a compelling vision of a US healthcare system where science, information technology, incentives, and care culture are brought together seamlessly to produce high‐quality healthcare.[1] At the center of this transformation is the learning healthcare system, a system characterized by its ability to leverage data arising from care provision to drive rapid improvements in care delivery.[2] When steeped within the right organizational milieu, these data help to close the virtuous cycle of continuous learning moving from science to evidence to care and back to new science. The anticipated end result is a healthcare system that can provide Americans with superior care at lower cost.
Hospital‐based practitioners will recognize the inpatient setting as an ideal demonstration opportunity for continuous learning. Hospital care is costly, accounting for more than 30% of all US healthcare costs[3]; intensive care alone accounts for a notable proportion of the US gross domestic product.[4] Inpatient care is associated with significant mortality and morbidity, and its use is often greatly increased in patients' last days.[5, 6] Fortunately, the inpatient setting also offers an ideal opportunity to leverage high‐quality data to help inform and improve care. The digitization of medicine means that far more data are now available through electronic health records, medical devices, and tests.[7] This is particularly true for inpatients, for whom a large volume of data are produced even over relatively short hospital stays.
Whereas the challenge to improve hospital care is daunting, there is an incredible opportunity to advance the quality of inpatient care through realizing the vision of the learning hospital system. In the sections that follow, we use an object lessonsepsis care within hospitals of the Kaiser Permanente Northern California (KPNC) integrated healthcare delivery systemto evaluate the challenges and insights gleaned from working toward building a learning hospital system. Then, we describe further steps that could enhance the use of inpatient data to drive improved care.
THE FRAMEWORK OF A LEARNING HEALTHCARE SYSTEM
Best Care at Lower Cost notes a fundamental paradox in US healthcare: although we have witnessed a dramatic expansion in biomedical knowledge, innovative therapies and surgical procedures, and clinical treatments to extend survival, US healthcare persistently falls short on the basic dimensions of quality, outcomes, cost, and equity.[1] The proposed path forward lies in building the learning healthcare system, a system characterized by continuous knowledge development, improvement, and application. Figure 1 shows the critical nodes in the framework for continuous learning, which include: (1) the development of new scientific knowledge (science), (2) the translation of science into clinical evidence of efficacy (evidence), and (3) the application of efficacious interventions through effective care delivery (care). In healthcare today, transitions between these nodes are rife with missed or wasted opportunities like delays in applying high‐quality evidence or poorly managed insights arising from scientific discovery. If such opportunities could be recovered, however, the quality of healthcare could be improved dramatically.[8]
The pursuit of continuous learning is aided by rapid changes in the quality and quantity of biomedical data available over the past decade, especially through the use of electronic health records, novel biomolecular tools, and digital sensors.[2, 7, 9] The Internet has ushered in a new era of data connectivity, for example, allowing for highly engaged communication between patients and providers as well as collaboration between professional or citizen scientists on data of unprecedented scale.[10] New methodologic approaches, including data mining and machine learning, increasingly leverage commodity hardware to conduct previously computationally intractable analyses.[9] Moreover, the development of domain ontologies fosters the discovery of meaningful insights from data of heterogeneous types.[11]
Ultimately, however, improvements in data alone are inadequate to achieve continuous learning. As shown in Figure 1, whereas data form the channels that allow for transitions from science to evidence to care, novel insights need to be steeped within the right culture, motivated by the right incentives, and supported by the right leaders.[1, 12] Within the sustainable learning healthcare system, knowledge generation feeds practice change with the support and guidance of system leadership; improved practice, in turn, generates new knowledge and completes the virtuous cycle of learning.
THE PROMISE OF CONTINUOUS LEARNING IN HOSPITAL SETTINGS
The hospital is an ideal setting in which to foster continuous learning because advances in inpatient care have the potential to substantially improve healthcare quality and value.[8] Americans were hospitalized roughly 37 million times in 2012; in total, these episodes cost $378 billion.[3] Over 700,000 patients die in US hospitals annually, with reports showing that many patients utilize greatly increased inpatient and critical care services near the end of their lives in a manner that appears misaligned with their preferences.[11, 13] Hospital care is also highly variable in quality and cost; this heterogeneity is not closely associated with improved outcomes.[14, 15] Preventable harm and medical injury occur commonly in hospitals and are now recognized to be a leading cause of inpatient death.[16] Finally, emerging research illuminates the substantial toll that acute care has on patients and families resulting in new comorbidity, functional or neuropsychiatric impairment, rehospitalization, and financial burden that persist long after patients are discharged.[17]
Fortunately, inpatient care also exhibits several qualities that improve the likelihood that continuous learning can be achieved. Although it is clear that hospitalizations occur within the arc of a patient's larger health trajectory, these distinct episodes offer the potential to observe patient trajectories and treatments evolving within relatively compressed time intervals; over that same interval, a large volume of data are produced. Stored within comprehensive electronic health records, these granular data now allow inpatient episodes to be digitally recapitulated with high fidelity, bolstering their use in driving care improvements.[18]
AN OBJECT LESSON IN THE LEARNING FRAMEWORK: SEPSIS CARE
Translating Science to Evidence in Sepsis
Although sepsis has attracted great attention in modern hospital care, sepsis was described long ago by Hippocrates to describe the process by which wounds fester.[19] Recast after the confirmation of germ theory, sepsis came to be known primarily as the blood poisoning resulting from pathogenic organisms.[20] However, with the advent of antibiotics, numerous scientific studies now recognize that sepsis actually results from the dysregulated host immune response to systemic infection, which can also cause organ dysfunction.[21] Based on this knowledge, landmark translational and clinical studies in the 2000s provided strong evidence that early identification of sepsis patients and aggressive infection control and resuscitation were associated with improved mortality (Figure 2, step 1).[22]
Translating Evidence to Care in Sepsis at KPNC
In 2007, the leadership of KPNC initiated a regional effort to improve the quality of care and reduce the variability in performance at its medical centers (Table 1).[23] Reviewing data from nearly 1000 inpatientsthe last 50 consecutive hospital deaths from each of 19 medical centersa mortality diagnostic based on Institute for Healthcare Improvement recommendations[24] revealed that sepsis had a major impact on hospital outcomes. For example, even though sepsis patients were still relatively under‐recognized at the time, accounting for fewer than 3% of hospitalizations, they contributed to one‐quarter of hospital deaths. In light of these compelling data, senior regional leadership identified reducing sepsis mortality as a key performance improvement goal (Figure 2, step 2).
| Time Period | Event Summary |
|---|---|
| |
| 2007 | Variability in hospital standardized mortality ratio observed, indicating an opportunity to drive improved outcomes. |
| Initiation of staggered implementation of unified electronic medical record across all KP sites (starting in 2006 and ending in 2009). | |
| Spring 2008 | Mortality diagnostic chart review completed identifying sepsis and infection‐related causes as key factors in hospital outcomes. |
| May 2008 | Regional Mortality Summit held with a focus on patient safety and mortality reduction efforts through performance improvement. Executive regional and local leadership alignment to focus on sepsis performance improvement. |
| Summer 2008 | Sepsis Steering Committee evaluates best available evidence, develops treatment algorithms, and plans for medical center pilots. |
| Fall 2008 | Pilot intervention deployed at 2 medical centers. |
| November 2008 | First Regional Sepsis Summit: development of sepsis performance improvement playbook, training materials, implementation plans, and measurement strategy. |
| November 2008 | All medical centers begin to form multidisciplinary sepsis teams and performance improvement committees, obtain equipment and supplies including assembly of a sepsis cart. Multidisciplinary teams included ED physician champion, ED nurse champion, improvement advisor, hospitalists, intensivists, quality improvement personnel, nurse educators, and even resident physicians. |
| January 2009 | Performance data collection begins on EGDT processes and outcomes. Initiation of 2 key elements to enhance screening for and detection of sepsis: (1) concomitant ordering of serum lactic acid along with blood cultures, and (2) definition of lactate >2.0 as a critical lab value. |
| Use of manual chart review for case finding and central database entry because of ongoing implementation of electronic medical record and limited sepsis‐specific data infrastructure. | |
| March 2009 | Regional train the trainer sessions occur and local educational spread efforts begin including: collaborative calls, in‐person training events, and medical center site visits. |
| August 2009 | Grant funding from the Gordon and Betty Moore Foundation begins with a planned 2‐year duration providing funding for improvement advisors with performance improvement expertise and data infrastructure development. |
| November 2009 | Second Regional Sepsis Summit. Identification of intermediate lactate sepsis patients having significant mortality. |
| January 2010 | Initiate measurement of performance for intermediate lactate sepsis patients with a focus on lactate clearance as an outcome measure of interest. |
| 2010 | Development of an intranet Web‐based data abstraction tool to identify cases and auto‐populate specific fields for review. Facilities were responsible for review of cases at the local level to foster rapid feedback cycles for local performance improvement. Standardized data query tools were deployed to foster local medical center engagement and system‐level evaluation. |
| Accompanying development of a sepsis performance improvement scorecard allowing for comparison of longitudinal performance metrics across all facilities. Scorecard elements included: proportion of lactates drawn following ED blood culture, EGDT‐specific bundle elements (ie, number of EGDT cases, antibiotics within 1 hour, first central venous pressure within 2 hours of EGDT start, target mean arterial pressure achievement), repeat lactate elements, balancing measures for central line placement (ie, pneumothorax, central line infection), and overall sepsis statistics. | |
| April 2011 | Third Regional Sepsis Summit. Refinement of EGDT bundle and further development of intermediate lactate bundle approach, including piloting specific treatment bundles targeting this population. Collaborative performance improvement environment in which successful strategies at 1 site were rapidly disseminated to other sites including the Sepsis Alert and the Sepsis Clock. |
| May 2012 | Research analysis of fluid volume and lactate clearance in intermediate lactate sepsis population begins. |
| February 2013 | Fourth Regional Sepsis Summit. Regional spread of intermediate lactate bundle including the use of fluids, antibiotics, and repeat lactate measurements. |
| May 2013 | Research analysis of the contribution of sepsis to hospital deaths (within KP and in national sample) as well as post‐sepsis resource utilization and mortality |
| March 2014 | Publication of ProCESS randomized clinical trial, requiring systemic reevaluation of EGDT‐based sepsis strategy. Subsequent publications of ARISE and ProMISe trials confirming findings from ProCESS. Updated approach under consideration and informally disseminated to practitioners. |
| October 2014 | Updated sepsis treatment guidelines and data capture strategy fully implemented moving away from a catheter‐based strategy for all EGDT‐eligible patients. |
| October 2015 | Sixth Regional Sepsis Summit held to adjust sepsis treatment and data measurement strategy to align more closely with CMS SEP‐1 guidelines. |
Based on the principles of performance improvement methodology, clinical and operational leaders established an environment with aligned culture, incentives, and leadership around sepsis care. The effort was launched in late 2008 at a Sepsis Summit, bringing together a multidisciplinary group of stakeholders (eg, hospitalist, emergency department, and intensive care chiefs of staff and nursing managers; medical center and nursing executive and operational leadership) and providing sepsis care pathways based on the best available evidence.[23] Regional investments in the digital infrastructure to support implementation resulted in the provision of granular data within monthly sepsis scorecards quantifying each medical center's performance and trends for a diverse set of sepsis bundle metrics.
The resulting changes in sepsis care were substantial. For example, improved early recognition of infected patients meeting the criteria for sepsis resulted in large changes in the standardized diagnostic criteria used to label patients (Figure 3A). Implementing screening strategies using serum lactate testing for any patient receiving blood cultures resulted in a roughly 10‐fold increase in the use of lactate testing in the emergency department (Figure 3B). Earlier recognition of sepsis also increased the number of patients receiving early antibiotics and receiving central venous catheters for quantitative resuscitation.[23]
CLOSING THE LOOP TOWARD CONTINUOUS LEARNING IN SEPSIS
Leveraging timely and actionable data steeped within an aligned organizational milieu resulted in large‐scale changes across a heterogeneous set of hospitals. However, to realize the true vision of a learning hospital system, a looming question remained: Could the data generated as the byproduct of routine care now be used to complete the virtuous cycle and drive new scientific discovery (Figure 2, step 3)?
Confirming Concordance in the Impact of Sepsis Nationally
The heightened identification of sepsis patients through program implementation revealed that the impact of sepsis on hospital mortality was greater than originally estimated; based on improved patient identification, sepsis now accounted for upward of 1 in 2 hospital deaths.[25] This sobering statistic confirmed that the investments in standardizing best sepsis care following the mortality diagnostic were critical. However, were similar estimates of sepsis‐attributable mortality consistent outside of the KPNC system? To study this, we examined US hospitalizations occurring across >1000 hospitals and involving >6 million hospital stays to estimate corresponding prevalence.[25] In this national sample, sepsis contributed to as many as half of hospital deaths in the United States in 2010, lending strong support to ongoing international and state‐based efforts to improve sepsis care. These studies also paved the way to use these data drawn from our large sepsis population to inform updated international consensus definitions for sepsis and septic shock.[26, 27, 28]
Identifying New Avenues for Reducing the Toll of Sepsis
A major concern of sepsis program leaders was the prior findings that sepsis hospitalizations among Medicare beneficiaries were associated with substantial new cognitive and functional disability.[29] This lingering toll of sepsis had been termed a hidden public health disaster.[30] To further understand the posthospital impact of sepsis and to begin investigating new avenues to reduce this impact, a cohort of patients was followed for 1 year following sepsis hospitalization.[31] Over that period, nearly half of sepsis survivors were rehospitalized. When compared with their presepsis levels of healthcare utilization, middle‐aged and elderly sepsis patients experienced a 3‐fold increase in their days using facility‐based care. Subsequent studies in other populations outside of KPNC have confirmed these concerning findings, resulting in new efforts to address postsepsis survivorship care.[32, 33]
Phenotyping New Targets for Standardized Sepsis Care
At its outset, the sepsis improvement program applied the best available evidence to treat patients with the most severe forms of sepsisseptic shock. However, once the initial implementation phase had succeeded, clinicians and operational leaders quickly realized from the emerging data that there was a far larger group of sepsis patients for whom treatment guidelines were poorly defined.[25, 34, 35] These were severe sepsis patients with so‐called intermediate lactate values between 2 mmol/L and 4 mmol/L; they comprised a substantial proportion of all sepsis patients dying in the hospital. Using data generated from the routine care of sepsis patients treated across 21 hospitals, the sepsis leadership group was able to rapidly assemble a cohort of intermediate lactate sepsis patients up to 20‐ to 100‐fold larger than that reported in prior studies and evaluate their outcomes.[34, 35]
The data used to evaluate these intermediate lactate sepsis patients now spurred a new implementation program in 2013 for a group of patients in whom there was essentially no existing evidence to guide care. Rapidly implemented within a mature sepsis performance improvement program, evaluations at the 6‐month and 1‐year intervals demonstrated significant decreases in mortality.[36] Importantly, to allay the justified concerns of clinicians, these evaluations also clearly showed no evidence of harm from more aggressive fluid resuscitation (eg, increased transfer to intensive care, increased rates of mechanical ventilation). Again, driven by clinician input, subgroup analyses further revealed that the implementation program was only associated with reduced mortality in patients who could be at risk for iatrogenic fluid overload (ie, those with a history of congestive heart failure or chronic kidney disease).[36] Spurred by these provocative findings, operational and clinical leaders are currently considering how to guide future care in these patients, especially with the emerging use of noninvasive methods to quantify patients' fluid responsiveness.
PRINCIPLES FOR LEVERAGING DATA IN THE LEARNING HOSPITAL SYSTEM
The object lesson of using data to drive improved sepsis care and further new scientific discovery offers some important insights for continuous learning.
Building a Digital Infrastructure for Utilizing Granular Hospital Data
As described above, current transitions between the nodes of the learning framework are rife with missed opportunities. Perhaps one of the most glaring is the inability to use highly granular data already collected within the electronic health record (eg, trajectories and trends across vital signs or laboratory results, large‐scale medication administration records to evaluate multidrug interactions). An essential starting point for continuous learning is investing in the digital infrastructure to improve the use of data beyond traditional claims (administrative dataadmission source codes, disposition codes, diagnoses, and procedures). As shown in Table 2, the first key step is incorporating laboratory data into the quality assessment/emmprovement process. In addition, using these data to automate severity of illness and risk adjustment metrics fosters use of similar comparison cohorts across time or disease types.[18, 37, 38, 39, 40]
| Data Type | Contents | Degree of Difficulty in Accessing | Degree of Difficulty in Analyzing |
|---|---|---|---|
| Administrative | Traditional claims data, diagnostic or procedural codes | Low | Low to moderate |
| Standard cohort profiling | Limited instances of vitals signs, laboratory, diagnostic testing, or treatment data | Low to moderate | Low to moderate |
| Metrics reporting for care improvement | Standard cohort identification, aggregated achievement of treatment targets, scorecard dissemination | Moderate | Moderate |
| Advanced cohort profiling | Time series of physiologic data, inpatient triage and treatment data within short temporal intervals | Moderate to high | High |
| Research‐grade discovery | Data with breadth (representative sample size) and depth (highly granular physiologic and treatment data) | High | Very high |
| Patient‐reported outcomes | Quality of life, functional and cognitive disability | Very high | High |
Employing Novel Methods to Address the Limitations of Using Real‐World Data
The rapid digitization of medicine through the use of electronic medical records offers tremendous opportunities to facilitate continuous learning. However, these opportunities are accompanied by important limitations.[41] Data collected as a byproduct of real‐world care can be vulnerable to many forms of bias and confounding, potentially clouding the validity and robustness of corresponding analytic results. Fortunately, advanced methods including causal inference are now used routinely to address some limitations.[42] In the context of a learning healthcare system, other opportunities for improved study design including cluster randomized trials or stepped wedge implementation can also be employed to preserve the statistical rigor of subsequent analyses.[43] Finally, emerging methods employing randomization through the electronic medical record alongside adaptive trial design offer great potential to increase the efficiency of continuous learning.[44]
Evaluating the Hospital as a Single System
Advances in contemporary hospital care require seamless transitions of patient care, screening strategies, and therapeutic approaches across multiple hospital domains and with diverse providers; these interventions also need to happen rapidly. Many traditional approaches to inpatient care have taken a bottom‐up approach (eg, studying a specific disease within a specific hospital ward like the intensive care unit) that have proven useful but may limit generalizability when applied to a real‐world hospital operating with Pareto optimality (ie, the trade‐off scenario where new resource allocation to 1 area also requires resource withdrawal from another area). In certain cases, an empiric approach, without initial preference for any specific ward or disease, can aid decision making by hospital operational and clinical leaders by providing a global picture of impact and value.
Focusing on Early Detection in Hospital Settings as Secondary Prevention
Once patients have been admitted to the hospital, a race against the clock begins. Each additional hour of hospitalization increases the risks of iatrogenic injury or medical harm manifested by immobility, disorientation and delirium, nosocomial infections, or medication errors, among others. In this context, detection systems that use granular hospital data to focus on the earliest detection of risk can aid critical approaches to secondary prevention (Although the hospitalization for sepsis cannot be avoided, careful attention to mobility can limit the risk of developing delirium. In turn, preventing delirium can limit the risk of new functional disability).
Contextualizing Hospital Care Within a Longitudinal Trajectory
Although we described the benefit of hospital episodes having well‐demarcated beginning and ending points, it remains essential to recognize that the harms associated with hospitalization extend well beyond discharge. In this context, hospitalizations can serve as waypoints in patients' health trajectories as well as an opportunity to achieve patient‐centered care including discussing and aligning goals of care with actual care provision. Furthermore, although we have seen steady declines in hospital mortality over time, it is highly likely that we will reach a nadir in mortality where additional metrics of hospital outcomes will need to include postdischarge events like readmission, long‐term mortality, quality of life, and the prevention of disability or decline.
CONCLUSION
Hospitalizations in the United States are costly and associated with high mortality and morbidity; the toll of hospitalization also extends well beyond hospital discharge. The promise of the learning hospital system has marked improvements in the quality of hospital care, especially where healthcare systems can steep critical investments in data and digital infrastructure within the right culture, incentives, and leadership. Where continuous learning is achieved, data generated during routine care offer the potential to yield new scientific discovery and drive further improvements in hospital care.
Disclosures
As part of our agreement with the Gordon and Betty Moore Foundation, we made a commitment to disseminate our findings in articles such as this one. However, the Foundation and its staff played no role in how we actually structured our articles, nor did they review or preapprove any of the manuscripts submitted as part of the dissemination component. None of the authors has any conflicts of interest to declare of relevance to this work, which was funded by a combination of funding from the Gordon and Betty Moore Foundation, The Permanente Medical Group, Inc., and Kaiser Foundation Hospitals, Inc. VXL was supported by NIH K23GM112018.
- Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press; 2012.
- , , , et al. Toward a science of learning systems: a research agenda for the high‐functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43–50.
- National Center for Health Statistics. Health, United States, 2014: With Special Feature on Adults Aged 55–64. Hyattsville, MD; 2015.
- , . Critical care medicine in the United States 2000‐2005: an analysis of bed numbers, occupancy rates, payer mix, and costs. Crit Care Med. 2010;38(1):65–71.
- , , , . Trends and variation in end‐of‐life care for medicare beneficiaries with severe chronic illness. A report of the Dartmouth Atlas Project. Lebanon, NH: The Dartmouth Institute for Health Policy and Clinical Practice; 2011.
- , , , et al. Change in end‐of‐life care for Medicare beneficiaries: site of death, place of care, and health care transitions in 2000, 2005, and 2009. JAMA. 2013;309(5):470–477.
- , , . Finding the missing link for big biomedical data. JAMA. 2014;311(24):2479–2480.
- . Code red and blue—safely limiting health care's GDP footprint. N Engl J Med. 2013;368(1):1–3.
- , . The inevitable application of big data to health care. JAMA. 2013;309(13):1351–1352.
- , . What is citizen science?—a scientometric meta‐analysis. PLoS One. 2016;11(1):e0147152.
- , , . Biomedical ontologies: a functional perspective. Brief Bioinform. 2008;9(1):75–90.
- . Rapid learning: a breakthrough agenda. Health Aff (Millwood). 2014;33(7):1155–1162.
- , , , et al. Are regional variations in end‐of‐life care intensity explained by patient preferences?: a study of the US Medicare population. Med Care. 2007;45(5):386–393.
- , . Extreme markup: the fifty US hospitals with the highest charge‐to‐cost ratios. Health Aff (Millwood). 2015;34(6):922–928.
- , , , . The price ain't right? Hospital prices and health spending on the privately insured. Health Care Pricing Project website. Available at: http://www.healthcarepricingproject.org/sites/default/files/pricing_variation_manuscript_0.pdf. Accessed February 15, 2016
- . A new, evidence‐based estimate of patient harms associated with hospital care. J Patient Saf. 2013;9(3):122–128.
- , , . Hospitalization‐associated disability: “she was probably able to ambulate, but I'm not sure”. JAMA. 2011;306(16):1782–1793.
- , , , , . Risk‐adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5):446–453.
- . New definitions for sepsis and septic shock: continuing evolution but with much still to be done. JAMA. 2016;315(8):757–759.
- , . Severe sepsis and septic shock. N Engl J Med. 2013;369(21):2063.
- , , . Sepsis‐induced immunosuppression: from cellular dysfunctions to immunotherapy. Nat Rev Immunol. 2013;13(12):862–874.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
- , , , et al. Kaiser Permanente's performance improvement system, part 3: multisite improvements in care for patients with sepsis. Jt Comm J Qual Patient Saf. 2011;37(11):483–493.
- , , , , . Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91(2):354–394.
- , , , et al. Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA. 2014;312(1):90–92.
- , , , et al. Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):762–774.
- , , , et al. Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):775–787.
- , , , et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):801–810.
- , , , . Long‐term cognitive impairment and functional disability among survivors of severe sepsis. JAMA. 2010;304(16):1787–1794.
- . The lingering consequences of sepsis: a hidden public health disaster? JAMA. 2010;304(16):1833–1834.
- , , , , , . Hospital readmission and healthcare utilization following sepsis in community settings. J Hosp Med. 2014;9(8):502–507.
- , , , , . Increased 1‐year healthcare use in survivors of severe sepsis. Am J Respir Crit Care Med. 2014;190(1):62–69.
- , , , et al. Post‐acute care use and hospital readmission after sepsis. Ann Am Thorac Soc. 2015;12(6):904–913.
- , , , , . Fluid volume, lactate values, and mortality in sepsis patients with intermediate lactate values. Ann Am Thorac Soc. 2013;10(5):466–473.
- , , . Prognosis of emergency department patients with suspected infection and intermediate lactate levels: a systematic review. J Crit Care. 2014;29(3):334–339.
- , , , et al. Multicenter implementation of a treatment bundle for sepsis patients with intermediate lactate values. Am J Respir Crit Care Med. 2016;193(11):1264–1270.
- , , , et al. Risk adjusting community‐acquired pneumonia hospital outcomes using automated databases. Am J Manag Care. 2008;14(3):158–166.
- , , , , , . Intra‐hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS). J Hosp Med. 2011;6(2):74–80.
- , , , , , . Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7(5):388–395.
- , , , , . An electronic Simplified Acute Physiology Score‐based risk adjustment score for critical illness in an integrated healthcare system. Crit Care Med. 2013;41(1):41–48.
- . Learning from big health care data. N Engl J Med. 2014;370(23):2161–2163.
- , . Getting the methods right—the foundation of patient‐centered outcomes research. N Engl J Med. 2012;367(9):787–790.
- , , , , . The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.
- . Fusing randomized trials with big data: the key to self‐learning health care systems? JAMA. 2015;314(8):767–768.
- Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press; 2012.
- , , , et al. Toward a science of learning systems: a research agenda for the high‐functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43–50.
- National Center for Health Statistics. Health, United States, 2014: With Special Feature on Adults Aged 55–64. Hyattsville, MD; 2015.
- , . Critical care medicine in the United States 2000‐2005: an analysis of bed numbers, occupancy rates, payer mix, and costs. Crit Care Med. 2010;38(1):65–71.
- , , , . Trends and variation in end‐of‐life care for medicare beneficiaries with severe chronic illness. A report of the Dartmouth Atlas Project. Lebanon, NH: The Dartmouth Institute for Health Policy and Clinical Practice; 2011.
- , , , et al. Change in end‐of‐life care for Medicare beneficiaries: site of death, place of care, and health care transitions in 2000, 2005, and 2009. JAMA. 2013;309(5):470–477.
- , , . Finding the missing link for big biomedical data. JAMA. 2014;311(24):2479–2480.
- . Code red and blue—safely limiting health care's GDP footprint. N Engl J Med. 2013;368(1):1–3.
- , . The inevitable application of big data to health care. JAMA. 2013;309(13):1351–1352.
- , . What is citizen science?—a scientometric meta‐analysis. PLoS One. 2016;11(1):e0147152.
- , , . Biomedical ontologies: a functional perspective. Brief Bioinform. 2008;9(1):75–90.
- . Rapid learning: a breakthrough agenda. Health Aff (Millwood). 2014;33(7):1155–1162.
- , , , et al. Are regional variations in end‐of‐life care intensity explained by patient preferences?: a study of the US Medicare population. Med Care. 2007;45(5):386–393.
- , . Extreme markup: the fifty US hospitals with the highest charge‐to‐cost ratios. Health Aff (Millwood). 2015;34(6):922–928.
- , , , . The price ain't right? Hospital prices and health spending on the privately insured. Health Care Pricing Project website. Available at: http://www.healthcarepricingproject.org/sites/default/files/pricing_variation_manuscript_0.pdf. Accessed February 15, 2016
- . A new, evidence‐based estimate of patient harms associated with hospital care. J Patient Saf. 2013;9(3):122–128.
- , , . Hospitalization‐associated disability: “she was probably able to ambulate, but I'm not sure”. JAMA. 2011;306(16):1782–1793.
- , , , , . Risk‐adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013;51(5):446–453.
- . New definitions for sepsis and septic shock: continuing evolution but with much still to be done. JAMA. 2016;315(8):757–759.
- , . Severe sepsis and septic shock. N Engl J Med. 2013;369(21):2063.
- , , . Sepsis‐induced immunosuppression: from cellular dysfunctions to immunotherapy. Nat Rev Immunol. 2013;13(12):862–874.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
- , , , et al. Kaiser Permanente's performance improvement system, part 3: multisite improvements in care for patients with sepsis. Jt Comm J Qual Patient Saf. 2011;37(11):483–493.
- , , , , . Understanding the components of quality improvement collaboratives: a systematic literature review. Milbank Q. 2013;91(2):354–394.
- , , , et al. Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA. 2014;312(1):90–92.
- , , , et al. Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):762–774.
- , , , et al. Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):775–787.
- , , , et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis‐3). JAMA. 2016;315(8):801–810.
- , , , . Long‐term cognitive impairment and functional disability among survivors of severe sepsis. JAMA. 2010;304(16):1787–1794.
- . The lingering consequences of sepsis: a hidden public health disaster? JAMA. 2010;304(16):1833–1834.
- , , , , , . Hospital readmission and healthcare utilization following sepsis in community settings. J Hosp Med. 2014;9(8):502–507.
- , , , , . Increased 1‐year healthcare use in survivors of severe sepsis. Am J Respir Crit Care Med. 2014;190(1):62–69.
- , , , et al. Post‐acute care use and hospital readmission after sepsis. Ann Am Thorac Soc. 2015;12(6):904–913.
- , , , , . Fluid volume, lactate values, and mortality in sepsis patients with intermediate lactate values. Ann Am Thorac Soc. 2013;10(5):466–473.
- , , . Prognosis of emergency department patients with suspected infection and intermediate lactate levels: a systematic review. J Crit Care. 2014;29(3):334–339.
- , , , et al. Multicenter implementation of a treatment bundle for sepsis patients with intermediate lactate values. Am J Respir Crit Care Med. 2016;193(11):1264–1270.
- , , , et al. Risk adjusting community‐acquired pneumonia hospital outcomes using automated databases. Am J Manag Care. 2008;14(3):158–166.
- , , , , , . Intra‐hospital transfers to a higher level of care: contribution to total hospital and intensive care unit (ICU) mortality and length of stay (LOS). J Hosp Med. 2011;6(2):74–80.
- , , , , , . Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7(5):388–395.
- , , , , . An electronic Simplified Acute Physiology Score‐based risk adjustment score for critical illness in an integrated healthcare system. Crit Care Med. 2013;41(1):41–48.
- . Learning from big health care data. N Engl J Med. 2014;370(23):2161–2163.
- , . Getting the methods right—the foundation of patient‐centered outcomes research. N Engl J Med. 2012;367(9):787–790.
- , , , , . The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391.
- . Fusing randomized trials with big data: the key to self‐learning health care systems? JAMA. 2015;314(8):767–768.
