User login
Incidence, predictors, and outcomes of hospital-acquired anemia
Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.8 However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.
The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.2 found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.
Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.
METHODS
Study Design, Population, and Data Sources
We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.
Definition of Hospital-Acquired Anemia
HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.13 If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).2,14
Characteristics
We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).15 We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.16
Outcomes
The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.
Statistical Analysis
We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).17
The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.
RESULTS
Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).
Epidemiology of HAA
Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).
Predictors of HAA
Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R2).
Incidence of Postdischarge Outcomes by Severity of HAA
The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).
Association of HAA and Postdischarge Outcomes
In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).
In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.
DISCUSSION
In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.
To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.3 found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.
Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.23-25
The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.
Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.26 Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.18,27,28
In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.
Acknowledgments
The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.
Disclosures
This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.
1. Kurniali PC, Curry S, Brennan KW, et al. A retrospective study investigating the incidence and predisposing factors of hospital-acquired anemia. Anemia. 2014;2014:634582. PubMed
2. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. J Hosp Med. 2013;8(9):506-512. PubMed
3. Salisbury AC, Alexander KP, Reid KJ, et al. Incidence, correlates, and outcomes of acute, hospital-acquired anemia in patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(4):337-346. PubMed
4. Salisbury AC, Amin AP, Reid KJ, et al. Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction. Am Heart J. 2011;162(2):300-309 e303. PubMed
5. Meroño O, Cladellas M, Recasens L, et al. In-hospital acquired anemia in acute coronary syndrome. Predictors, in-hospital prognosis and one-year mortality. Rev Esp Cardiol (Engl Ed). 2012;65(8):742-748. PubMed
6. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. PubMed
7. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20(6):520-524. PubMed
8. Rennke S, Fang MC. Hazards of hospitalization: more than just “never events”. Arch Intern Med. 2011;171(18):1653-1654. PubMed
9. Choi JS, Kim YA, Kang YU, et al. Clinical impact of hospital-acquired anemia in association with acute kidney injury and chronic kidney disease in patients with acute myocardial infarction. PLoS One. 2013;8(9):e75583. PubMed
10. Salisbury AC, Kosiborod M, Amin AP, et al. Recovery from hospital-acquired anemia after acute myocardial infarction and effect on outcomes. Am J Cardiol. 2011;108(7):949-954. PubMed
11. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
12. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
13. World Health Organization. Hemoglobin concentrations for the diagnosis of anaemia and assessment of severity. http://www.who.int/vmnis/indicators/haemoglobin.pdf. Accessed March 15, 2016.
14. Martin ND, Scantling D. Hospital-acquired anemia: a contemporary review of etiologies and prevention strategies. J Infus Nurs. 2015;38(5):330-338. PubMed
15. Agency for Healthcare Research and Quality, Rockville, MD. Clinical classification software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project. 2015 http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed November 18, 2015.
16. Agency for Healthcare Research and Quality, Rockville, MD. Procedure classes2015. Healthcare Cost and Utilization Project. 2015. https://www.hcup-us.ahrq.gov/toolssoftware/procedure/procedure.jsp. Accessed November 18, 2015.
17. Corwin HL, Gettinger A, Pearl RG, et al. The CRIT Study: Anemia and blood transfusion in the critically ill--current clinical practice in the United States. Crit Care Med. 2004;32(1):39-52. PubMed
18. Aronson D, Suleiman M, Agmon Y, et al. Changes in haemoglobin levels during hospital course and long-term outcome after acute myocardial infarction. Eur Heart J. 2007;28(11):1289-1296. PubMed
19. Lyon AW, Chin AC, Slotsve GA, Lyon ME. Simulation of repetitive diagnostic blood loss and onset of iatrogenic anemia in critical care patients with a mathematical model. Comput Biol Med. 2013;43(2):84-90. PubMed
20. van der Bom JG, Cannegieter SC. Hospital-acquired anemia: the contribution of diagnostic blood loss. J Thromb Haemost. 2015;13(6):1157-1159. PubMed
21. Sanchez-Giron F, Alvarez-Mora F. Reduction of blood loss from laboratory testing in hospitalized adult patients using small-volume (pediatric) tubes. Arch Pathol Lab Med. 2008;132(12):1916-1919. PubMed
22. Smoller BR, Kruskall MS. Phlebotomy for diagnostic laboratory tests in adults. Pattern of use and effect on transfusion requirements. N Engl J Med. 1986;314(19):1233-1235. PubMed
23. Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. Cochrane Database Syst Rev. 2012(4):CD002042. PubMed
24. Carson JL, Carless PA, Hébert PC. Outcomes using lower vs higher hemoglobin thresholds for red blood cell transfusion. JAMA. 2013;309(1):83-84. PubMed
25. Carson JL, Sieber F, Cook DR, et al. Liberal versus restrictive blood transfusion strategy: 3-year survival and cause of death results from the FOCUS randomised controlled trial. Lancet. 2015;385(9974):1183-1189. PubMed
26. Rajkomar A, McCulloch CE, Fang MC. Low diagnostic utility of rechecking hemoglobins within 24 hours in hospitalized patients. Am J Med. 2016;129(11):1194-1197. PubMed
27. Reade MC, Weissfeld L, Angus DC, Kellum JA, Milbrandt EB. The prevalence of anemia and its association with 90-day mortality in hospitalized community-acquired pneumonia. BMC Pulm Med. 2010;10:15. PubMed
28. Halm EA, Wang JJ, Boockvar K, et al. The effect of perioperative anemia on clinical and functional outcomes in patients with hip fracture. J Orthop Trauma. 2004;18(6):369-374. PubMed
Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.8 However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.
The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.2 found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.
Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.
METHODS
Study Design, Population, and Data Sources
We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.
Definition of Hospital-Acquired Anemia
HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.13 If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).2,14
Characteristics
We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).15 We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.16
Outcomes
The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.
Statistical Analysis
We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).17
The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.
RESULTS
Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).
Epidemiology of HAA
Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).
Predictors of HAA
Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R2).
Incidence of Postdischarge Outcomes by Severity of HAA
The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).
Association of HAA and Postdischarge Outcomes
In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).
In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.
DISCUSSION
In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.
To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.3 found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.
Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.23-25
The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.
Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.26 Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.18,27,28
In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.
Acknowledgments
The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.
Disclosures
This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.
Hospital-acquired anemia (HAA) is defined as having a normal hemoglobin value upon admission but developing anemia during the course of hospitalization. The condition is common, with an incidence ranging from approximately 25% when defined by using the hemoglobin value prior to discharge to 74% when using the nadir hemoglobin value during hospitalization.1-5 While there are many potential etiologies for HAA, given that iatrogenic blood loss from phlebotomy may lead to its development,6,7 HAA has been postulated to be a hazard of hospitalization that is potentially preventable.8 However, it is unclear whether the development of HAA portends worse outcomes after hospital discharge.
The limited number of studies on the association between HAA and postdischarge outcomes has been restricted to patients hospitalized for acute myocardial infarction (AMI).3,9,10 Among this subpopulation, HAA is independently associated with greater morbidity and mortality following hospital discharge.3,9,10 In a more broadly representative population of hospitalized adults, Koch et al.2 found that the development of HAA is associated with greater length of stay (LOS), hospital charges, and inpatient mortality. However, given that HAA was defined by the lowest hemoglobin level during hospitalization (and not necessarily the last value prior to discharge), it is unclear if the worse outcomes observed were the cause of the HAA, rather than its effect, since hospital LOS is a robust predictor for the development of HAA, as well as a major driver of hospital costs and a prognostic marker for inpatient mortality.3,9 Furthermore, this study evaluated outcomes only during the index hospitalization, so it is unclear if patients who develop HAA have worse clinical outcomes after discharge.
Therefore, in this study, we used clinically granular electronic health record (EHR) data from a diverse cohort of consecutive medicine inpatients hospitalized for any reason at 1 of 6 hospitals to: 1) describe the epidemiology of HAA; 2) identify predictors of its development; and 3) examine its association with 30-day postdischarge adverse outcomes. We hypothesized that the development of HAA would be independently associated with 30-day readmission and mortality in a dose-dependent fashion, with increasing severity of HAA associated with worse outcomes.
METHODS
Study Design, Population, and Data Sources
We conducted a retrospective observational cohort study using EHR data collected from November 1, 2009 to October 30, 2010 from 6 hospitals in the north Texas region. One site was a university-affiliated safety-net hospital; the remaining 5 community hospitals were a mix of teaching and nonteaching sites. All hospitals used the Epic EHR system (Epic Systems Corporation, Verona, Wisconsin). Details of this cohort have been published.11,12This study included consecutive hospitalizations among adults age 18 years or older who were discharged from a medicine inpatient service with any diagnosis. We excluded hospitalizations by individuals who were anemic within the first 24 hours of admission (hematocrit less than 36% for women and less than 40% for men), were missing a hematocrit value within the first 24 hours of hospitalization or a repeat hematocrit value prior to discharge, had a hospitalization in the preceding 30 days (ie, index hospitalization was considered a readmission), died in the hospital, were transferred to another hospital, or left against medical advice. For individuals with multiple eligible hospitalizations during the study period, we included only the first hospitalization. We also excluded those discharged to hospice, given that this population of individuals may have intentionally desired less aggressive care.
Definition of Hospital-Acquired Anemia
HAA was defined as having a normal hematocrit value (36% or greater for women and 40% or greater for men) within the first 24 hours of admission and a hematocrit value at the time of hospital discharge lower than the World Health Organization’s sex-specific cut points.13 If there was more than 1 hematocrit value on the same day, we chose the lowest value. Based on prior studies, HAA was further categorized by severity as mild (hematocrit greater than 33% and less than 36% in women; and greater than 33% and less than 40% in men), moderate (hematocrit greater than 27% and 33% or less for all), or severe (hematocrit 27% or less for all).2,14
Characteristics
We extracted information on sociodemographic characteristics, comorbidities, LOS, procedures, blood transfusions, and laboratory values from the EHR. Hospitalizations in the 12 months preceding the index hospitalization were ascertained from the EHR and from an all-payer regional hospitalization database that captures hospitalizations from 75 acute care hospitals within a 100-mile radius of Dallas-Fort Worth. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes were categorized according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS).15 We defined a diagnosis for hemorrhage and coagulation, and hemorrhagic disorder as the presence of any ICD-9-CM code (primary or secondary) that mapped to the AHRQ CCS diagnoses 60 and 153, and 62, respectively. Procedures were categorized as minor diagnostic, minor therapeutic, major diagnostic, and major therapeutic using the AHRQ Healthcare Cost and Utilization Procedure Classes tool.16
Outcomes
The primary outcome was a composite of death or readmission within 30 days of hospital discharge. Hospital readmissions were ascertained at the index hospital and at any of 75 acute care hospitals in the region as described earlier. Death was ascertained from each of the hospitals’ EHR and administrative data and the Social Security Death Index. Individuals who had both outcomes (eg, a 30-day readmission and death) were considered to have only 1 occurrence of the composite primary outcome measure. Our secondary outcomes were death and readmission within 30 days of discharge, considered as separate outcomes.
Statistical Analysis
We used logistic regression models to evaluate predictors of HAA and to estimate the association of HAA on subsequent 30-day adverse outcomes after hospital discharge. All models accounted for clustering of patients by hospital. For the outcomes analyses, models were adjusted for potential confounders based on prior literature and our group’s expertise, which included age, sex, race/ethnicity, Charlson comorbidity index, prior hospitalizations, nonelective admission status, creatinine level on admission, blood urea nitrogen (BUN) to creatinine ratio of more than 20:1 on admission, LOS, receipt of a major diagnostic or therapeutic procedure during the index hospitalization, a discharge diagnosis for hemorrhage, and a discharge diagnosis for a coagulation or hemorrhagic disorder. For the mortality analyses, given the limited number of 30-day deaths after hospital discharge in our cohort, we collapsed moderate and severe HAA into a single category. In sensitivity analyses, we repeated the adjusted model, but excluded patients in our cohort who had received at least 1 blood transfusion during the index hospitalization (2.6%) given its potential for harm, and patients with a primary discharge diagnosis for AMI (3.1%).17
The functional forms of continuous variables were assessed using restricted cubic splines and locally weighted scatterplot smoothing techniques. All analyses were performed using STATA statistical software version 12.0 (StataCorp, College Station, Texas). The University of Texas Southwestern Medical Center institutional review board approved this study.
RESULTS
Of 53,995 consecutive medicine hospitalizations among adults age 18 years or older during our study period, 11,309 index hospitalizations were included in our study cohort (Supplemental Figure 1). The majority of patients excluded were because of having documented anemia within the first 24 hours of admission (n=24,950). With increasing severity of HAA, patients were older, more likely to be female, non-Hispanic white, electively admitted, have fewer comorbidities, less likely to be hospitalized in the past year, more likely to have had a major procedure, receive a blood transfusion, have a longer LOS, and have a primary or secondary discharge diagnosis for a hemorrhage or a coagulation or hemorrhagic disorder (Table 1).
Epidemiology of HAA
Among this cohort of patients without anemia on admission, the median hematocrit value on admission was 40.6 g/dL and on discharge was 38.9 g/dL. One-third of patients with normal hematocrit value at admission developed HAA, with 21.6% developing mild HAA, 10.1% developing moderate HAA, and 1.4% developing severe HAA. The median discharge hematocrit value was 36 g/dL (interquartile range [IQR]), 35-38 g/dL) for the group of patients who developed mild HAA, 31 g/dL (IQR, 30-32 g/dL) for moderate HAA, and 26 g/dL (IQR, 25-27 g/dL) for severe HAA (Supplemental Figure 2). Among the severe HAA group, 135 of the 159 patients (85%) had a major procedure (n=123, accounting for 219 unique major procedures), a diagnosis for hemorrhage (n=30), and/or a diagnosis for a coagulation or hemorrhagic disorder (n=23) during the index hospitalization. Of the 219 major procedures among patients with severe HAA, most were musculoskeletal (92 procedures), cardiovascular (61 procedures), or digestive system-related (41 procedures). The most common types of procedures were coronary artery bypass graft (36 procedures), hip replacement (25 procedures), knee replacement (17 procedures), and femur fracture reduction (15 procedures). The 10 most common principal discharge diagnoses of the index hospitalization by HAA group are shown in Supplemental Table 1. For the severe HAA group, the most common diagnosis was hip fracture (20.8%).
Predictors of HAA
Compared to no or mild HAA, female sex, elective admission status, serum creatinine on admission, BUN to creatinine ratio greater than 20 to 1, hospital LOS, and undergoing a major diagnostic or therapeutic procedure were predictors for the development of moderate or severe HAA (Table 2). The model explained 23% of the variance (McFadden’s pseudo R2).
Incidence of Postdischarge Outcomes by Severity of HAA
The severity of HAA was associated with a dose-dependent increase in the incidence of 30-day adverse outcomes, such that patients with increasing severity of HAA had greater 30-day composite, mortality, and readmission outcomes (P < 0.001; Figure). The 30-day postdischarge composite outcome was primarily driven by hospital readmissions given the low mortality rate in our cohort. Patients who did not develop HAA had an incidence of 9.7% for the composite outcome, whereas patients with severe HAA had an incidence of 16.4%. Among the 24 patients with severe HAA but who had not undergone a major procedure or had a discharge diagnosis for hemorrhage or for a coagulation or hemorrhagic disorder, only 3 (12.5%) had a composite postdischarge adverse outcome (2 readmissions and 1 death). The median time to readmission was similar between groups, but more patients with severe HAA had an early readmission within 7 days of hospital discharge than patients who did not develop HAA (6.9% vs. 2.9%, P = 0.001; Supplemental Table 2).
Association of HAA and Postdischarge Outcomes
In unadjusted analyses, compared to not developing HAA, mild, moderate, and severe HAA were associated with a 29%, 61%, and 81% increase in the odds for a composite outcome, respectively (Table 3). After adjustment for confounders, the effect size for HAA attenuated and was no longer statistically significant for mild and moderate HAA. However, severe HAA was significantly associated with a 39% increase in the odds for the composite outcome and a 41% increase in the odds for 30-day readmission (P = 0.008 and P = 0.02, respectively).
In sensitivity analyses, the exclusion of individuals who received at least 1 blood transfusion during the index hospitalization (n=298) and individuals who had a primary discharge diagnosis for AMI (n=353) did not substantively change the estimates of the association between severe HAA and postdischarge outcomes (Supplemental Tables 3 and 4). However, because of the fewer number of adverse events for each analysis, the confidence intervals were wider and the association of severe HAA and the composite outcome and readmission were no longer statistically significant in these subcohorts.
DISCUSSION
In this large and diverse sample of medical inpatients, we found that HAA occurs in one-third of adults with normal hematocrit value at admission, where 10.1% of the cohort developed moderately severe HAA and 1.4% developed severe HAA by the time of discharge. Length of stay and undergoing a major diagnostic or therapeutic procedure were the 2 strongest potentially modifiable predictors of developing moderate or severe HAA. Severe HAA was independently associated with a 39% increase in the odds of being readmitted or dying within 30 days after hospital discharge compared to not developing HAA. However, the associations between mild or moderate HAA with adverse outcomes were attenuated after adjusting for confounders and were no longer statistically significant.
To our knowledge, this is the first study on the postdischarge adverse outcomes of HAA among a diverse cohort of medical inpatients hospitalized for any reason. In a more restricted population, Salisbury et al.3 found that patients hospitalized for AMI who developed moderate to severe HAA (hemoglobin value at discharge of 11 g/dL or less) had greater 1-year mortality than those without HAA (8.4% vs. 2.6%, P < 0.001), and had an 82% increase in the hazard for mortality (95% confidence interval, hazard ratio 1.11-2.98). Others have similarly shown that HAA is common among patients hospitalized with AMI and is associated with greater mortality.5,9,18 Our study extends upon this prior research by showing that severe HAA increases the risk for adverse outcomes for all adult inpatients, not only those hospitalized for AMI or among those receiving blood transfusions.
Despite the increased harm associated with severe HAA, it is unclear whether HAA is a preventable hazard of hospitalization, as suggested by others.6,8 Most patients in our cohort who developed severe HAA underwent a major procedure, had a discharge diagnosis for hemorrhage, and/or had a discharge diagnosis for a coagulation or hemorrhagic disorder. Thus, blood loss due to phlebotomy, 1 of the more modifiable etiologies of HAA, was unlikely to have been the primary driver for most patients who developed severe HAA. Since it has been estimated to take 15 days of daily phlebotomy of 53 mL of whole blood in females of average body weight (and 20 days for average weight males) with no bone marrow synthesis for severe anemia to develop, it is even less likely that phlebotomy was the principal etiology given an 8-day median LOS among patients with severe HAA.19,20 However, since the etiology of HAA can be multifactorial, limiting blood loss due to phlebotomy by using smaller volume tubes, blood conservation devices, or reducing unnecessary testing may mitigate the development of severe HAA.21,22 Additionally, since more than three-quarters of patients who developed severe HAA underwent a major procedure, more care and attention to minimizing operative blood loss could lessen the severity of HAA and facilitate better recovery. If minimizing blood loss is not feasible, in the absence of symptoms related to anemia or ongoing blood loss, randomized controlled trials overwhelmingly support a restrictive transfusion strategy using a hemoglobin value threshold of 7 mg/dL, even in the postoperative setting.23-25
The implications of mild to moderate HAA are less clear. The odds ratios for mild and moderate HAA, while not statistically significant, suggest a small increase in harm compared to not developing HAA. Furthermore, the upper boundary of the confidence intervals for mild and moderate HAA cannot exclude a possible 30% and 56% increase in the odds for the 30-day composite outcome, respectively. Thus, a better powered study, including more patients and extending the time interval for ascertaining postdischarge adverse events beyond 30 days, may reveal a harmful association. Lastly, our study assessed only the association of HAA with 30-day readmission and mortality. Examining the association between HAA and other patient-centered outcomes such as fatigue, functional impairment, and prolonged posthospitalization recovery time may uncover other important adverse effects of mild and moderate HAA, both of which occur far more frequently than severe HAA.
Our findings should be interpreted in the context of several limitations. First, although we included a diverse group of patients from a multihospital cohort, generalizability to other settings is uncertain. Second, as this was a retrospective study using EHR data, we had limited information to infer the precise mechanism of HAA for each patient. However, procedure codes and discharge diagnoses enabled us to assess which patients underwent a major procedure or had a hemorrhage or hemorrhagic disorder during the hospitalization. Third, given the relatively few number of patients with severe HAA in our cohort, we were unable to assess if the association of severe HAA differed by suspected etiology. Lastly, because we were unable to ascertain the timing of the hematocrit values within the first 24 hours of admission, we excluded both patients with preexisting anemia on admission and those who developed HAA within the first 24 hours of admission, which is not uncommon.26 Thus, we were unable to assess the effect of acute on chronic anemia arising during hospitalization and HAA that develops within the first 24 hours, both of which may also be harmful.18,27,28
In conclusion, severe HAA occurs in 1.4% of all medical hospitalizations and is associated with increased odds of death or readmission within 30 days. Since most patients with severe HAA had undergone a major procedure or had a discharge diagnosis of hemorrhage or a coagulation or hemorrhagic disorder, it is unclear if severe HAA is potentially preventable through preventing blood loss from phlebotomy or by reducing iatrogenic injury during procedures. Future research should assess the potential preventability of severe HAA, and examine other patient-centered outcomes potentially related to anemia, including fatigue, functional impairment, and trajectory of posthospital recovery.
Acknowledgments
The authors would like to acknowledge Ruben Amarasingham, MD, MBA, President and Chief Executive Officer of the Parkland Center for Clinical Innovation, and Ferdinand Velasco, MD, Chief Health Information Officer at Texas Health Resources, for their assistance in assembling the 6 hospital cohort used in this study. The authors would also like to thank Valy Fontil, MD, MAS, Assistant Professor of Medicine at the University of California San Francisco School of Medicine, and Elizabeth Rogers, MD, MAS, Assistant Professor of Internal Medicine and Pediatrics at the University of Minnesota Medical School, for their constructive feedback on an earlier version of this manuscript.
Disclosures
This work was supported by the Agency for Healthcare Research and Quality-funded UT Southwestern Center for Patient-Centered Outcomes Research (R24 HS022418-01); the Commonwealth Foundation (#20100323); the UT Southwestern KL2 Scholars Program supported by the National Institutes of Health (KL2 TR001103); the National Center for Advancing Translational Sciences at the National Institute of Health (U54 RFA-TR-12-006); and the National Institute on Aging (K23AG052603). The study sponsors had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. The authors have no financial conflicts of interest to disclose.
1. Kurniali PC, Curry S, Brennan KW, et al. A retrospective study investigating the incidence and predisposing factors of hospital-acquired anemia. Anemia. 2014;2014:634582. PubMed
2. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. J Hosp Med. 2013;8(9):506-512. PubMed
3. Salisbury AC, Alexander KP, Reid KJ, et al. Incidence, correlates, and outcomes of acute, hospital-acquired anemia in patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(4):337-346. PubMed
4. Salisbury AC, Amin AP, Reid KJ, et al. Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction. Am Heart J. 2011;162(2):300-309 e303. PubMed
5. Meroño O, Cladellas M, Recasens L, et al. In-hospital acquired anemia in acute coronary syndrome. Predictors, in-hospital prognosis and one-year mortality. Rev Esp Cardiol (Engl Ed). 2012;65(8):742-748. PubMed
6. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. PubMed
7. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20(6):520-524. PubMed
8. Rennke S, Fang MC. Hazards of hospitalization: more than just “never events”. Arch Intern Med. 2011;171(18):1653-1654. PubMed
9. Choi JS, Kim YA, Kang YU, et al. Clinical impact of hospital-acquired anemia in association with acute kidney injury and chronic kidney disease in patients with acute myocardial infarction. PLoS One. 2013;8(9):e75583. PubMed
10. Salisbury AC, Kosiborod M, Amin AP, et al. Recovery from hospital-acquired anemia after acute myocardial infarction and effect on outcomes. Am J Cardiol. 2011;108(7):949-954. PubMed
11. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
12. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
13. World Health Organization. Hemoglobin concentrations for the diagnosis of anaemia and assessment of severity. http://www.who.int/vmnis/indicators/haemoglobin.pdf. Accessed March 15, 2016.
14. Martin ND, Scantling D. Hospital-acquired anemia: a contemporary review of etiologies and prevention strategies. J Infus Nurs. 2015;38(5):330-338. PubMed
15. Agency for Healthcare Research and Quality, Rockville, MD. Clinical classification software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project. 2015 http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed November 18, 2015.
16. Agency for Healthcare Research and Quality, Rockville, MD. Procedure classes2015. Healthcare Cost and Utilization Project. 2015. https://www.hcup-us.ahrq.gov/toolssoftware/procedure/procedure.jsp. Accessed November 18, 2015.
17. Corwin HL, Gettinger A, Pearl RG, et al. The CRIT Study: Anemia and blood transfusion in the critically ill--current clinical practice in the United States. Crit Care Med. 2004;32(1):39-52. PubMed
18. Aronson D, Suleiman M, Agmon Y, et al. Changes in haemoglobin levels during hospital course and long-term outcome after acute myocardial infarction. Eur Heart J. 2007;28(11):1289-1296. PubMed
19. Lyon AW, Chin AC, Slotsve GA, Lyon ME. Simulation of repetitive diagnostic blood loss and onset of iatrogenic anemia in critical care patients with a mathematical model. Comput Biol Med. 2013;43(2):84-90. PubMed
20. van der Bom JG, Cannegieter SC. Hospital-acquired anemia: the contribution of diagnostic blood loss. J Thromb Haemost. 2015;13(6):1157-1159. PubMed
21. Sanchez-Giron F, Alvarez-Mora F. Reduction of blood loss from laboratory testing in hospitalized adult patients using small-volume (pediatric) tubes. Arch Pathol Lab Med. 2008;132(12):1916-1919. PubMed
22. Smoller BR, Kruskall MS. Phlebotomy for diagnostic laboratory tests in adults. Pattern of use and effect on transfusion requirements. N Engl J Med. 1986;314(19):1233-1235. PubMed
23. Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. Cochrane Database Syst Rev. 2012(4):CD002042. PubMed
24. Carson JL, Carless PA, Hébert PC. Outcomes using lower vs higher hemoglobin thresholds for red blood cell transfusion. JAMA. 2013;309(1):83-84. PubMed
25. Carson JL, Sieber F, Cook DR, et al. Liberal versus restrictive blood transfusion strategy: 3-year survival and cause of death results from the FOCUS randomised controlled trial. Lancet. 2015;385(9974):1183-1189. PubMed
26. Rajkomar A, McCulloch CE, Fang MC. Low diagnostic utility of rechecking hemoglobins within 24 hours in hospitalized patients. Am J Med. 2016;129(11):1194-1197. PubMed
27. Reade MC, Weissfeld L, Angus DC, Kellum JA, Milbrandt EB. The prevalence of anemia and its association with 90-day mortality in hospitalized community-acquired pneumonia. BMC Pulm Med. 2010;10:15. PubMed
28. Halm EA, Wang JJ, Boockvar K, et al. The effect of perioperative anemia on clinical and functional outcomes in patients with hip fracture. J Orthop Trauma. 2004;18(6):369-374. PubMed
1. Kurniali PC, Curry S, Brennan KW, et al. A retrospective study investigating the incidence and predisposing factors of hospital-acquired anemia. Anemia. 2014;2014:634582. PubMed
2. Koch CG, Li L, Sun Z, et al. Hospital-acquired anemia: prevalence, outcomes, and healthcare implications. J Hosp Med. 2013;8(9):506-512. PubMed
3. Salisbury AC, Alexander KP, Reid KJ, et al. Incidence, correlates, and outcomes of acute, hospital-acquired anemia in patients with acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010;3(4):337-346. PubMed
4. Salisbury AC, Amin AP, Reid KJ, et al. Hospital-acquired anemia and in-hospital mortality in patients with acute myocardial infarction. Am Heart J. 2011;162(2):300-309 e303. PubMed
5. Meroño O, Cladellas M, Recasens L, et al. In-hospital acquired anemia in acute coronary syndrome. Predictors, in-hospital prognosis and one-year mortality. Rev Esp Cardiol (Engl Ed). 2012;65(8):742-748. PubMed
6. Salisbury AC, Reid KJ, Alexander KP, et al. Diagnostic blood loss from phlebotomy and hospital-acquired anemia during acute myocardial infarction. Arch Intern Med. 2011;171(18):1646-1653. PubMed
7. Thavendiranathan P, Bagai A, Ebidia A, Detsky AS, Choudhry NK. Do blood tests cause anemia in hospitalized patients? The effect of diagnostic phlebotomy on hemoglobin and hematocrit levels. J Gen Intern Med. 2005;20(6):520-524. PubMed
8. Rennke S, Fang MC. Hazards of hospitalization: more than just “never events”. Arch Intern Med. 2011;171(18):1653-1654. PubMed
9. Choi JS, Kim YA, Kang YU, et al. Clinical impact of hospital-acquired anemia in association with acute kidney injury and chronic kidney disease in patients with acute myocardial infarction. PLoS One. 2013;8(9):e75583. PubMed
10. Salisbury AC, Kosiborod M, Amin AP, et al. Recovery from hospital-acquired anemia after acute myocardial infarction and effect on outcomes. Am J Cardiol. 2011;108(7):949-954. PubMed
11. Nguyen OK, Makam AN, Clark C, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J Hosp Med. 2016;11(7):473-480. PubMed
12. Amarasingham R, Velasco F, Xie B, et al. Electronic medical record-based multicondition models to predict the risk of 30 day readmission or death among adult medicine patients: validation and comparison to existing models. BMC Med Inform Decis Mak. 2015;15:39. PubMed
13. World Health Organization. Hemoglobin concentrations for the diagnosis of anaemia and assessment of severity. http://www.who.int/vmnis/indicators/haemoglobin.pdf. Accessed March 15, 2016.
14. Martin ND, Scantling D. Hospital-acquired anemia: a contemporary review of etiologies and prevention strategies. J Infus Nurs. 2015;38(5):330-338. PubMed
15. Agency for Healthcare Research and Quality, Rockville, MD. Clinical classification software (CCS) for ICD-9-CM. Healthcare Cost and Utilization Project. 2015 http://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp. Accessed November 18, 2015.
16. Agency for Healthcare Research and Quality, Rockville, MD. Procedure classes2015. Healthcare Cost and Utilization Project. 2015. https://www.hcup-us.ahrq.gov/toolssoftware/procedure/procedure.jsp. Accessed November 18, 2015.
17. Corwin HL, Gettinger A, Pearl RG, et al. The CRIT Study: Anemia and blood transfusion in the critically ill--current clinical practice in the United States. Crit Care Med. 2004;32(1):39-52. PubMed
18. Aronson D, Suleiman M, Agmon Y, et al. Changes in haemoglobin levels during hospital course and long-term outcome after acute myocardial infarction. Eur Heart J. 2007;28(11):1289-1296. PubMed
19. Lyon AW, Chin AC, Slotsve GA, Lyon ME. Simulation of repetitive diagnostic blood loss and onset of iatrogenic anemia in critical care patients with a mathematical model. Comput Biol Med. 2013;43(2):84-90. PubMed
20. van der Bom JG, Cannegieter SC. Hospital-acquired anemia: the contribution of diagnostic blood loss. J Thromb Haemost. 2015;13(6):1157-1159. PubMed
21. Sanchez-Giron F, Alvarez-Mora F. Reduction of blood loss from laboratory testing in hospitalized adult patients using small-volume (pediatric) tubes. Arch Pathol Lab Med. 2008;132(12):1916-1919. PubMed
22. Smoller BR, Kruskall MS. Phlebotomy for diagnostic laboratory tests in adults. Pattern of use and effect on transfusion requirements. N Engl J Med. 1986;314(19):1233-1235. PubMed
23. Carson JL, Carless PA, Hebert PC. Transfusion thresholds and other strategies for guiding allogeneic red blood cell transfusion. Cochrane Database Syst Rev. 2012(4):CD002042. PubMed
24. Carson JL, Carless PA, Hébert PC. Outcomes using lower vs higher hemoglobin thresholds for red blood cell transfusion. JAMA. 2013;309(1):83-84. PubMed
25. Carson JL, Sieber F, Cook DR, et al. Liberal versus restrictive blood transfusion strategy: 3-year survival and cause of death results from the FOCUS randomised controlled trial. Lancet. 2015;385(9974):1183-1189. PubMed
26. Rajkomar A, McCulloch CE, Fang MC. Low diagnostic utility of rechecking hemoglobins within 24 hours in hospitalized patients. Am J Med. 2016;129(11):1194-1197. PubMed
27. Reade MC, Weissfeld L, Angus DC, Kellum JA, Milbrandt EB. The prevalence of anemia and its association with 90-day mortality in hospitalized community-acquired pneumonia. BMC Pulm Med. 2010;10:15. PubMed
28. Halm EA, Wang JJ, Boockvar K, et al. The effect of perioperative anemia on clinical and functional outcomes in patients with hip fracture. J Orthop Trauma. 2004;18(6):369-374. PubMed
© 2017 Society of Hospital Medicine
Automated Sepsis Alert Systems
Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]
Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).
However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.
MATERIALS AND METHODS
Data Sources and Search Strategies
We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).
Study Selection
Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.
Data Extraction and Quality Assessment
Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]
For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]
Data Synthesis
To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).
RESULTS
Description of Studies
Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).
| Source | Site No./Type | Setting | Alert Threshold | Gold Standard Definition | Gold Standard Measurement | No. | Study Qualitya |
|---|---|---|---|---|---|---|---|
| |||||||
| Hooper et al., 201210 | 1/academic | MICU | 2 SIRS criteriab | Reviewer judgment, not otherwise specified | Chart review | 560 | High |
| Meurer et al., 200913 | 1/academic | ED | 2 SIRS criteria | Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria | Chart review | 248 | Low |
| Nelson J. et al., 201114 | 1/academic | ED | 2 SIRS criteria and 2 SBP measurements <90 mm Hg | Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved | Chart review | 1,386 | High |
| Nguyen et al., 201415 | 1/academic | ED | 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) | Reviewer judgment to confirm SIRS, shock, and presence of a serious infection | Chart review | 1,095 | Low |
| Thiel et al., 201016 | 1/academic | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsc | Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy | ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer | 27,674 | Low |
| Source | Design | Site No./ Type | Setting | No. | Alert System Type | Alert Threshold | Alert Notificationa | Treatment Recommendation | Study Qualityb |
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
| Berger et al., 201017 | Before‐after (6 months pre and 6 months post) | 1/academic | ED | 5796c | CPOE system | 2 SIRS criteria | CPOE passive alert | Yes: lactate collection | Low |
| Hooper et al., 201210 | RCT | 1/academic | MICU | 443 | EHR | 2 SIRS criteriad | Text page and EHR passive alert | No | High |
| McRee et al., 201418 | Before‐after (6 months pre and 6 months post) | 1/academic | Wards | 171e | EHR | 2 SIRS criteria | Notified nurse, specifics unclear | No, but the nurse completed a sepsis risk evaluation flow sheet | Low |
| Nelson et al., 201114 | Before‐after (3 months pre and 3 months post) | 1/academic | ED | 184f | EHR | 2 SIRS criteria and 2 or more SBP readings <90 mm Hg | Text page and EHR passive alert | Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others | Low |
| Sawyer et al., 201119 | Prospective, nonrandomized (2 intervention and 4 control wards) | 1/academic | Wards | 300 | EHR | Recursive partitioning regression tree algorithm including vitals and lab valuesg | Text page to charge nurse who then assessed patient and informed treating physicianh | No | High |
Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]
Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems
The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).
| Source | Setting | Alert Threshold | Prevalence, % | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | LR+, (95% CI) | LR, (95% CI) |
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
| Hooper et al., 201210 | MICU | 2 SIRS criteriaa | 36.3 | 98.9 (95.799.8) | 18.1 (14.222.9) | 40.7 (36.145.5) | 96.7 (87.599.4) | 1.21 (1.14‐1.27) | 0.06 (0.01‐0.25) |
| Meurer et al., 200913 | ED | 2 SIRS criteria | 27.8 | 36.2 (25.348.8) | 79.9 (73.185.3) | 41.0 (28.854.3) | 76.5 (69.682.2) | 1.80 (1.17‐2.76) | 0.80 (0.67‐0.96) |
| Nelson et al., 201114 | ED | 2 SIRS criteria and 2 SBP measurements<90 mm Hg | 0.8 | 63.6 (31.687.8) | 99.6 (99.099.8) | 53.8 (26.179.6) | 99.7 (99.299.9) | 145.8 (58.4364.1) | 0.37 (0.17‐0.80) |
| Nguyen et al., 201415 | ED | 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) | Unable to estimateb | Unable to estimateb | Unable to estimateb | 44.7 (41.248.2) | 100.0c (98.8100.0) | Unable to estimateb | Unable to estimateb |
| Thiel et al., 201016 | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsd | 4.7 | 17.1 (15.119.3) | 96.7 (96.596.9) | 20.5 (18.223.0) | 95.9 (95.796.2) | 5.22 (4.56‐5.98) | 0.86 (0.84‐0.88) |
All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).
There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.
Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]
| Study | Patient Selection | Index Test | Reference Standard | Flow and Timing |
|---|---|---|---|---|
| ||||
| Hooper et al., 201210 | +++ | +++ | ++b | +++ |
| Meurer et al., 200913 | +++ | +++ | ++b | +c |
| Nelson et al., 201114 | +++ | +++ | ++b | +++ |
| Nguyen et al., 201415 | +d | +++ | +e | +++ |
| Thiel et al., 201016 | +++ | +++ | +f | +g |
Effectiveness of Automated Electronic Sepsis Alert Systems
Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]
Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]
| Source | Outcomes Evaluated | Key Findings | Quality |
|---|---|---|---|
| |||
| Hooper et al., 201210 | Primary: time to receipt of antibiotic (new or changed) | No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) | High |
| Secondary: sepsis‐related process measures and outcomes | No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3) | ||
| Sawyer et al., 201119 | Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert | Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) | High |
| Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality | Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7) | ||
| Berger et al., 201017 | Primary: lactate collection in ED | Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) | Low |
| Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients | Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6) | ||
| McRee et al., 201418 | Stage of sepsis, length of stay, mortality, discharge location | Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) | Low |
| Nelson et al., 201114 | Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation | Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). | Low |
Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.
None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]
DISCUSSION
The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.
We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]
The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]
In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.
Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.
CONCLUSION
Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.
Acknowledgements
The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.
Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.
- , . National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
- , , , . Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8.
- , , , . The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554.
- , , , et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
- , , , et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693.
- , . Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19.
- , , , , , . Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936.
- , , , et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303.
- , , , et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101.
- , , , et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536.
- , , , et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513–523.
- , , , et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45.
- , , , . Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504.
- , , , et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
- , , , , , . Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25.
- , , , , . A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394–407.
- , , , , . The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546–549.
- , , , et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473.
- . The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64–S74.
- , , , et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
- , , , , , . Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451–456.
- , . How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243.
- , , , et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806–810.
- , , , et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331–345.
Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]
Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).
However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.
MATERIALS AND METHODS
Data Sources and Search Strategies
We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).
Study Selection
Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.
Data Extraction and Quality Assessment
Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]
For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]
Data Synthesis
To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).
RESULTS
Description of Studies
Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).
| Source | Site No./Type | Setting | Alert Threshold | Gold Standard Definition | Gold Standard Measurement | No. | Study Qualitya |
|---|---|---|---|---|---|---|---|
| |||||||
| Hooper et al., 201210 | 1/academic | MICU | 2 SIRS criteriab | Reviewer judgment, not otherwise specified | Chart review | 560 | High |
| Meurer et al., 200913 | 1/academic | ED | 2 SIRS criteria | Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria | Chart review | 248 | Low |
| Nelson J. et al., 201114 | 1/academic | ED | 2 SIRS criteria and 2 SBP measurements <90 mm Hg | Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved | Chart review | 1,386 | High |
| Nguyen et al., 201415 | 1/academic | ED | 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) | Reviewer judgment to confirm SIRS, shock, and presence of a serious infection | Chart review | 1,095 | Low |
| Thiel et al., 201016 | 1/academic | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsc | Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy | ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer | 27,674 | Low |
| Source | Design | Site No./ Type | Setting | No. | Alert System Type | Alert Threshold | Alert Notificationa | Treatment Recommendation | Study Qualityb |
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
| Berger et al., 201017 | Before‐after (6 months pre and 6 months post) | 1/academic | ED | 5796c | CPOE system | 2 SIRS criteria | CPOE passive alert | Yes: lactate collection | Low |
| Hooper et al., 201210 | RCT | 1/academic | MICU | 443 | EHR | 2 SIRS criteriad | Text page and EHR passive alert | No | High |
| McRee et al., 201418 | Before‐after (6 months pre and 6 months post) | 1/academic | Wards | 171e | EHR | 2 SIRS criteria | Notified nurse, specifics unclear | No, but the nurse completed a sepsis risk evaluation flow sheet | Low |
| Nelson et al., 201114 | Before‐after (3 months pre and 3 months post) | 1/academic | ED | 184f | EHR | 2 SIRS criteria and 2 or more SBP readings <90 mm Hg | Text page and EHR passive alert | Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others | Low |
| Sawyer et al., 201119 | Prospective, nonrandomized (2 intervention and 4 control wards) | 1/academic | Wards | 300 | EHR | Recursive partitioning regression tree algorithm including vitals and lab valuesg | Text page to charge nurse who then assessed patient and informed treating physicianh | No | High |
Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]
Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems
The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).
| Source | Setting | Alert Threshold | Prevalence, % | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | LR+, (95% CI) | LR, (95% CI) |
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
| Hooper et al., 201210 | MICU | 2 SIRS criteriaa | 36.3 | 98.9 (95.799.8) | 18.1 (14.222.9) | 40.7 (36.145.5) | 96.7 (87.599.4) | 1.21 (1.14‐1.27) | 0.06 (0.01‐0.25) |
| Meurer et al., 200913 | ED | 2 SIRS criteria | 27.8 | 36.2 (25.348.8) | 79.9 (73.185.3) | 41.0 (28.854.3) | 76.5 (69.682.2) | 1.80 (1.17‐2.76) | 0.80 (0.67‐0.96) |
| Nelson et al., 201114 | ED | 2 SIRS criteria and 2 SBP measurements<90 mm Hg | 0.8 | 63.6 (31.687.8) | 99.6 (99.099.8) | 53.8 (26.179.6) | 99.7 (99.299.9) | 145.8 (58.4364.1) | 0.37 (0.17‐0.80) |
| Nguyen et al., 201415 | ED | 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) | Unable to estimateb | Unable to estimateb | Unable to estimateb | 44.7 (41.248.2) | 100.0c (98.8100.0) | Unable to estimateb | Unable to estimateb |
| Thiel et al., 201016 | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsd | 4.7 | 17.1 (15.119.3) | 96.7 (96.596.9) | 20.5 (18.223.0) | 95.9 (95.796.2) | 5.22 (4.56‐5.98) | 0.86 (0.84‐0.88) |
All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).
There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.
Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]
| Study | Patient Selection | Index Test | Reference Standard | Flow and Timing |
|---|---|---|---|---|
| ||||
| Hooper et al., 201210 | +++ | +++ | ++b | +++ |
| Meurer et al., 200913 | +++ | +++ | ++b | +c |
| Nelson et al., 201114 | +++ | +++ | ++b | +++ |
| Nguyen et al., 201415 | +d | +++ | +e | +++ |
| Thiel et al., 201016 | +++ | +++ | +f | +g |
Effectiveness of Automated Electronic Sepsis Alert Systems
Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]
Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]
| Source | Outcomes Evaluated | Key Findings | Quality |
|---|---|---|---|
| |||
| Hooper et al., 201210 | Primary: time to receipt of antibiotic (new or changed) | No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) | High |
| Secondary: sepsis‐related process measures and outcomes | No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3) | ||
| Sawyer et al., 201119 | Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert | Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) | High |
| Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality | Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7) | ||
| Berger et al., 201017 | Primary: lactate collection in ED | Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) | Low |
| Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients | Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6) | ||
| McRee et al., 201418 | Stage of sepsis, length of stay, mortality, discharge location | Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) | Low |
| Nelson et al., 201114 | Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation | Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). | Low |
Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.
None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]
DISCUSSION
The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.
We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]
The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]
In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.
Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.
CONCLUSION
Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.
Acknowledgements
The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.
Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.
Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.[1] Rates of sepsis and sepsis‐related mortality are rising in the United States.[2, 3] Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.[4, 5, 6] Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.[4] Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.[7, 8] As such, delays in administering evidence‐based therapies are common.[8, 9]
Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real time or near real time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (eg, through electronic health record [EHR], e‐mail, or pager alerts).
However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our 2 objectives were: (1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real‐time or near real‐time, and (2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.
MATERIALS AND METHODS
Data Sources and Search Strategies
We searched PubMed MEDLINE, Embase, The Cochrane Library, and the Cumulative Index to Nursing and Allied Health Literature from database inception through June 27, 2014, for all studies that contained the following 3 concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5; Thomson‐Reuters Corp., New York, NY) (see Supporting Information, Appendix, in the online version of this article for our complete search strategy).
Study Selection
Two authors (A.N.M. and O.K.N.) reviewed the citation titles, abstracts, and full‐text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: (1) adult patients (aged 18 years) receiving care either in the emergency department or hospital, (2) outcomes of interest including diagnostic accuracy in identification of sepsis, and/or effectiveness of sepsis alerts on process measures and clinical outcomes evaluated using empiric data, and (3) sepsis alert systems used real time or near real time electronically available data to enable proactive, timely management. We excluded studies that: (1) tested the effect of other electronic interventions that were not sepsis alerts (ie, computerized order sets) for sepsis management; (2) studies solely focused on detecting and treating central line‐associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device‐related infections; and (3) studies evaluating the effectiveness of sepsis alerts without a control group.
Data Extraction and Quality Assessment
Two reviewers (A.N.M. and O.K.N.) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in 1 study were resolved by contacting the corresponding author.[10]
For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies revised tool.[11] For studies evaluating the effectiveness of sepsis alert systems, studies were considered high quality if a contemporaneous control group was present to account for temporal trends (eg, randomized controlled trial or observational analysis with a concurrent control). Fair‐quality studies were before‐and‐after studies that adjusted for potential confounders between time periods. Low‐quality studies included those that did not account for temporal trends, such as before‐and‐after studies using only historical controls without adjustment. Studies that did not use an intention‐to‐treat analysis were also considered low quality. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.[12]
Data Synthesis
To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta‐analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR). Effectiveness was assessed by changes in sepsis care process measures (ie, time to antibiotics) and outcomes (length of stay, mortality).
RESULTS
Description of Studies
Of 1293 titles, 183 qualified for abstract review, 84 for full‐text review, and 8 articles met our inclusion criteria (see Supporting Figure in the online version of this article). Five articles evaluated the diagnostic accuracy of sepsis identification,[10, 13, 14, 15, 16] and 5 articles[10, 14, 17, 18, 19] evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single‐site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).
| Source | Site No./Type | Setting | Alert Threshold | Gold Standard Definition | Gold Standard Measurement | No. | Study Qualitya |
|---|---|---|---|---|---|---|---|
| |||||||
| Hooper et al., 201210 | 1/academic | MICU | 2 SIRS criteriab | Reviewer judgment, not otherwise specified | Chart review | 560 | High |
| Meurer et al., 200913 | 1/academic | ED | 2 SIRS criteria | Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria | Chart review | 248 | Low |
| Nelson J. et al., 201114 | 1/academic | ED | 2 SIRS criteria and 2 SBP measurements <90 mm Hg | Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved | Chart review | 1,386 | High |
| Nguyen et al., 201415 | 1/academic | ED | 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) | Reviewer judgment to confirm SIRS, shock, and presence of a serious infection | Chart review | 1,095 | Low |
| Thiel et al., 201016 | 1/academic | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsc | Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy | ICD‐9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer | 27,674 | Low |
| Source | Design | Site No./ Type | Setting | No. | Alert System Type | Alert Threshold | Alert Notificationa | Treatment Recommendation | Study Qualityb |
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
| Berger et al., 201017 | Before‐after (6 months pre and 6 months post) | 1/academic | ED | 5796c | CPOE system | 2 SIRS criteria | CPOE passive alert | Yes: lactate collection | Low |
| Hooper et al., 201210 | RCT | 1/academic | MICU | 443 | EHR | 2 SIRS criteriad | Text page and EHR passive alert | No | High |
| McRee et al., 201418 | Before‐after (6 months pre and 6 months post) | 1/academic | Wards | 171e | EHR | 2 SIRS criteria | Notified nurse, specifics unclear | No, but the nurse completed a sepsis risk evaluation flow sheet | Low |
| Nelson et al., 201114 | Before‐after (3 months pre and 3 months post) | 1/academic | ED | 184f | EHR | 2 SIRS criteria and 2 or more SBP readings <90 mm Hg | Text page and EHR passive alert | Yes: fluid resuscitation, blood culture collection, antibiotic administration, among others | Low |
| Sawyer et al., 201119 | Prospective, nonrandomized (2 intervention and 4 control wards) | 1/academic | Wards | 300 | EHR | Recursive partitioning regression tree algorithm including vitals and lab valuesg | Text page to charge nurse who then assessed patient and informed treating physicianh | No | High |
Among the 8 included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.[10, 13, 17, 18]
Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems
The prevalence of sepsis varied substantially between the studies depending on the gold standard definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The 2 studies[14, 16] that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8%4.7%) compared to the 2 studies[10, 13] that defined sepsis as having only 2 or more SIRS criteria with a presumed diagnosis of an infection (27.8%32.5%).
| Source | Setting | Alert Threshold | Prevalence, % | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | LR+, (95% CI) | LR, (95% CI) |
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
| Hooper et al., 201210 | MICU | 2 SIRS criteriaa | 36.3 | 98.9 (95.799.8) | 18.1 (14.222.9) | 40.7 (36.145.5) | 96.7 (87.599.4) | 1.21 (1.14‐1.27) | 0.06 (0.01‐0.25) |
| Meurer et al., 200913 | ED | 2 SIRS criteria | 27.8 | 36.2 (25.348.8) | 79.9 (73.185.3) | 41.0 (28.854.3) | 76.5 (69.682.2) | 1.80 (1.17‐2.76) | 0.80 (0.67‐0.96) |
| Nelson et al., 201114 | ED | 2 SIRS criteria and 2 SBP measurements<90 mm Hg | 0.8 | 63.6 (31.687.8) | 99.6 (99.099.8) | 53.8 (26.179.6) | 99.7 (99.299.9) | 145.8 (58.4364.1) | 0.37 (0.17‐0.80) |
| Nguyen et al., 201415 | ED | 2 SIRS criteria and 1 sign of shock (SBP 90 mm Hg or lactic acid 2.0 mmol/L) | Unable to estimateb | Unable to estimateb | Unable to estimateb | 44.7 (41.248.2) | 100.0c (98.8100.0) | Unable to estimateb | Unable to estimateb |
| Thiel et al., 201016 | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsd | 4.7 | 17.1 (15.119.3) | 96.7 (96.596.9) | 20.5 (18.223.0) | 95.9 (95.796.2) | 5.22 (4.56‐5.98) | 0.86 (0.84‐0.88) |
All alert systems had suboptimal PPV (20.5%‐53.8%). The 2 studies that designed the sepsis alert to activate by SIRS criteria alone[10, 13] had a positive predictive value of 41% and a positive LR of 1.21 to 1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.[10] that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR=0.06) compared to a similar alert system used by Meurer et al.[13] that studied patients in the ED (NPV=76.5%, negative LR=0.80).
There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.[13, 14] The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV=53.8%, LR+=145.8; NPV=99.7%, LR=0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV=41.0%, LR+=1.80; NPV=76.5%, LR=0.80). Furthermore, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.
Two studies evaluating the diagnostic accuracy of the alert system were deemed to be high quality (Table 4). Three studies were considered low quality1 study did not include all patients in their assessment of diagnostic accuracy13; 1 study consecutively selected alert cases but randomly selected nonalert cases, greatly limiting the assessment of diagnostic accuracy15; and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge International Classification of Diseases, Ninth Revision diagnoses without chart review), with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real‐time data).[16]
| Study | Patient Selection | Index Test | Reference Standard | Flow and Timing |
|---|---|---|---|---|
| ||||
| Hooper et al., 201210 | +++ | +++ | ++b | +++ |
| Meurer et al., 200913 | +++ | +++ | ++b | +c |
| Nelson et al., 201114 | +++ | +++ | ++b | +++ |
| Nguyen et al., 201415 | +d | +++ | +e | +++ |
| Thiel et al., 201016 | +++ | +++ | +f | +g |
Effectiveness of Automated Electronic Sepsis Alert Systems
Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, 2 studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),[10, 14] 1 study notified the provider by a passive electronic alert alone,[17] and 1 study only employed an automated text page.[19] Furthermore, if the sepsis alert was activated, 2 studies suggested specific clinical management decisions,[14, 17] 2 studies left clinical management decisions solely to the discretion of the treating provider,[10, 19] and 1 study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.[18]
Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be high‐quality studies based on the use of a contemporaneous control group to account for temporal trends and an intention‐to‐treat analysis.[10, 19] The 2 studies evaluating the effectiveness of a sepsis alert system in the ED were considered low quality due to before‐and‐after designs without an intention‐to‐treat analysis.[14, 17]
| Source | Outcomes Evaluated | Key Findings | Quality |
|---|---|---|---|
| |||
| Hooper et al., 201210 | Primary: time to receipt of antibiotic (new or changed) | No difference (6.1 hours for control vs 6.0 hours for intervention, P=0.95) | High |
| Secondary: sepsis‐related process measures and outcomes | No difference in amount of 6 hour IV fluid administration (964 mL vs 1,019 mL, P=0.6), collection of blood cultures (adjusted HR 1.01; 95% CI, 0.76 to 1.35), collection of lactate (adjusted HR 0.84; 95% CI, 0.54 to 1.30), ICU length of stay (3.0 vs 3.0 days, P=0.2), hospital length of stay (4.7 vs 5.7 days, P=0.08), and hospital mortality (10% for control vs 14% for intervention, P=0.3) | ||
| Sawyer et al., 201119 | Primary: sepsis‐related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert | Increases in receiving 1 measure (56% for control vs 71% for intervention, P=0.02), antibiotic escalation (24% vs 36%, P=0.04), IV fluid administration (24% vs 38%, P=0.01), and oxygen therapy (8% vs 20%, P=0.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs 52%, P=0.06) and vasopressor initiation (3% vs 6%, P=0.4) | High |
| Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in‐hospital mortality | Similar rate of ICU transfer (23% for control vs 26% for intervention, P=0.6), hospital length of stay (7 vs 9 days, median, P=0.8), hospital length of stay after alert (5 vs 6 days, median, P=0.7), and in‐hospital mortality (12% vs 10%, P=0.7) | ||
| Berger et al., 201017 | Primary: lactate collection in ED | Increase in lactate collection in the ED (5.2% before vs 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0% to 9.0%) | Low |
| Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (4 mmol/L), and in‐hospital mortality among hospitalized patients | Increase in lactate collection among hospitalized patients (15.3% vs 34.2%, absolute increase of 18.9%, 95% CI, 15.0% to 22.8%); decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% CI, 15.8% to 0.6%), and no significant difference in mortality (5.7% vs 5.2%, absolute decrease of 0.5%, 95% CI, 1.6% to 2.6%, P=0.6) | ||
| McRee et al., 201418 | Stage of sepsis, length of stay, mortality, discharge location | Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs 21.9% after, P>0.05); no difference in length‐of‐stay (8.5 days before vs 8.7 days after, P>0.05). Decrease in mortality (9.3% before vs 1.0% after, P<0.05) and proportion of patients discharged home (25.3% before vs 49.0% after, P<0.05) | Low |
| Nelson et al., 201114 | Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation | Increases in blood culture collection (OR 2.9; 95% CI, 1.1 to 7.7) and CXR (OR 3.2; 95% CI, 1.1 to 9.5); nonsignificant increases in lactate collection (OR 1.7; 95% CI, 0.9 to 3.2) and antibiotic administration (OR 2.8; 95% CI, 0.9 to 8.3). Only blood cultures were collected in a more timely manner (median of 86 minutes before vs 81 minutes after alert implementation, P=0.03). | Low |
Neither of the 2 high‐quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.[10, 19] The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In a randomized controlled trial of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.[10] However, in a well‐designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.[19] Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,[14, 17] but without improvement in mortality.[17] The single study that showed improvement in clinical outcomes (in‐hospital mortality and disposition location) was of low quality due to the prestudypoststudy design without adjustment for potential confounders and lack of an intention‐to‐treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).[18] Additionally, the preintervention group had a higher proportion of individuals with septic shock compared to the postintervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the 2 groups rather than due to the intervention.
None of the studies included in this review explicitly reported on the potential harms (eg, excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.[10, 19] Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs 14.8%, absolute decrease of 7.6%, 95% confidence interval, 15.8% to 0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.[10] Furthermore, 3 studies did not perform intention‐to‐treat analyses; rather, they included only patients who triggered the alert and also had provider‐suspected or confirmed sepsis,[14, 17] or had a discharge diagnosis for sepsis.[18]
DISCUSSION
The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.
We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (eg, SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with noninfectious diseases,[20] sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of alert fatigueexcessive electronic warnings resulting in physicians disregarding clinically useful alerts.[21] The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average 6 alarms per hour with only 15% of alarms considered to be clinically relevant.[22]
The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.[23] First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a ceiling effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,[10] so there likely is a high noise‐to‐signal ratio with resultant alert fatigue.[21]
In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there are little data to describe the optimal design of sepsis alerts[24, 25] or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (eg, direct prompting with specific clinical management recommendations) or the configuration of the alert (eg, interruptive alert or informational).[24, 25] Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems that also alerted other providers (eg, nurses or rapid response teams). Few studies provided data on harms of these systems (eg, excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (eg, randomized or quasiexperimental designs) or used an intention‐to‐treat approach. Studies that exclude false positive alerts in analyses could bias estimates toward making sepsis alert systems appear more effective than they actually were. Finally, although presumably, deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical benefits and cost‐effectiveness of implementing automated sepsis alerts.
Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non‐English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing 1 of our prespecified research questions compared to the number of peer‐reviewed publications identified by our search strategy.
CONCLUSION
Automated electronic sepsis alert systems have promise in delivering early goal‐directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor PPV and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems that avoid the potential for alert fatigue while improving outcomes.
Acknowledgements
The authors thank Gloria Won, MLIS, for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement.
Disclosures: Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025‐07‐00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428‐01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372). Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: all authors. Acquisition of data: Makam and Nguyen. Analysis and interpretation of data: all authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: all authors. Statistical analysis: Makam and Nguyen. The authors have no conflicts of interest to disclose.
- , . National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
- , , , . Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8.
- , , , . The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554.
- , , , et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
- , , , et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693.
- , . Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19.
- , , , , , . Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936.
- , , , et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303.
- , , , et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101.
- , , , et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536.
- , , , et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513–523.
- , , , et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45.
- , , , . Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504.
- , , , et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
- , , , , , . Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25.
- , , , , . A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394–407.
- , , , , . The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546–549.
- , , , et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473.
- . The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64–S74.
- , , , et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
- , , , , , . Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451–456.
- , . How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243.
- , , , et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806–810.
- , , , et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331–345.
- , . National inpatient hospital costs: the most expensive conditions by payer, 2011: statistical brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Rockville, MD: Agency for Healthcare Research and Quality; 2013.
- , , , . Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8.
- , , , . The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554.
- , , , et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377.
- , , , et al. A randomized trial of protocol‐based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693.
- , . Implementation of early goal‐directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19.
- , , , , , . Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936.
- , , , et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303.
- , , , et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101.
- , , , et al. QUADAS‐2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536.
- , , , et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health‐care program. J Clin Epidemiol. 2010;63(5):513–523.
- , , , et al. Real‐time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45.
- , , , . Prospective trial of real‐time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504.
- , , , et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
- , , , , , . Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25.
- , , , , . A Computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394–407.
- , , , , . The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546–549.
- , , , et al. Implementation of a real‐time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473.
- . The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(suppl 1):S64–S74.
- , , , et al. Overrides of medication‐related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(3):487–491.
- , , , , , . Intensive care unit alarms–how many do we need? Crit Care Med. 2010;38(2):451–456.
- , . How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243.
- , , , et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(pt 2):806–810.
- , , , et al. Best practices in clinical decision support: the case of preventive care reminders. Appl Clin Inform. 2010;1(3):331–345.