Kirsten N Kangelaris, MD, MAS

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

METHODS

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

RESULTS

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

The AUROCs of the scores were statistically significantly different for both in-hospital mortality (P = .0026) and ED-to-ICU admission (P < .0001). As a sensitivity analysis, we varied the binary cut-points of qSOFA, the Shock Index, and NEWS2 and calculated test characteristics within our study cohort (Appendix). For our institution, the qSOFA cut-point with the highest AUROC would be qSOFA > 0 for both in-hospital mortality (AUROC, 0.699; 95% CI, 0.687-0.711) and ED-to-ICU admission (AUROC, 0.716; 95% CI, 0.707-0.724), with 36.5% of the cohort meeting qSOFA. The NEWS2 cut-point with the highest AUROC would be NEWS2 ≥7 for both in-hospital mortality (AUROC, 0.653; 95% CI, 0.640-0.666) and ED-to-ICU admission (AUROC, 0.677; 95% CI, 0.668-0.686), with 20.3% of the cohort meeting NEWS2 at this cut-point. The standard Shock Index cut-point ≥0.7 exhibited the highest AUROC for in-hospital mortality and ED-to-ICU admission at our institution.

DISCUSSION

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

CONCLUSION

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

Acknowledgment

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

METHODS

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

RESULTS

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

The AUROCs of the scores were statistically significantly different for both in-hospital mortality (P = .0026) and ED-to-ICU admission (P < .0001). As a sensitivity analysis, we varied the binary cut-points of qSOFA, the Shock Index, and NEWS2 and calculated test characteristics within our study cohort (Appendix). For our institution, the qSOFA cut-point with the highest AUROC would be qSOFA > 0 for both in-hospital mortality (AUROC, 0.699; 95% CI, 0.687-0.711) and ED-to-ICU admission (AUROC, 0.716; 95% CI, 0.707-0.724), with 36.5% of the cohort meeting qSOFA. The NEWS2 cut-point with the highest AUROC would be NEWS2 ≥7 for both in-hospital mortality (AUROC, 0.653; 95% CI, 0.640-0.666) and ED-to-ICU admission (AUROC, 0.677; 95% CI, 0.668-0.686), with 20.3% of the cohort meeting NEWS2 at this cut-point. The standard Shock Index cut-point ≥0.7 exhibited the highest AUROC for in-hospital mortality and ED-to-ICU admission at our institution.

DISCUSSION

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

CONCLUSION

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

Acknowledgment

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

METHODS

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

RESULTS

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

The AUROCs of the scores were statistically significantly different for both in-hospital mortality (P = .0026) and ED-to-ICU admission (P < .0001). As a sensitivity analysis, we varied the binary cut-points of qSOFA, the Shock Index, and NEWS2 and calculated test characteristics within our study cohort (Appendix). For our institution, the qSOFA cut-point with the highest AUROC would be qSOFA > 0 for both in-hospital mortality (AUROC, 0.699; 95% CI, 0.687-0.711) and ED-to-ICU admission (AUROC, 0.716; 95% CI, 0.707-0.724), with 36.5% of the cohort meeting qSOFA. The NEWS2 cut-point with the highest AUROC would be NEWS2 ≥7 for both in-hospital mortality (AUROC, 0.653; 95% CI, 0.640-0.666) and ED-to-ICU admission (AUROC, 0.677; 95% CI, 0.668-0.686), with 20.3% of the cohort meeting NEWS2 at this cut-point. The standard Shock Index cut-point ≥0.7 exhibited the highest AUROC for in-hospital mortality and ED-to-ICU admission at our institution.

DISCUSSION

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

CONCLUSION

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

Acknowledgment

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

METHODS

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

 

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

 

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

 

RESULTS

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

In a subgroup analysis using the fully adjusted model restricted to patients who were mechanically ventilated during hospitalization, the association between LEP and mortality was no longer present (OR 1.15; 95% CI: 0.76-1.72; P = .51).

 

DISCUSSION

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

 

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

METHODS

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

 

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

 

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

 

RESULTS

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

In a subgroup analysis using the fully adjusted model restricted to patients who were mechanically ventilated during hospitalization, the association between LEP and mortality was no longer present (OR 1.15; 95% CI: 0.76-1.72; P = .51).

 

DISCUSSION

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

 

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

METHODS

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

 

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

 

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

 

RESULTS

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

In a subgroup analysis using the fully adjusted model restricted to patients who were mechanically ventilated during hospitalization, the association between LEP and mortality was no longer present (OR 1.15; 95% CI: 0.76-1.72; P = .51).

 

DISCUSSION

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

 

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.