Priya A Prasad, PhD, MPH

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

METHODS

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

RESULTS

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

The AUROCs of the scores were statistically significantly different for both in-hospital mortality (P = .0026) and ED-to-ICU admission (P < .0001). As a sensitivity analysis, we varied the binary cut-points of qSOFA, the Shock Index, and NEWS2 and calculated test characteristics within our study cohort (Appendix). For our institution, the qSOFA cut-point with the highest AUROC would be qSOFA > 0 for both in-hospital mortality (AUROC, 0.699; 95% CI, 0.687-0.711) and ED-to-ICU admission (AUROC, 0.716; 95% CI, 0.707-0.724), with 36.5% of the cohort meeting qSOFA. The NEWS2 cut-point with the highest AUROC would be NEWS2 ≥7 for both in-hospital mortality (AUROC, 0.653; 95% CI, 0.640-0.666) and ED-to-ICU admission (AUROC, 0.677; 95% CI, 0.668-0.686), with 20.3% of the cohort meeting NEWS2 at this cut-point. The standard Shock Index cut-point ≥0.7 exhibited the highest AUROC for in-hospital mortality and ED-to-ICU admission at our institution.

DISCUSSION

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

CONCLUSION

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

Acknowledgment

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

METHODS

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

RESULTS

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

The AUROCs of the scores were statistically significantly different for both in-hospital mortality (P = .0026) and ED-to-ICU admission (P < .0001). As a sensitivity analysis, we varied the binary cut-points of qSOFA, the Shock Index, and NEWS2 and calculated test characteristics within our study cohort (Appendix). For our institution, the qSOFA cut-point with the highest AUROC would be qSOFA > 0 for both in-hospital mortality (AUROC, 0.699; 95% CI, 0.687-0.711) and ED-to-ICU admission (AUROC, 0.716; 95% CI, 0.707-0.724), with 36.5% of the cohort meeting qSOFA. The NEWS2 cut-point with the highest AUROC would be NEWS2 ≥7 for both in-hospital mortality (AUROC, 0.653; 95% CI, 0.640-0.666) and ED-to-ICU admission (AUROC, 0.677; 95% CI, 0.668-0.686), with 20.3% of the cohort meeting NEWS2 at this cut-point. The standard Shock Index cut-point ≥0.7 exhibited the highest AUROC for in-hospital mortality and ED-to-ICU admission at our institution.

DISCUSSION

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

CONCLUSION

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

Acknowledgment

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is the leading cause of in-hospital mortality in the United States.¹ Sepsis is present on admission in 85% of cases, and each hour delay in antibiotic treatment is associated with 4% to 7% increased odds of mortality.^2,3 Prompt identification and treatment of sepsis is essential for reducing morbidity and mortality, but identifying sepsis during triage is challenging.²

Risk stratification scores that rely solely on data readily available at the bedside have been developed to quickly identify those at greatest risk of poor outcomes from sepsis in real time. The quick Sequential Organ Failure Assessment (qSOFA) score, the National Early Warning System (NEWS2), and the Shock Index are easy-to-calculate measures that use routinely collected clinical data that are not subject to laboratory delay. These scores can be incorporated into electronic health record (EHR)-based alerts and can be calculated longitudinally to track the risk of poor outcomes over time. qSOFA was developed to quantify patient risk at bedside in non-intensive care unit (ICU) settings, but there is no consensus about its ability to predict adverse outcomes such as mortality and ICU admission.^4-6 The United Kingdom’s National Health Service uses NEWS2 to identify patients at risk for sepsis.⁷ NEWS has been shown to have similar or better sensitivity in identifying poorer outcomes in sepsis patients compared with systemic inflammatory response syndrome (SIRS) criteria and qSOFA.^4,8-11 However, since the latest update of NEWS2 in 2017, there has been little study of its predictive ability. The Shock Index is a simple bedside score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before systemic shock onset. Although it was not developed for infection and has not been regularly applied in the sepsis literature, the Shock Index might be useful for identifying patients at increased risk of poor outcomes. Patients with higher and sustained Shock Index scores are more likely to experience morbidity, such as hyperlactatemia, vasopressor use, and organ failure, and also have an increased risk of mortality.^12-14

Although the predictive abilities of these bedside risk stratification scores have been assessed individually using standard binary cut-points, the comparative performance of qSOFA, the Shock Index, and NEWS2 has not been evaluated in patients presenting to an emergency department (ED) with suspected sepsis. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, and NEWS2 calculated at ED triage for predicting in-hospital mortality and ED-to-ICU admission in patients with suspected sepsis to help health systems and providers select screening measures.

METHODS

Design and Setting

We conducted a retrospective cohort study of ED patients who presented with suspected sepsis to the University of California San Francisco (UCSF) Helen Diller Medical Center at Parnassus Heights between June 1, 2012, and December 31, 2018. Our institution is a 785-bed academic teaching hospital with approximately 30,000 ED encounters per year. The study was approved with a waiver of informed consent by the UCSF Human Research Protection Program.

Participants

We use an Epic-based EHR platform (Epic 2017, Epic Systems Corporation) for clinical care, which was implemented on June 1, 2012. All data elements were obtained from Clarity, the relational database that stores Epic’s inpatient data. The study included encounters for patients age ≥18 years who had blood cultures ordered within 24 hours of ED presentation and administration of intravenous antibiotics within 24 hours. Repeat encounters were treated independently in our analysis.

Outcomes and Measures

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict in-hospital mortality and admission to the ICU from the ED (ED-to-ICU admission). We used the most abnormal vital signs and clinical assessments gathered within 30 minutes of ED presentation to identify patients who were qSOFA-positive, Shock Index-positive, and NEWS2-positive based on standard cut-points of risk. Data elements used to calculate qSOFA, Shock Index, and NEWS2 included blood pressure, heart rate, respiratory rate, Glasgow Coma Scale (GCS) score, oxygen saturation, requirement for supplemental oxygen, and temperature (Table 1). Patients were considered positive if they had a qSOFAscore ≥2, Shock Index of >0.7, and NEWS2 ≥5 based on triage vital signs.^7,15,16 We considered patients to have altered mental status, a criterion used for NEWS2, if they had a GCS score <15 instead of using the “alert, verbal, confusion, pain, unresponsive” scale, which is not captured in our EHR, a method that has been used in earlier studies.^17,18 Missing assessments were considered normal. Although our primary analysis focused on the scores calculated within 30 minutes of ED presentation, we performed a sensitivity analysis examining scores calculated within 1 hour of ED presentation in the event of a delay in gathering triage vital sign data.

We compared demographic and clinical characteristics of patients who were positive for qSOFA, the Shock Index, and NEWS2. Demographic data were extracted from the EHR and included primary language, age, sex, and insurance status. All International Classification of Diseases (ICD)-9/10 diagnosis codes were pulled from Clarity billing tables. We used the Elixhauser comorbidity groupings¹⁹ of ICD-9/10 codes present on admission to identify preexisting comorbidities and underlying organ dysfunction. To estimate burden of comorbid illnesses, we calculated the validated van Walraven comorbidity index,²⁰ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities. Admission level of care (acute, stepdown, or intensive care) was collected for inpatient admissions to assess initial illness severity.²¹ We also evaluated discharge disposition and in-hospital mortality. Index blood culture results were collected, and dates and timestamps of mechanical ventilation, fluid, vasopressor, and antibiotic administration were obtained for the duration of the encounter.

UCSF uses an automated, real-time, algorithm-based severe sepsis alert that is triggered when a patient meets ≥2 SIRS criteria and again when the patient meets severe sepsis or septic shock criteria (ie, ≥2 SIRS criteria in addition to end-organ dysfunction and/or fluid nonresponsive hypotension). This sepsis screening alert was in use for the duration of our study.²²

Statistical Analysis

We performed a subgroup analysis among those who were diagnosed with sepsis, according to the 2016 Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Sepsis is defined as a change in Sequential Organ Failure Assessment (SOFA) score of ≥2 points within the first 48 hours.²³ Additionally, patients meeting Sepsis-3 criteria needed to (1) receive ≥4 days of sequential antibiotic therapy or experience death or discharge to hospice before 4 days of antibiotic therapy or (2) have a validated sepsis discharge billing code. These parameters were added to increase the specificity of our sample.²⁴

All statistical analyses were conducted using Stata 14 (StataCorp). We summarized differences in demographic and clinical characteristics among the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for one or more scores. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for each score to predict in-hospital mortality and ED-to-ICU admission. To allow comparison with other studies, we also created a composite outcome of either in-hospital mortality or ED-to-ICU admission. To assess score discrimination to predict in-hospital mortality and ED-to-ICU admission, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% CI using the “roctab” command considering a binary cut-point, as well as the full range of scores measured in the cohort. The AUROC range from 0.50 to 1.00 and a score in the 0.70 to 0.80 range can be considered fair.²⁵ We assessed significant differences between severity score AUROCs using the DeLong method²⁶ implemented through Stata 14’s “roccomp” command. As a sensitivity analysis, we explored whether the standard cut-points for qSOFA, the Shock Index, and NEWS2 provided the highest AUROC in our population by calculating test characteristics for several score cut-points.

RESULTS

Within our sample 23,837 ED patients had blood cultures ordered within 24 hours of ED presentation and were considered to have suspected sepsis. The mean age of the cohort was 60.8 years, and 1,612 (6.8%) had positive blood cultures. A total of 12,928 patients (54.2%) were found to have sepsis. We documented 1,427 in-hospital deaths (6.0%) and 3,149 (13.2%) ED-to-ICU admissions. At ED triage 1,921 (8.1%) were qSOFA-positive, 4,273 (17.9%) were Shock Index-positive, and 11,832 (49.6%) were NEWS2-positive. At ED triage, blood pressure, heart rate, respiratory rate, and oxygen saturated were documented in >99% of patients, 93.5% had temperature documented, and 28.5% had GCS recorded. If the window of assessment was widened to 1 hour, GCS was only documented among 44.2% of those with suspected sepsis.

Demographic Characteristics and Clinical Course

We identified significant differences when comparing demographic and clinical characteristics among patients who scored positive for the three severity measures at triage (Table 2). Although no hypothesis testing was conducted because patients could meet one or more scores, qSOFA-positive patients were older (median 70, 66, and 64 years, respectively), more likely to have Medicare as the primary payor (67.6% vs 59.7% vs 56.6%), to have chronic renal failure (26.1%, 23.1%, and 23.3%, respectively), to have a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index (median 15, 12, and 11, respectively), and to be admitted to the ICU from the ED (48.1%, 36.3%, and 21.0%, respectively) compared with those positive for the Shock Index or NEWS2.

qSOFA-positive patients received antibiotics more quickly than those who were Shock Index-positive or NEWS2-positive (median 1.5, 1.8, and 2.8 hours after admission, respectively). In addition, those who were qSOFA-positive were more likely to have a positive blood culture (10.9%, 9.4%, and 8.5%, respectively) and to receive an EHR-based diagnosis of sepsis (77.0%, 69.6%, and 60.9%, respectively) than those who were Shock Index- or NEWS2-positive. Those who were qSOFA-positive also were more likely to be mechanically ventilated during their hospital stay (25.4%, 19.2%, and 10.8%, respectively) and to receive vasopressors (33.5%, 22.5%, and 12.2%, respectively). In-hospital mortality also was more common among those who were qSOFA-positive at triage (23.4%, 15.3%, and 9.2%, respectively).

Because both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n = 6,794). These patients were older (median age 63 and 61 years, P < .0001), more likely to be male (54.9% and 53.4%, P = .0031), more likely to have renal failure (22.8% and 20.1%, P < .0001), more likely to have liver disease (14.2% and 12.8%, P = .006), had a higher van Walraven comorbidity score on presentation (median 10 and 8, P < .0001), and were more likely to go directly to the ICU from the ED (20.2% and 10.6%, P < .0001). However, among the 6,397 GCS scores documented at triage, only 1,579 (24.7%) were abnormal.

Test Characteristics of qSOFA, Shock Index, and NEWS2 for Predicting In-hospital Mortality and ED-to-ICU Admission

Among 23,837 patients with suspected sepsis, NEWS2 had the highest sensitivity for predicting in-hospital mortality (76.0%; 95% CI, 73.7%-78.2%) and ED-to-ICU admission (78.9%; 95% CI, 77.5%-80.4%) but had the lowest specificity for in-hospital mortality (52.0%; 95% CI, 51.4%-52.7%) and for ED-to-ICU admission (54.8%; 95% CI, 54.1%-55.5%) (Table 3). qSOFA had the lowest sensitivity for in-hospital mortality (31.5%; 95% CI, 29.1%-33.9%) and ED-to-ICU admission (29.3%; 95% CI, 27.7%-30.9%) but the highest specificity for in-hospital mortality (93.4%; 95% CI, 93.1%-93.8%) and ED-to-ICU admission (95.2%; 95% CI, 94.9%-95.5%). The Shock Index had a sensitivity that fell between qSOFA and NEWS2 for in-hospital mortality (45.8%; 95% CI, 43.2%-48.5%) and ED-to-ICU admission (49.2%; 95% CI, 47.5%-51.0%). The specificity of the Shock Index also was between qSOFA and NEWS2 for in-hospital mortality (83.9%; 95% CI, 83.4%-84.3%) and ED-to-ICU admission (86.8%; 95% CI, 86.4%-87.3%). All three scores exhibited relatively low PPV, ranging from 9.2% to 23.4% for in-hospital mortality and 21.0% to 48.0% for ED-to-ICU triage. Conversely, all three scores exhibited relatively high NPV, ranging from 95.5% to 97.1% for in-hospital mortality and 89.8% to 94.5% for ED-to-ICU triage. The patterns in sensitivity and specificity for in-hospital mortality and ED-to-ICU admission were similar among the 12,928 patients who received an EHR-based sepsis diagnosis with the tests generally demonstrating lower specificities, higher PPVs, and lower NPVs (Table 3).

When considering a binary cutoff, the Shock Index exhibited the highest AUROC for in-hospital mortality (0.648; 95% CI, 0.635-0.662) and had a significantly higher AUROC than qSOFA (AUROC, 0.625; 95% CI, 0.612-0.637; P = .0005), but there was no difference compared with NEWS2 (AUROC, 0.640; 95% CI, 0.628-0.652; P = .2112). NEWS2 had a significantly higher AUROC than qSOFA for predicting in-hospital mortality (P = .0227). The Shock Index also exhibited the highest AUROC for ED-to-ICU admission (0.680; 95% CI, 0.617-0.689), which was significantly higher than the AUROC for qSOFA (P < .0001) and NEWS2 (P = 0.0151). NEWS2 had a significantly higher AUROC than qSOFA for predicting ED-to-ICU admission (P < .0001). Similar findings were seen in patients found to have sepsis. When considering the range of possible scores measured in our cohort, qSOFA and NEWS2 exhibited higher AUROCs for in-hospital mortality and ED-to-ICU admission than the Shock Index among patients with suspected infection and the subgroup with a sepsis diagnosis (Figure).

The AUROCs of the scores were statistically significantly different for both in-hospital mortality (P = .0026) and ED-to-ICU admission (P < .0001). As a sensitivity analysis, we varied the binary cut-points of qSOFA, the Shock Index, and NEWS2 and calculated test characteristics within our study cohort (Appendix). For our institution, the qSOFA cut-point with the highest AUROC would be qSOFA > 0 for both in-hospital mortality (AUROC, 0.699; 95% CI, 0.687-0.711) and ED-to-ICU admission (AUROC, 0.716; 95% CI, 0.707-0.724), with 36.5% of the cohort meeting qSOFA. The NEWS2 cut-point with the highest AUROC would be NEWS2 ≥7 for both in-hospital mortality (AUROC, 0.653; 95% CI, 0.640-0.666) and ED-to-ICU admission (AUROC, 0.677; 95% CI, 0.668-0.686), with 20.3% of the cohort meeting NEWS2 at this cut-point. The standard Shock Index cut-point ≥0.7 exhibited the highest AUROC for in-hospital mortality and ED-to-ICU admission at our institution.

DISCUSSION

In this retrospective cohort study of 23,837 patients who presented to the ED with suspected sepsis, the standard qSOFA threshold was met least frequently, followed by the Shock Index and NEWS2. NEWS2 had the highest sensitivity but the lowest specificity for predicting in-hospital mortality and ED-to-ICU admission, making it a challenging bedside risk stratification scale for identifying patients at risk of poor clinical outcomes. When comparing predictive performance among the three scales, qSOFA had the highest specificity and the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission in this cohort of patients with suspected sepsis. These trends in sensitivity, specificity, and AUROC were consistent among those who met EHR criteria for a sepsis diagnosis. In the analysis of the three scoring systems using all available cut-points, qSOFA and NEWS2 had the highest AUROCs, followed by the Shock Index.

Considering the rapid progression from organ dysfunction to death in sepsis patients, as well as the difficulty establishing a sepsis diagnosis at triage,²³ providers must quickly identify patients at increased risk of poor outcomes when they present to the ED. Sepsis alerts often are built using SIRS criteria,²⁷ including the one used for sepsis surveillance at UCSF since 2012,²² but the white blood cell count criterion is subject to a laboratory lag and could lead to a delay in identification. Implementation of a point-of-care bedside score alert that uses readily available clinical data could allow providers to identify patients at greatest risk of poor outcomes immediately at ED presentation and triage, which motivated us to explore the predictive performance of qSOFA, the Shock Index, and NEWS2.

Our study is the first to provide a head-to-head comparison of the predictive performance of qSOFA, the Shock Index, and NEWS2, three easy-to-calculate bedside risk scores that use EHR data collected among patients with suspected sepsis. The Sepsis-3 guidelines recommend qSOFA to quickly identify non-ICU patients at greatest risk of poor outcomes because the measure exhibited predictive performance similar to the more extensive SOFA score outside the ICU.^16,23 Although some studies have confirmed qSOFA’s high predictive performance,^28-31 our test characteristics and AUROC findings are in line with other published analyses.^4,6,10,17 The UK National Health Service is using NEWS2 to screen for patients at risk of poor outcomes from sepsis. Several analyses that assessed the predictive ability of NEWS have reported estimates in line with our findings.^4,10,32 The Shock Index was introduced in 1967 and provided a metric to evaluate hemodynamic stability based on heart rate and systolic blood pressure.³³ The Shock Index has been studied in several contexts, including sepsis,³⁴ and studies show that a sustained Shock Index is associated with increased odds of vasopressor administration, higher prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^13,14

For our study, we were particularly interested in exploring how the Shock Index would compare with more frequently used severity scores such as qSOFA and NEWS2 among patients with suspected sepsis, given the simplicity of its calculation and the easy availability of required data. In our cohort of 23,837 patients, only 159 people had missing blood pressure and only 71 had omitted heart rate. In contrast, both qSOFA and NEWS2 include an assessment of level of consciousness that can be subject to variability in assessment methods and EHR documentation across institutions.¹¹ In our cohort, GCS within 30 minutes of ED presentation was missing in 72 patients, which could have led to incomplete calculation of qSOFA and NEWS2 if a missing value was not actually within normal limits.

Several investigations relate qSOFA to NEWS but few compare qSOFA with the newer NEWS2, and even fewer evaluate the Shock Index with any of these scores.^{10,11,18,29,35-37} In general, studies have shown that NEWS exhibits a higher AUROC for predicting mortality, sepsis with organ dysfunction, and ICU admission, often as a composite outcome.^{4,11,18,37,38} A handful of studies compare the Shock Index to SIRS; however, little has been done to compare the Shock Index to qSOFA or NEWS2, scores that have been used specifically for sepsis and might be more predictive of poor outcomes than SIRS.³³ In our study, the Shock Index had a higher AUROC than either qSOFA or NEWS2 for predicting in-hospital mortality and ED-to-ICU admission measured as separate outcomes and as a composite outcome using standard cut-points for these scores.

When selecting a severity score to apply in an institution, it is important to carefully evaluate the score’s test characteristics, in addition to considering the availability of reliable data. Tests with high sensitivity and NPV for the population being studied can be useful to rule out disease or risk of poor outcome, while tests with high specificity and PPV can be useful to rule in disease or risk of poor outcome.³⁹ When considering specificity, qSOFA’s performance was superior to the Shock Index and NEWS2 in our study, but a small percentage of the population was identified using a cut-point of qSOFA ≥2. If we used qSOFA and applied this standard cut-point at our institution, we could be confident that those identified were at increased risk, but we would miss a significant number of patients who would experience a poor outcome. When considering sensitivity, performance of NEWS2 was superior to qSOFA and the Shock Index in our study, but one-half of the population was identified using a cut-point of NEWS2 ≥5. If we were to apply this standard NEWS2 cut-point at our institution, we would assume that one-half of our population was at risk, which might drive resource use towards patients who will not experience a poor outcome. Although none of the scores exhibited a robust AUROC measure, the Shock Index had the highest AUROC for in-hospital mortality and ED-to-ICU admission when using the standard binary cut-point, and its sensitivity and specificity is between that of qSOFA and NEWS2, potentially making it a score to use in settings where qSOFA and NEWS2 score components, such as altered mentation, are not reliably collected. Finally, our sensitivity analysis varying the binary cut-point of each score within our population demonstrated that the standard cut-points might not be as useful within a specific population and might need to be tailored for implementation, balancing sensitivity, specificity, PPV, and NPV to meet local priorities and ICU capacity.

Our study has limitations. It is a single-center, retrospective analysis, factors that could reduce generalizability. However, it does include a large and diverse patient population spanning several years. Missing GCS data could have affected the predictive ability of qSOFA and NEWS2 in our cohort. We could not reliably perform imputation of GCS because of the high missingness and therefore we assumed missing was normal, as was done in the Sepsis-3 derivation studies.¹⁶ Previous studies have attempted to impute GCS and have not observed improved performance of qSOFA to predict mortality.⁴⁰ Because manually collected variables such as GCS are less reliably documented in the EHR, there might be limitations in their use for triage risk scores.

Although the current analysis focused on the predictive performance of qSOFA, the Shock Index, and NEWS2 at triage, performance of these scores could affect the ED team’s treatment decisions before handoff to the hospitalist team and the expected level of care the patient will receive after in-patient admission. These tests also have the advantage of being easy to calculate at the bedside over time, which could provide an objective assessment of longitudinal predicted prognosis. Future work should assess the longitudinal performance of each of these scores among those with suspected sepsis and to determine the impact using these scores would have on clinical and resource utilization outcomes.

CONCLUSION

Local priorities should drive selection of a screening tool, balancing sensitivity, specificity, PPV, and NPV to achieve the institution’s goals. qSOFA, Shock Index, and NEWS2 are risk stratification tools that can be easily implemented at ED triage using data available at the bedside. Although none of these scores performed strongly when comparing AUROCs, qSOFA was highly specific for identifying patients with poor outcomes, and NEWS2 was the most sensitive for ruling out those at high risk among patients with suspected sepsis. The Shock Index exhibited a sensitivity and specificity that fell between qSOFA and NEWS2 and also might be considered to identify those at increased risk, given its ease of implementation, particularly in settings where altered mentation is unreliably or inconsistently documented.

Acknowledgment

The authors thank the UCSF Division of Hospital Medicine Data Core for their assistance with data acquisition.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

METHODS

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

 

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

 

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

 

RESULTS

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

In a subgroup analysis using the fully adjusted model restricted to patients who were mechanically ventilated during hospitalization, the association between LEP and mortality was no longer present (OR 1.15; 95% CI: 0.76-1.72; P = .51).

 

DISCUSSION

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

 

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

METHODS

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

 

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

 

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

 

RESULTS

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

In a subgroup analysis using the fully adjusted model restricted to patients who were mechanically ventilated during hospitalization, the association between LEP and mortality was no longer present (OR 1.15; 95% CI: 0.76-1.72; P = .51).

 

DISCUSSION

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

 

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Sepsis is defined as a life-threatening organ dysfunction that occurs in response to systemic infection.^1,2 It is frequently fatal, common in hospital medicine, and a leading contributor to critical illness, morbidity, and healthcare expenditures.^2-5 While sepsis care and outcomes have improved in the past decade,^6,7 inpatient mortality remains high.⁸

A number of studies have sought to determine whether race plays a role in sepsis mortality. While Black patients with sepsis have frequently been identified as having the highest rates of death,^9-14 similar observations have been made for most non-White races/ethnicities.^13-15 Studies have also demonstrated higher rates of hospital-acquired infections among Asian and Latino patients.¹⁶

There are several possible explanations for why racial minorities experience disparate outcomes in sepsis, including access to care, comorbidities, implicit biases, and biological or environmental factors,^17-20 as well as characteristics of hospitals most likely to care for racial minorities.^13,15,21 One explanation that has not been explored is that racial disparities in sepsis are mediated by language. Limited English proficiency (LEP) has previously been associated with increased rates of adverse hospital events,²² longer length of stay,²³ and greater likelihood of readmission.²⁴ LEP has also been shown to represent a significant barrier to accessing healthcare and preventive screening.²⁵ The role of LEP in sepsis mortality, however, has yet to be examined.

The diverse patient population at the University of California, San Francisco (UCSF) provides a unique opportunity to build upon existing literature by further exploring racial differences in sepsis, specifically by investigating the role of LEP. The objective of this study was to determine the association between LEP and inpatient mortality among adults hospitalized with sepsis.

METHODS

Setting

The study was conducted at the University of California, San Francisco, California (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board with waiver of informed consent. UCSF cares for a population of patients who are racially and linguistically diverse, with high proportions of patients of East Asian descent and with LEP. According to recent United States census estimates, more than half of San Francisco County residents identify as non-White (35% Asians, 15% Hispanic/Latino, 6% Black), and 44% report speaking a language other than English at home.²⁶

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin). We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables. We identified all patients ≥18 years of age presenting to the emergency department (ED) between June 1, 2012 and December 31, 2016 with suspected serious infection, defined as having blood cultures ordered within 72 hours of ED presentation (N = 25,441). Patients who did not receive at least two doses of intravenous (IV) antibiotics within 48 hours were excluded, as they were unlikely to have serious infections.

 

We defined sepsis based on Sepsis-3 consensus guidelines² as a change in sequential [sepsis-related] organ failure assessment (SOFA) score ≥2 within the first 48 hours of ED presentation. The SOFA score is comprised of six variables representing different organ systems, each rated 0-4 based on the degree of dysfunction.² Patient vital signs, laboratory data, vasopressor medication doses, and ventilator settings were used to determine the exact timestamp at which each patient attained a change in SOFA score ≥2. Missing values were considered to be normal. To adjust for baseline organ dysfunction, SOFA elements associated with elevated bilirubin and/or creatinine were excluded for patients with chronic liver/kidney disease based on Elixhauser comorbidities.²⁷ We chose to focus on the first 48 hours in an attempt to capture patients with the most severe illnesses and the highest probability of true sepsis.

All primary and secondary International Classification of Diseases (ICD)-9/10 diagnosis codes were extracted from Clarity coding tables at the time of hospital discharge. Diagnosis codes signifying bacterial infection were grouped into the following categories based on type/location: pneumonia; bacteremia; urinary tract infection; and skin and soft tissue infection. All remaining diagnostic codes indicating bacterial infections at other sites were categorized as “Other”. If no codes indicating infection were present, patients were categorized as “None coded”. Patients with discharge diagnosis codes of “sepsis” were also identified. Dates and times of antibiotic administrations were obtained from the medications table. Time to first antibiotic was defined as the time in minutes from ED presentation to initiation of the first IV antibacterial medication. This variable was transformed using a natural log transformation based on best fit for normal distribution.

We limited our analyses to 8,974 patients who were diagnosed with sepsis as defined above and had either (1) ≥4 qualifying antibiotic days (QADs) or (2) an ICD-9/10 discharge diagnosis code of “sepsis” (Figure). QADs were defined based on the recent publication by Rhee et al. as having received four or more consecutive days of antibiotics, with the first dose given IV within 48 hours of presentation.²⁸ Patients who died or were discharged to hospice prior to the 4th QAD were also included. These additional parameters were added to increase specificity of the study sample for patients with true sepsis. Patients admitted to all levels of care (acute care, transitional care unit [TCU], intensive care unit [ICU]) and under all hospital services were included. There were no missing data for mortality, race, or language. We chose to focus on patients with sepsis in this initial study as this is a common diagnosis in hospital medicine that is enriched for high mortality.

Primary Outcome

The primary outcome of the study was inpatient mortality, which was obtained from the hospital encounters table in Clarity.

Primary Predictors

The primary predictor of interest was LEP. The encounter numbers from the dataset were used to link to self-reported demographic data, including “preferred language” and need for interpreter services. A manual chart review of 60 patients speaking the top six languages was conducted to verify the accuracy of the data on language and interpreter use (KNK). Defining the gold standard for LEP as having any chart note indicating non-English language and/or that an interpreter was used, the “interpreter needed” variable in Epic was found to have a positive predictive value for LEP of 100%. Therefore, patients in the study cohort were defined as having LEP if they met both of the following criteria: (1) a self-reported “preferred language” other than English and (2) having the “interpreter needed” variable indicating “yes”.

 

Covariate Data Collection

Additional data were obtained from the demographics tables, including age, race, sex, and insurance status. Race and ethnicity were combined into a single five-category variable including White, Asian, Black, Latino, and Other. This approach has been suggested as the best way to operationalize these variables²⁹ and has been utilized by similar studies in the literature.^9,14,15 We considered the Asian race to include all people of East Asian, Southeast Asian, or South Asian descent, which is consistent with the United States Census Bureau definition.³⁰ Patients identifying as Native Hawaiians/Pacific Islanders, Native Americans/Alaskan Natives, as well as those with unspecified race or ethnicity, were categorized as Other. Insurance status was categorized as Commercial, Medicare, Medicaid, or Other.

We estimated illness severity in several ways. First, the total qualifying SOFA score was calculated for each patient, which was defined as the total score achieved at the time that SOFA criteria were first met (≥2, within 48 hours). Second, we dichotomized patients based on whether they had received mechanical ventilation at any point during hospitalization. Finally, we used admission location as a surrogate marker for severity at the time of initial hospitalization.

To estimate the burden of baseline comorbidities, we calculated the van Walraven score (VWS),³¹ a validated modification of the Elixhauser Comorbidity Index.²⁷ This score conveys an estimated risk of in hospital death based on ICD-9/10 diagnosis codes for preexisting conditions, which ranges from <1% for the minimum score of –19 to >99% for the maximum score of 89.

Statistical Analyses

All statistical analyses were performed using Stata software version 15 (StataCorp LLC, College Station, Texas). Baseline demographics and patient characteristics were stratified by LEP. These were compared using two-sample t-tests or chi-squared tests of significance. Wilcoxon rank-sum tests were used for non-normally distributed variables. Inpatient mortality was compared across all races stratified by LEP using chi-squared tests of significance.

We fit a series of multivariable logistic regression models to examine the association between race and inpatient mortality adjusting for LEP and other patient/clinical characteristics. We first examined the unadjusted association between mortality and race; then adjusted for LEP alone; and finally adjusted for all covariates of interest, including LEP, age, sex, insurance status, year, admission level of care, VWS, total qualifying SOFA score, need for mechanical ventilation, site of infection, and time to first IV antibiotic. A subgroup analysis was also performed using the fully adjusted model restricted to patients who were mechanically ventilated. This population was selected because the patients (1) have among the highest severity of illness and (2) share a common barrier to communication, regardless of English proficiency.

Several potential interactions between LEP with other covariates were explored, including age, race, ICU admission level of care, and need for mechanical ventilation. Lastly, a mediation analysis was performed based on Baron & Kenny’s four-step model³² in order to calculate the proportion of the association between race and mortality explained by the proposed mediator (LEP).

To evaluate for the likelihood of residual confounding, we calculated an E-value, which is defined as the minimum strength of association that an unmeasured confounder would need to have with both the predictor and outcome variables, above and beyond the measured covariates, in order to fully explain away an observed predictor-outcome association.^33,34

 

RESULTS

We identified 8,974 patients hospitalized with sepsis based on the above inclusion criteria. This represented a medically complex, racially and linguistically diverse population (Table 1). The cohort was comprised of 24% Asian, 12% Black, and 11% Latino patients. Among those categorized as Other race, Native Americans/Alaskan Natives and Native Hawaiians/Pacific Islanders accounted for 4% (n = 31) and 21% (n = 159), respectively. A fifth of all patients had LEP (n = 1,716), 62% of whom were Asian (n = 1,064). Patients with LEP tended to be older, female, and to have a greater number of comorbid conditions (Table 1). The total qualifying SOFA score was also higher among patients with LEP (median 5; interquartile range [IQR]: 4-8 vs 5; IQR: 3-7; P <.001), though there was no association between LEP and mechanical ventilation (P = .22). The prevalence of LEP differed significantly across races, with 50% LEP among Asians, 32% among Latinos, 5% among White patients (P < .001). Only eight Black patients had LEP. More than 40 unique languages were represented in the cohort, with English, Cantonese, Spanish, Russian, and Mandarin accounting for ~95% (Appendix Table 1). Among Latino patients, 63% spoke English and 36% spoke Spanish.

In-hospital mortality was significantly higher among patients who had LEP (n = 268/1,716, 16%) compared to non-LEP patients (n = 678/7,258, 9%), with 80% greater unadjusted odds of mortality (OR 1.80; 95% CI: 1.54-2.09; P < .001). Notably we also found that Asian race was associated with a 1.57 unadjusted odds of mortality compared to White race (95% CI: 1.34-1.85; P < .001). Age, VWS, total qualifying SOFA score, mechanical ventilation, and admission level of care all exhibited a positive dose-response association with mortality (Appendix Table 2). In unadjusted analyses, there was no evidence of interaction between LEP and age (P = .38), LEP and race (P = .45), LEP and ICU admission level of care (P = .31), or LEP and mechanical ventilation (P = .19). Asian patients had the highest overall mortality (14% total, 17% with LEP). LEP was associated with increased unadjusted mortality among White, Asian, and Other races compared to their non-LEP counterparts (Appendix Figure 1). There was no significant difference in mortality between Latino patients with and without LEP. The sample size for Black patients with LEP (n = 8) was too small to draw conclusions about mortality.

Following multivariable logistic regression modeling for the association between race and mortality, we found that the increased odds of death among Asian patients was partially attenuated after adjusting for LEP (odds ratio [OR] 1.23, 95% CI: 1.02-1.48; P = .03; Table 2). Meanwhile, LEP was associated with a 1.66 odds of mortality (95% CI: 1.38-1.99; P < .001) after adjustment for race. In the full multivariable model adjusting for demographics and clinical characteristics, illness severity, and comorbidities, LEP was associated with a 31% increase in the odds of mortality compared to non-LEP (95% CI: 1.06-1.63; P = .02). In this model, the association between Asian race and mortality was now fully attenuated, with a point estimate near 1.0 (OR 0.98; 95% CI: 0.79-1.22; P = .87). Markers of illness severity, including total qualifying SOFA score (OR 1.23; 95% CI: 1.20-1.27; P < .001) and need for mechanical ventilation (OR 1.88; 95% CI: 1.52-2.33; P < .001), were both associated with greater odds of death. Based on a four-step mediation analysis, LEP was found to be a partial mediator to the association between Asian race and mortality (76% proportion explained). The E-value for the association between LEP and mortality was 1.95, with an E-value for the corresponding confidence interval of 1.29.

In a subgroup analysis using the fully adjusted model restricted to patients who were mechanically ventilated during hospitalization, the association between LEP and mortality was no longer present (OR 1.15; 95% CI: 0.76-1.72; P = .51).

 

DISCUSSION

At a single US academic medical center serving a diverse population, we found that LEP was associated with sepsis mortality across all races except Black and Latino, conveying a 31% increase in the odds of death after adjusting for illness severity, comorbidities, and baseline characteristics. The higher mortality among Asian patients was largely mediated by LEP (76% proportion explained). While previous studies have variably found Black, Asian, Latino, and other non-White races/ethnicities to be at an increased risk of death from sepsis,^9-15 LEP has not been previously evaluated as a mediator of sepsis mortality. We were uniquely suited to uncover such an association due to the racial and linguistic diversity of our patient population. LEP has previously been implicated in poor health outcomes among hospitalized patients in general.^22-24 Future studies will be necessary to determine whether similar associations between LEP and mortality are observed among broader patient populations outside of sepsis.

There are a number of possible explanations for how LEP could mediate the association between race and mortality. First, LEP is known to be associated with greater difficulties in accessing medical care,²⁵ which could result in poorer baseline control of chronic comorbid conditions, fewer opportunities for preventive screening, and greater reluctance to seek medical attention when ill, theoretically leading to more severe presentations and worse outcomes. Indeed, LEP patients in our cohort had both a shorter median time to receiving their first antibiotic, as well as a higher total qualifying SOFA score, both of which may suggest more severe initial presentations. LEP is also known to contribute to, or exacerbate, the impact of low health literacy, which is itself associated with poor health.³⁵ Second, implicit biases may also have been present, as they are known to be common among healthcare providers and have been shown to negatively impact patient care.³⁶

Finally,it is possible that the association is related to the language barrier itself, which impacts providers’ ability to take an appropriate clinical history, and can lead to clinical errors or delays in care.³⁷ The fact that the association between LEP and mortality was eliminated when the analysis was restricted to mechanically ventilated patients seems to support this, since differences in language proficiency become irrelevant in this subgroup. While we are unable to comment on causality based on this observational study, we included a directed acyclic graph (DAG) in the supplemental materials, which shows one proposed model for describing these associations (Appendix Figure 2).

Assuming that the language barrier itself does, at least in part, drive the observed association, LEP represents a potentially modifiable risk factor that could be a target for quality improvement interventions. There is evidence that the use of medical interpreters among patients with LEP leads to greater satisfaction, fewer errors, and improved clinical outcomes;³⁸ however, several recent studies have documented underutilization of professional interpreter services, even when readily available.^39,40 At our institution, phone and video interpreter services are available 24/7 for approximately 150 languages. Due to limitations inherent to the EHR, we were unable to ascertain the extent to which these services were used in the present study. Heavy clinical workloads, connectivity issues, and missing or faulty equipment represent theoretical barriers to utilization of these services.

There are some limitations to our study. First, by utilizing a large database of electronic data, the quality of our analyses was reliant on the accuracy of the EHR. Demographic data such as language may have been subject to misclassification due to self-reporting. We attempted to minimize this by also including the need for interpreter services within the definition of LEP, which was validated by manual chart review. Second, generalizability is limited in this single-center study conducted at an institution with unique demographics, wherein nearly two-thirds of the LEP patients were Asian, and the Chinese-speaking population outnumbered those who speak Spanish.

Finally, the most important limitation to our study is the potential for residual confounding. While we attempted to mitigate this by adjusting for as many clinically relevant covariates as possible, there may still be unmeasured confounders to the association between LEP and mortality, such as access to outpatient care, functional status, interpreter use, and other markers of illness severity like the number and type of supportive therapies received. Based on our E-value calculations, with an observed OR of 1.31 for the association between LEP and mortality, an unmeasured confounder with an OR of 1.95 would fully explain away this association, while an OR of 1.29 would shift the confidence interval to include the null. These values suggest at least some risk of residual confounding. The fact that our fully adjusted model included multiple covariates, including several markers of illness severity, does somewhat lessen the likelihood of a confounder achieving these values, since they represent the minimum strength of an unmeasured confounder above and beyond the measured covariates. Regardless, the finding that patients with LEP are more likely to die from sepsis remains an important one, recognizing the need for further studies including multicenter investigations.

In this study, we showed that LEP was associated with sepsis mortality across nearly all races in our cohort. While Asian race was associated with a higher unadjusted odds of death compared to White race, this was attenuated after adjusting for LEP. This may suggest that some of the racial disparities in sepsis identified in prior studies were in fact mediated by language proficiency. Further studies will be required to explore the causal nature of this novel association. If modifiable factors are identified, this could represent a potential target for future quality improvement initiatives aimed at improving sepsis outcomes.

 

Disclaimer

The contents are solely the responsibility of the authors and do not necessarily represent the official views of the University of California, San Francisco or the National Institutes of Health.

Elevated blood pressure (BP) is common among hospitalized adults, with prevalence estimates between 50% and 70%.¹ Many factors can cause or exacerbate BP elevations in the setting of acute illness, such as pain, anxiety, medication withdrawal, and volume status, among others.² While there are clear evidence-based recommendations for treating hypertension (HTN) in the ambulatory setting,³ guidelines for the management of elevated BP in the hospital are lacking.^4,5

Hypertensive crises are generally recognized as warranting rapid reduction in BP;^6-8 however, these represent the minority of cases.^9,10 Far more common in the hospital are patients with asymptomatic elevated BP, a population for which there is no high-quality evidence and no guidelines supporting the use of intravenous (IV) antihypertensives.^11,12 Treatment with such medications has been associated with highly variable clinical responses^13-15 and may result in adverse events, such as hypotension.¹⁰

To date, only a small number of studies have investigated the treatment of asymptomatic elevated BP among hospitalized adults.^10,13-15 These have suggested that IV antihypertensives are utilized frequently in this setting, often for only modestly elevated BPs; however, the studies have tended to be small, not racially diverse, and limited to noncritically ill patients. Furthermore, while it is generally accepted that reducing the use of IV antihypertensives among asymptomatic patients would have no adverse impact, to our knowledge there have been no published studies which have instituted such an initiative while measuring balancing outcomes.

The purpose of this study was to further the existing literature by defining the prevalence and effects of IV antihypertensive medication utilization among a medically complex, multiracial population of asymptomatic medical inpatients using a large electronic dataset and to evaluate the impact of a division-wide, two-tiered quality improvement (QI) initiative on the rates of IV antihypertensive utilization and patient outcomes.

METHODS

Setting

The study was conducted at the University of California, San Francisco (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board. General medicine patients at UCSF are distributed between teaching and direct-care (hospitalist) services. The intensive care unit (ICU) is “open,” meaning the medicine service acts as the primary team for all nonsurgical ICU patients. This study included all adult general medicine patients admitted to UCSF Medical Center between January 1, 2017 and March 1, 2018, including those in the ICU.

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin) for all clinical care. We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables, including orders, medications, laboratory and radiology results, vital signs, patient demographics, and notes. We identified all adult patients hospitalized on the general medicine service with ≥1 episode of elevated BP (>160/90 mm Hg) at any point during their hospitalization who were not on a vasopressor medication at the time of the vital sign recording.

 

We further identified all instances in which either IV labetalol or hydralazine were administered to these patients. These two agents were chosen because they are the only IV antihypertensives used commonly at our institution for the treatment of asymptomatic elevated BP among internal medicine patients. Only those orders placed by a general medicine provider or reconciled by a general medicine provider upon transfer from another service were included. For each medication administration timestamp, we collected vital signs before and after the administration, along with the ordering provider and the clinical indication that was documented in the electronic order. To determine if a medication was administered with concern for end-organ injury, we also extracted orders that could serve as a proxy for the provider’s clinical assessment—namely electrocardiograms, serum troponins, chest x-rays, and computerized tomography scans of the head—which were placed in the one hour preceding or 15 minutes following administration of an IV antihypertensive medication.

To assess for comorbid conditions, including a preexisting diagnosis of HTN, we collected International Classification of Diseases (ICD)-9/10 diagnosis codes. Further, we also extracted All Patient Refined Diagnosis-Related Group (APR-DRG) weights, which are a standardized measure of illness severity based on relative resource consumption during hospitalization.^16,17

Patients were categorized as having either “symptomatic” or “asymptomatic” elevated BP. We defined symptomatic elevated BP as having received treatment with an IV medication with provider concern for end-organ injury, as defined above. We further identified all patients in which tight BP control may be clinically indicated on the basis of the presence of any of the following ICD-9/10 diagnosis codes at the time of hospital discharge: myocardial infarction, ischemic stroke, intracranial hemorrhage, subarachnoid hemorrhage, subdural hematoma, aortic dissection, hypertensive emergency, or hypertensive encephalopathy. All patients with symptomatic elevated BP or any of the above ICD-9/10 diagnoses were excluded from the analysis, since administration of IV antihypertensive medications would plausibly be warranted in these clinical scenarios.

The encounter numbers from the dataset were used to link to patient demographic data, which included age, sex, race, ethnicity, primary language, and insurance status. Finally, we identified all instances of rapid response calls, ICU transfers, and code blues (cardiopulmonary arrests) for each patient in the dataset.

Blood Pressure Measurements

BP data were collected from invasive BP (IBP) monitoring devices and noninvasive BP cuffs. For patients with BP measurements recorded concomitantly from both IBP (ie, arterial lines) in addition to noninvasive BP cuffs, the arterial line reading was favored. All systolic BP (SBP) readings >240 mm Hg from arterial lines were excluded, as this has previously been described as the upper physiologic limit for IBP readings.¹⁸

Primary Outcome

The primary outcome for the study was the proportion of patients treated with IV antihypertensive medications (labetalol or hydralazine). Using aggregate data, we calculated the number of patients who were treated at least once with an IV antihypertensive in a given month (numerator), divided by the number of patients with ≥1 episode of asymptomatic elevated BP that month (denominator). The denominator was considered to be the population of patients “at risk” of being treated with IV antihypertensive medications. For patients with multiple admissions during the study period, each admission was considered separately. These results are displayed in the upper portion of the run chart (Figure).

 

Secondary Outcomes

To investigate blood pressure trends over time, we analyzed BP in three ways. First, we analyzed the median SBP for the entire population. Second, to determine clinical responses to IV antihypertensive medications among patients receiving treatment, we calculated the population medians for the pretreatment SBP, the change in SBP from pretreatment baseline, and the posttreatment SBP. Third, we calculated the average median SBP on a monthly basis for the duration of the study. This was achieved by calculating the median value of all SBPs for an individual patient, then averaging across all patients in a given month. The average monthly median SBPs are displayed in the lower portion of the Figure.

To investigate whether the intervention was associated with negative patient outcomes, the proportions of several balancing outcomes were compared between pre- and postintervention periods, including ICU transfers, rapid response calls, and code blues (cardiopulmonary arrests).

Development and Implementation of an Intervention to Reduce Excessive IV Antihypertensive Use

After establishing the baseline prevalence of IV antihypertensive medication use at our institution, we developed a QI initiative with the goal of reducing IV antihypertensive medication utilization by the general medicine service for the treatment of asymptomatic patients. We hypothesized that potential contributors to overutilization might include lack of education, provider/nursing discomfort, and a system designed to mandate provider notification for even modestly elevated BPs. The QI initiative, which took place between October 2017 and December 2017, was designed to address these potential contributors and was comprised of a division-wide, two-tiered, bundled intervention. Our choice of a two-tiered approach was based on the fact that successful culture change is challenging, along with the existing evidence that multifaceted QI interventions are more often successful than single-tiered approaches.¹⁹

The first tier of the initiative included an educational campaign referred to colloquially as “NoIVForHighBP,” which targeted residents, hospitalists, and nursing staff. The campaign consisted of a series of presentations, best practice updates, handouts, and posters displayed prominently in shared workspaces. The educational content focused on alternative approaches to the management of asymptomatic elevated BP in the hospital, such as identification and treatment of pain, anxiety, volume overload, or other contributing factors (see supplemental materials). These educational outreaches occurred periodically between October 4, 2017 and November 20, 2017, with the bulk of the educational efforts taking place during November. Therefore, November 1, 2017 was designated the start date for the intervention period.

The second tier of the intervention included the liberalization of the EHR BP notification parameters on the standard inpatient admission order set from >160/90 mm Hg to >180/90 mm Hg. This change took effect on 12/6/2017. The decision to modify the BP notification parameters was based on the hypothesis that mandatory notifications for modestly elevated BPs may prompt providers to reflexively order IV antihypertensive medications, especially during times of cross-coverage or high clinical workload.

Statistical Analysis

All statistical analyses were performed using Stata software version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, Texas: StataCorp LLC). Baseline patient characteristics were compared using nonparametric tests of significance. Population median SBPs were compared between pre- and postintervention periods using Mood’s Median Test, which was selected because the data were distributed nonnormally, and variances between samples were unequal.

 

Among patients treated with IV antihypertensive medications, we compared the proportion of pretreatment SBPs falling into each of three specified ranges (SBP <180 mm Hg, SBP 180-199 mm Hg, and SBP >200 mm Hg) between baseline and intervention periods using chi-squared tests.

Using aggregate data, we compared the unadjusted proportion of patients treated with IV antihypertensive medications between pre- and postintervention periods using a chi-squared test. Next, using patient-level data, a logistic regression analysis was performed to examine the association between receipt of IV antihypertensive medications and time (dichotomized between pre- and postintervention periods) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Rates of balancing outcomes were compared using chi-squared tests. A logistic regression analysis using patient-level data was also performed to investigate the association between each of these outcomes and the intervention period (pre vs post) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

RESULTS

Baseline Period

We identified 2,306 patients with ≥1 episode of asymptomatic elevated BP during the 10-month preintervention period. Patients on average experienced 9 episodes of elevated BP per hospitalization, representing 21,207 potential opportunities for treatment. Baseline characteristics are summarized in Table 1. In general, this represents an older population that was medically complex and multiracial.

Of these patients, 251 (11%) received IV hydralazine and/or labetalol at least once during their hospitalization, with a total of 597 doses administered. Among those treated, a median of 2 doses were given per patient (IQR: 1-4), 64% of which were hydralazine. The majority (380 [64%]) were ordered on an “as needed” basis, while 217 (36%) were administered as a one-time dose. Three-quarters of all doses were ordered by the teaching service (456 [76%]), with the remaining 24% ordered by the direct-care (hospitalist) service.

During the baseline period among patients receiving IV antihypertensive medications, the median SBP of the population prior to treatment was 187 mm Hg (IQR 177-199; Table 2). Treatment was initiated in 30% of patients for an SBP <180 mm Hg and in 75% for an SBP <200 mm Hg. The median time to follow-up BP check was 34 minutes (IQR 15-58). The median decrement in SBP was 20 mm Hg (IQR 5-37); however, the response to treatment was highly variable, with 2% of patients experiencing no change and 14% experiencing an increase in SBP. Seventy-nine patients (14%) had a decrement in SBP >25% following treatment.

Description of Quality Improvement Results

Following the QI initiative, a total of 934 patients experienced 9,743 episodes of asymptomatic elevated blood pressure over a 4-month period (November 1, 2017 to February 28, 2018). As shown in Table 1, patients in the postintervention period had a slightly higher median age (67 [IQR 55-80] vs 69 [IQR 57-83]; P = .01), a higher median APR-DRG weight (1.34 [IQR 0.99-1.77] vs 1.48 [1.00-1.82]; P < .001), and a longer median length of stay (4.6 [2.8-8.0] days vs 5.1 [2.9-9.2] days; P = .004). There was also a higher proportion of nonEnglish speakers, fewer Black patients, and a lower proportion of preexisting HTN, in the postintervention period.

 

Of the 934 patients with ≥1 episode of asymptomatic elevated BP, 70 (7%) were treated with IV antihypertensive medications, with a total of 196 doses administered. The proportion of patients treated per month during the postintervention period ranged from 6% to 8%, which was the lowest of the entire study period and below the baseline average of 10% (Figure).

In a patient-level logistic regression pre-post analysis adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight, patients admitted to the general medicine service during the postintervention period had 38% lower odds of receiving IV antihypertensive medications than those admitted during the baseline period (OR = 0.62; 95% CI 0.47-0.83; P = .001). In this adjusted model, the following factors were independently associated with increased odds of receiving treatment: APR-DRG weight (OR 1.13; 95% CI 1.07-1.20; P < .001), Black race (OR 1.81; 95% CI 1.29-2.53; P = .001), length of stay (OR 1.02; 95% CI 1.01-1.03; P < .001), and preexisting HTN (OR 4.25; 95% CI 2.75-6.56; P < .001). Older age was associated with lower odds of treatment (Table 2).

Among patients who received treatment, there were no differences between pre- and postintervention periods in the proportion of pretreatment SBP <180 mm Hg (29% vs 32%; P = .40), 180-199 mm Hg (47% vs 40%; P = .10), or >200 mm Hg (25% vs 28%; P = .31; Table 3).

Population-level median SBP was similar between pre- and postintervention periods (167 mm Hg vs 168 mm Hg, P = .78), as were unadjusted rates of rapid response calls, ICU transfers, and code blues (Table 3). After adjustment for baseline characteristics and illness severity at the patient level, the odds of rapid response calls (OR 0.84; 95% CI 0.65-1.10; P = .21) and ICU transfers (OR 1.01; 95% CI 0.75-1.38; P = .93) did not differ between pre- and postintervention periods. A regression model was not fit for cardiopulmonary arrests due to the low absolute number of events.

CONCLUSIONS

Our results suggest that treatment of asymptomatic elevated BP using IV antihypertensive medications is common practice at our institution. We found that treatment is often initiated for only modestly elevated BPs and that the clinical response to these medications is highly variable. In the baseline period, one in seven patients experienced a decrement in BP >25% following treatment, which could potentially cause harm.¹¹ There is no evidence, neither are there any consensus guidelines, to support the rapid reduction of BP among asymptomatic patients, making this a potential valuable opportunity for reducing unnecessary treatment, minimizing waste, and avoiding harm.

While there are a few previously published studies with similar results, we add to the existing literature by studying a larger population of more than 3,000 total patients, which was uniquely multiracial, including a high proportion of non-English speakers. Furthermore, our cohort included patients in the ICU, which is reflected in the higher-than-average APR-DRG weights. Despite being critically ill, these patients arguably still do not warrant aggressive treatment of elevated BP when asymptomatic. By excluding symptomatic BP elevations using surrogate markers for end-organ damage in addition to discharge diagnosis codes indicative of conditions in which tight BP control may be warranted, we were able to study a more critically ill patient population. We were also able to describe which baseline patient characteristics convey higher adjusted odds of receiving treatment, such as preexisting HTN, younger age, illness severity, and black race.

Perhaps most significantly, our study is the first to demonstrate an effective QI intervention aimed at reducing unnecessary utilization of IV antihypertensives. We found that this can feasibly be accomplished through a combination of educational efforts and systems changes, which could easily be replicated at other institutions. While the absolute reduction in the number of patients receiving treatment was modest, if these findings were to be widely accepted and resulted in a wide-spread change in culture, there would be a potential for greater impact.

Despite the reduction in the proportion of patients receiving IV antihypertensive medications, we found no change in the median SBP compared with the baseline period, which seems to support that the intervention was well tolerated. We also found no difference in the number of ICU transfers, rapid response calls, and cardiopulmonary arrests between groups. While these findings are both reassuring, it is impossible to draw definitive conclusions about safety given the small absolute number of patients having received treatment in each group. Fortunately, current guidelines and literature support the safety of such an intervention, as there is no existing evidence to suggest that failing to rapidly lower BP among asymptomatic patients is potentially harmful.¹¹

There are several limitations to our study. First, by utilizing a large electronic dataset, the quality of our analyses was reliant on the accuracy of the recorded EHR data. Second, in the absence of a controlled trial or control group, we cannot say definitively that our QI initiative was the direct cause of the improved rates of IV antihypertensive utilization, though the effect did persist after adjusting for baseline characteristics in patient-level models. Third, our follow-up period was relatively short, with fewer than half as many patients as in the preintervention period. This is an important limitation, since the impact of QI interventions often diminishes over time. We plan to continually monitor IV antihypertensive use, feed those data back to our group, and revitalize educational efforts should rates begin to rise. Fourth, we were unable to directly measure which patients had true end-organ injury and instead used orders placed around the time of medication administration as a surrogate marker. While this is an imperfect measure, we feel that in cases where a provider was concerned enough to even test for end-organ injury, the use of IV antihypertensives was likely justified and was therefore appropriately excluded from the analysis. Lastly, we were limited in our ability to describe associations with true clinical outcomes, such as stroke or myocardial infarction, which could theoretically be propagated by either the use or the avoidance of IV antihypertensive medications. Fortunately, based on clinical guidelines and existing evidence, there is no reason to believe that reducing IV antihypertensive use would result in increased rates of these outcomes.

Our study reaffirms the fact that overutilization of IV antihypertensive medications among asymptomatic hospitalized patients is pervasive across hospital systems. This represents a potential target for a concerted change in culture, which we have demonstrated can be feasibly accomplished through education and systems changes.

 

Disclosures

Dr. Auerbach has current or pending grants from the CDC, PCORI, and FDA that are unrelated to this research manuscript. He also receives royalties from UpToDate, and received an honorarium for being editor of JHM. Dr. Jacobs received a $1,000 Resident/Fellow Travel Grant from the Society of Hospital Medicine to support the cost of travel to SHM, where he presented this research as a poster in 2018. Dr. Prasad receives money from EpiExcellence, LLC for consultation, which is unrelated to this research manuscript. All other authors have nothing to disclose.

 

Elevated blood pressure (BP) is common among hospitalized adults, with prevalence estimates between 50% and 70%.¹ Many factors can cause or exacerbate BP elevations in the setting of acute illness, such as pain, anxiety, medication withdrawal, and volume status, among others.² While there are clear evidence-based recommendations for treating hypertension (HTN) in the ambulatory setting,³ guidelines for the management of elevated BP in the hospital are lacking.^4,5

Hypertensive crises are generally recognized as warranting rapid reduction in BP;^6-8 however, these represent the minority of cases.^9,10 Far more common in the hospital are patients with asymptomatic elevated BP, a population for which there is no high-quality evidence and no guidelines supporting the use of intravenous (IV) antihypertensives.^11,12 Treatment with such medications has been associated with highly variable clinical responses^13-15 and may result in adverse events, such as hypotension.¹⁰

To date, only a small number of studies have investigated the treatment of asymptomatic elevated BP among hospitalized adults.^10,13-15 These have suggested that IV antihypertensives are utilized frequently in this setting, often for only modestly elevated BPs; however, the studies have tended to be small, not racially diverse, and limited to noncritically ill patients. Furthermore, while it is generally accepted that reducing the use of IV antihypertensives among asymptomatic patients would have no adverse impact, to our knowledge there have been no published studies which have instituted such an initiative while measuring balancing outcomes.

The purpose of this study was to further the existing literature by defining the prevalence and effects of IV antihypertensive medication utilization among a medically complex, multiracial population of asymptomatic medical inpatients using a large electronic dataset and to evaluate the impact of a division-wide, two-tiered quality improvement (QI) initiative on the rates of IV antihypertensive utilization and patient outcomes.

METHODS

Setting

The study was conducted at the University of California, San Francisco (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board. General medicine patients at UCSF are distributed between teaching and direct-care (hospitalist) services. The intensive care unit (ICU) is “open,” meaning the medicine service acts as the primary team for all nonsurgical ICU patients. This study included all adult general medicine patients admitted to UCSF Medical Center between January 1, 2017 and March 1, 2018, including those in the ICU.

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin) for all clinical care. We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables, including orders, medications, laboratory and radiology results, vital signs, patient demographics, and notes. We identified all adult patients hospitalized on the general medicine service with ≥1 episode of elevated BP (>160/90 mm Hg) at any point during their hospitalization who were not on a vasopressor medication at the time of the vital sign recording.

 

We further identified all instances in which either IV labetalol or hydralazine were administered to these patients. These two agents were chosen because they are the only IV antihypertensives used commonly at our institution for the treatment of asymptomatic elevated BP among internal medicine patients. Only those orders placed by a general medicine provider or reconciled by a general medicine provider upon transfer from another service were included. For each medication administration timestamp, we collected vital signs before and after the administration, along with the ordering provider and the clinical indication that was documented in the electronic order. To determine if a medication was administered with concern for end-organ injury, we also extracted orders that could serve as a proxy for the provider’s clinical assessment—namely electrocardiograms, serum troponins, chest x-rays, and computerized tomography scans of the head—which were placed in the one hour preceding or 15 minutes following administration of an IV antihypertensive medication.

To assess for comorbid conditions, including a preexisting diagnosis of HTN, we collected International Classification of Diseases (ICD)-9/10 diagnosis codes. Further, we also extracted All Patient Refined Diagnosis-Related Group (APR-DRG) weights, which are a standardized measure of illness severity based on relative resource consumption during hospitalization.^16,17

Patients were categorized as having either “symptomatic” or “asymptomatic” elevated BP. We defined symptomatic elevated BP as having received treatment with an IV medication with provider concern for end-organ injury, as defined above. We further identified all patients in which tight BP control may be clinically indicated on the basis of the presence of any of the following ICD-9/10 diagnosis codes at the time of hospital discharge: myocardial infarction, ischemic stroke, intracranial hemorrhage, subarachnoid hemorrhage, subdural hematoma, aortic dissection, hypertensive emergency, or hypertensive encephalopathy. All patients with symptomatic elevated BP or any of the above ICD-9/10 diagnoses were excluded from the analysis, since administration of IV antihypertensive medications would plausibly be warranted in these clinical scenarios.

The encounter numbers from the dataset were used to link to patient demographic data, which included age, sex, race, ethnicity, primary language, and insurance status. Finally, we identified all instances of rapid response calls, ICU transfers, and code blues (cardiopulmonary arrests) for each patient in the dataset.

Blood Pressure Measurements

BP data were collected from invasive BP (IBP) monitoring devices and noninvasive BP cuffs. For patients with BP measurements recorded concomitantly from both IBP (ie, arterial lines) in addition to noninvasive BP cuffs, the arterial line reading was favored. All systolic BP (SBP) readings >240 mm Hg from arterial lines were excluded, as this has previously been described as the upper physiologic limit for IBP readings.¹⁸

Primary Outcome

The primary outcome for the study was the proportion of patients treated with IV antihypertensive medications (labetalol or hydralazine). Using aggregate data, we calculated the number of patients who were treated at least once with an IV antihypertensive in a given month (numerator), divided by the number of patients with ≥1 episode of asymptomatic elevated BP that month (denominator). The denominator was considered to be the population of patients “at risk” of being treated with IV antihypertensive medications. For patients with multiple admissions during the study period, each admission was considered separately. These results are displayed in the upper portion of the run chart (Figure).

 

Secondary Outcomes

To investigate blood pressure trends over time, we analyzed BP in three ways. First, we analyzed the median SBP for the entire population. Second, to determine clinical responses to IV antihypertensive medications among patients receiving treatment, we calculated the population medians for the pretreatment SBP, the change in SBP from pretreatment baseline, and the posttreatment SBP. Third, we calculated the average median SBP on a monthly basis for the duration of the study. This was achieved by calculating the median value of all SBPs for an individual patient, then averaging across all patients in a given month. The average monthly median SBPs are displayed in the lower portion of the Figure.

To investigate whether the intervention was associated with negative patient outcomes, the proportions of several balancing outcomes were compared between pre- and postintervention periods, including ICU transfers, rapid response calls, and code blues (cardiopulmonary arrests).

Development and Implementation of an Intervention to Reduce Excessive IV Antihypertensive Use

After establishing the baseline prevalence of IV antihypertensive medication use at our institution, we developed a QI initiative with the goal of reducing IV antihypertensive medication utilization by the general medicine service for the treatment of asymptomatic patients. We hypothesized that potential contributors to overutilization might include lack of education, provider/nursing discomfort, and a system designed to mandate provider notification for even modestly elevated BPs. The QI initiative, which took place between October 2017 and December 2017, was designed to address these potential contributors and was comprised of a division-wide, two-tiered, bundled intervention. Our choice of a two-tiered approach was based on the fact that successful culture change is challenging, along with the existing evidence that multifaceted QI interventions are more often successful than single-tiered approaches.¹⁹

The first tier of the initiative included an educational campaign referred to colloquially as “NoIVForHighBP,” which targeted residents, hospitalists, and nursing staff. The campaign consisted of a series of presentations, best practice updates, handouts, and posters displayed prominently in shared workspaces. The educational content focused on alternative approaches to the management of asymptomatic elevated BP in the hospital, such as identification and treatment of pain, anxiety, volume overload, or other contributing factors (see supplemental materials). These educational outreaches occurred periodically between October 4, 2017 and November 20, 2017, with the bulk of the educational efforts taking place during November. Therefore, November 1, 2017 was designated the start date for the intervention period.

The second tier of the intervention included the liberalization of the EHR BP notification parameters on the standard inpatient admission order set from >160/90 mm Hg to >180/90 mm Hg. This change took effect on 12/6/2017. The decision to modify the BP notification parameters was based on the hypothesis that mandatory notifications for modestly elevated BPs may prompt providers to reflexively order IV antihypertensive medications, especially during times of cross-coverage or high clinical workload.

Statistical Analysis

All statistical analyses were performed using Stata software version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, Texas: StataCorp LLC). Baseline patient characteristics were compared using nonparametric tests of significance. Population median SBPs were compared between pre- and postintervention periods using Mood’s Median Test, which was selected because the data were distributed nonnormally, and variances between samples were unequal.

 

Among patients treated with IV antihypertensive medications, we compared the proportion of pretreatment SBPs falling into each of three specified ranges (SBP <180 mm Hg, SBP 180-199 mm Hg, and SBP >200 mm Hg) between baseline and intervention periods using chi-squared tests.

Using aggregate data, we compared the unadjusted proportion of patients treated with IV antihypertensive medications between pre- and postintervention periods using a chi-squared test. Next, using patient-level data, a logistic regression analysis was performed to examine the association between receipt of IV antihypertensive medications and time (dichotomized between pre- and postintervention periods) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Rates of balancing outcomes were compared using chi-squared tests. A logistic regression analysis using patient-level data was also performed to investigate the association between each of these outcomes and the intervention period (pre vs post) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

RESULTS

Baseline Period

We identified 2,306 patients with ≥1 episode of asymptomatic elevated BP during the 10-month preintervention period. Patients on average experienced 9 episodes of elevated BP per hospitalization, representing 21,207 potential opportunities for treatment. Baseline characteristics are summarized in Table 1. In general, this represents an older population that was medically complex and multiracial.

Of these patients, 251 (11%) received IV hydralazine and/or labetalol at least once during their hospitalization, with a total of 597 doses administered. Among those treated, a median of 2 doses were given per patient (IQR: 1-4), 64% of which were hydralazine. The majority (380 [64%]) were ordered on an “as needed” basis, while 217 (36%) were administered as a one-time dose. Three-quarters of all doses were ordered by the teaching service (456 [76%]), with the remaining 24% ordered by the direct-care (hospitalist) service.

During the baseline period among patients receiving IV antihypertensive medications, the median SBP of the population prior to treatment was 187 mm Hg (IQR 177-199; Table 2). Treatment was initiated in 30% of patients for an SBP <180 mm Hg and in 75% for an SBP <200 mm Hg. The median time to follow-up BP check was 34 minutes (IQR 15-58). The median decrement in SBP was 20 mm Hg (IQR 5-37); however, the response to treatment was highly variable, with 2% of patients experiencing no change and 14% experiencing an increase in SBP. Seventy-nine patients (14%) had a decrement in SBP >25% following treatment.

Description of Quality Improvement Results

Following the QI initiative, a total of 934 patients experienced 9,743 episodes of asymptomatic elevated blood pressure over a 4-month period (November 1, 2017 to February 28, 2018). As shown in Table 1, patients in the postintervention period had a slightly higher median age (67 [IQR 55-80] vs 69 [IQR 57-83]; P = .01), a higher median APR-DRG weight (1.34 [IQR 0.99-1.77] vs 1.48 [1.00-1.82]; P < .001), and a longer median length of stay (4.6 [2.8-8.0] days vs 5.1 [2.9-9.2] days; P = .004). There was also a higher proportion of nonEnglish speakers, fewer Black patients, and a lower proportion of preexisting HTN, in the postintervention period.

 

Of the 934 patients with ≥1 episode of asymptomatic elevated BP, 70 (7%) were treated with IV antihypertensive medications, with a total of 196 doses administered. The proportion of patients treated per month during the postintervention period ranged from 6% to 8%, which was the lowest of the entire study period and below the baseline average of 10% (Figure).

In a patient-level logistic regression pre-post analysis adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight, patients admitted to the general medicine service during the postintervention period had 38% lower odds of receiving IV antihypertensive medications than those admitted during the baseline period (OR = 0.62; 95% CI 0.47-0.83; P = .001). In this adjusted model, the following factors were independently associated with increased odds of receiving treatment: APR-DRG weight (OR 1.13; 95% CI 1.07-1.20; P < .001), Black race (OR 1.81; 95% CI 1.29-2.53; P = .001), length of stay (OR 1.02; 95% CI 1.01-1.03; P < .001), and preexisting HTN (OR 4.25; 95% CI 2.75-6.56; P < .001). Older age was associated with lower odds of treatment (Table 2).

Among patients who received treatment, there were no differences between pre- and postintervention periods in the proportion of pretreatment SBP <180 mm Hg (29% vs 32%; P = .40), 180-199 mm Hg (47% vs 40%; P = .10), or >200 mm Hg (25% vs 28%; P = .31; Table 3).

Population-level median SBP was similar between pre- and postintervention periods (167 mm Hg vs 168 mm Hg, P = .78), as were unadjusted rates of rapid response calls, ICU transfers, and code blues (Table 3). After adjustment for baseline characteristics and illness severity at the patient level, the odds of rapid response calls (OR 0.84; 95% CI 0.65-1.10; P = .21) and ICU transfers (OR 1.01; 95% CI 0.75-1.38; P = .93) did not differ between pre- and postintervention periods. A regression model was not fit for cardiopulmonary arrests due to the low absolute number of events.

CONCLUSIONS

Our results suggest that treatment of asymptomatic elevated BP using IV antihypertensive medications is common practice at our institution. We found that treatment is often initiated for only modestly elevated BPs and that the clinical response to these medications is highly variable. In the baseline period, one in seven patients experienced a decrement in BP >25% following treatment, which could potentially cause harm.¹¹ There is no evidence, neither are there any consensus guidelines, to support the rapid reduction of BP among asymptomatic patients, making this a potential valuable opportunity for reducing unnecessary treatment, minimizing waste, and avoiding harm.

While there are a few previously published studies with similar results, we add to the existing literature by studying a larger population of more than 3,000 total patients, which was uniquely multiracial, including a high proportion of non-English speakers. Furthermore, our cohort included patients in the ICU, which is reflected in the higher-than-average APR-DRG weights. Despite being critically ill, these patients arguably still do not warrant aggressive treatment of elevated BP when asymptomatic. By excluding symptomatic BP elevations using surrogate markers for end-organ damage in addition to discharge diagnosis codes indicative of conditions in which tight BP control may be warranted, we were able to study a more critically ill patient population. We were also able to describe which baseline patient characteristics convey higher adjusted odds of receiving treatment, such as preexisting HTN, younger age, illness severity, and black race.

Perhaps most significantly, our study is the first to demonstrate an effective QI intervention aimed at reducing unnecessary utilization of IV antihypertensives. We found that this can feasibly be accomplished through a combination of educational efforts and systems changes, which could easily be replicated at other institutions. While the absolute reduction in the number of patients receiving treatment was modest, if these findings were to be widely accepted and resulted in a wide-spread change in culture, there would be a potential for greater impact.

Despite the reduction in the proportion of patients receiving IV antihypertensive medications, we found no change in the median SBP compared with the baseline period, which seems to support that the intervention was well tolerated. We also found no difference in the number of ICU transfers, rapid response calls, and cardiopulmonary arrests between groups. While these findings are both reassuring, it is impossible to draw definitive conclusions about safety given the small absolute number of patients having received treatment in each group. Fortunately, current guidelines and literature support the safety of such an intervention, as there is no existing evidence to suggest that failing to rapidly lower BP among asymptomatic patients is potentially harmful.¹¹

There are several limitations to our study. First, by utilizing a large electronic dataset, the quality of our analyses was reliant on the accuracy of the recorded EHR data. Second, in the absence of a controlled trial or control group, we cannot say definitively that our QI initiative was the direct cause of the improved rates of IV antihypertensive utilization, though the effect did persist after adjusting for baseline characteristics in patient-level models. Third, our follow-up period was relatively short, with fewer than half as many patients as in the preintervention period. This is an important limitation, since the impact of QI interventions often diminishes over time. We plan to continually monitor IV antihypertensive use, feed those data back to our group, and revitalize educational efforts should rates begin to rise. Fourth, we were unable to directly measure which patients had true end-organ injury and instead used orders placed around the time of medication administration as a surrogate marker. While this is an imperfect measure, we feel that in cases where a provider was concerned enough to even test for end-organ injury, the use of IV antihypertensives was likely justified and was therefore appropriately excluded from the analysis. Lastly, we were limited in our ability to describe associations with true clinical outcomes, such as stroke or myocardial infarction, which could theoretically be propagated by either the use or the avoidance of IV antihypertensive medications. Fortunately, based on clinical guidelines and existing evidence, there is no reason to believe that reducing IV antihypertensive use would result in increased rates of these outcomes.

Our study reaffirms the fact that overutilization of IV antihypertensive medications among asymptomatic hospitalized patients is pervasive across hospital systems. This represents a potential target for a concerted change in culture, which we have demonstrated can be feasibly accomplished through education and systems changes.

 

Disclosures

Dr. Auerbach has current or pending grants from the CDC, PCORI, and FDA that are unrelated to this research manuscript. He also receives royalties from UpToDate, and received an honorarium for being editor of JHM. Dr. Jacobs received a $1,000 Resident/Fellow Travel Grant from the Society of Hospital Medicine to support the cost of travel to SHM, where he presented this research as a poster in 2018. Dr. Prasad receives money from EpiExcellence, LLC for consultation, which is unrelated to this research manuscript. All other authors have nothing to disclose.

 

Elevated blood pressure (BP) is common among hospitalized adults, with prevalence estimates between 50% and 70%.¹ Many factors can cause or exacerbate BP elevations in the setting of acute illness, such as pain, anxiety, medication withdrawal, and volume status, among others.² While there are clear evidence-based recommendations for treating hypertension (HTN) in the ambulatory setting,³ guidelines for the management of elevated BP in the hospital are lacking.^4,5

Hypertensive crises are generally recognized as warranting rapid reduction in BP;^6-8 however, these represent the minority of cases.^9,10 Far more common in the hospital are patients with asymptomatic elevated BP, a population for which there is no high-quality evidence and no guidelines supporting the use of intravenous (IV) antihypertensives.^11,12 Treatment with such medications has been associated with highly variable clinical responses^13-15 and may result in adverse events, such as hypotension.¹⁰

To date, only a small number of studies have investigated the treatment of asymptomatic elevated BP among hospitalized adults.^10,13-15 These have suggested that IV antihypertensives are utilized frequently in this setting, often for only modestly elevated BPs; however, the studies have tended to be small, not racially diverse, and limited to noncritically ill patients. Furthermore, while it is generally accepted that reducing the use of IV antihypertensives among asymptomatic patients would have no adverse impact, to our knowledge there have been no published studies which have instituted such an initiative while measuring balancing outcomes.

The purpose of this study was to further the existing literature by defining the prevalence and effects of IV antihypertensive medication utilization among a medically complex, multiracial population of asymptomatic medical inpatients using a large electronic dataset and to evaluate the impact of a division-wide, two-tiered quality improvement (QI) initiative on the rates of IV antihypertensive utilization and patient outcomes.

METHODS

Setting

The study was conducted at the University of California, San Francisco (UCSF), an 800-bed tertiary care, academic medical center. It was approved by the UCSF Institutional Review Board. General medicine patients at UCSF are distributed between teaching and direct-care (hospitalist) services. The intensive care unit (ICU) is “open,” meaning the medicine service acts as the primary team for all nonsurgical ICU patients. This study included all adult general medicine patients admitted to UCSF Medical Center between January 1, 2017 and March 1, 2018, including those in the ICU.

Study Population and Data Collection

The UCSF Medical Center uses the electronic health record (EHR) Epic (Epic 2017, Epic Systems Corporation, Verona, Wisconsin) for all clinical care. We obtained computerized EHR data from Clarity, the relational database that stores Epic’s inpatient data in thousands of tables, including orders, medications, laboratory and radiology results, vital signs, patient demographics, and notes. We identified all adult patients hospitalized on the general medicine service with ≥1 episode of elevated BP (>160/90 mm Hg) at any point during their hospitalization who were not on a vasopressor medication at the time of the vital sign recording.

 

We further identified all instances in which either IV labetalol or hydralazine were administered to these patients. These two agents were chosen because they are the only IV antihypertensives used commonly at our institution for the treatment of asymptomatic elevated BP among internal medicine patients. Only those orders placed by a general medicine provider or reconciled by a general medicine provider upon transfer from another service were included. For each medication administration timestamp, we collected vital signs before and after the administration, along with the ordering provider and the clinical indication that was documented in the electronic order. To determine if a medication was administered with concern for end-organ injury, we also extracted orders that could serve as a proxy for the provider’s clinical assessment—namely electrocardiograms, serum troponins, chest x-rays, and computerized tomography scans of the head—which were placed in the one hour preceding or 15 minutes following administration of an IV antihypertensive medication.

To assess for comorbid conditions, including a preexisting diagnosis of HTN, we collected International Classification of Diseases (ICD)-9/10 diagnosis codes. Further, we also extracted All Patient Refined Diagnosis-Related Group (APR-DRG) weights, which are a standardized measure of illness severity based on relative resource consumption during hospitalization.^16,17

Patients were categorized as having either “symptomatic” or “asymptomatic” elevated BP. We defined symptomatic elevated BP as having received treatment with an IV medication with provider concern for end-organ injury, as defined above. We further identified all patients in which tight BP control may be clinically indicated on the basis of the presence of any of the following ICD-9/10 diagnosis codes at the time of hospital discharge: myocardial infarction, ischemic stroke, intracranial hemorrhage, subarachnoid hemorrhage, subdural hematoma, aortic dissection, hypertensive emergency, or hypertensive encephalopathy. All patients with symptomatic elevated BP or any of the above ICD-9/10 diagnoses were excluded from the analysis, since administration of IV antihypertensive medications would plausibly be warranted in these clinical scenarios.

The encounter numbers from the dataset were used to link to patient demographic data, which included age, sex, race, ethnicity, primary language, and insurance status. Finally, we identified all instances of rapid response calls, ICU transfers, and code blues (cardiopulmonary arrests) for each patient in the dataset.

Blood Pressure Measurements

BP data were collected from invasive BP (IBP) monitoring devices and noninvasive BP cuffs. For patients with BP measurements recorded concomitantly from both IBP (ie, arterial lines) in addition to noninvasive BP cuffs, the arterial line reading was favored. All systolic BP (SBP) readings >240 mm Hg from arterial lines were excluded, as this has previously been described as the upper physiologic limit for IBP readings.¹⁸

Primary Outcome

The primary outcome for the study was the proportion of patients treated with IV antihypertensive medications (labetalol or hydralazine). Using aggregate data, we calculated the number of patients who were treated at least once with an IV antihypertensive in a given month (numerator), divided by the number of patients with ≥1 episode of asymptomatic elevated BP that month (denominator). The denominator was considered to be the population of patients “at risk” of being treated with IV antihypertensive medications. For patients with multiple admissions during the study period, each admission was considered separately. These results are displayed in the upper portion of the run chart (Figure).

 

Secondary Outcomes

To investigate blood pressure trends over time, we analyzed BP in three ways. First, we analyzed the median SBP for the entire population. Second, to determine clinical responses to IV antihypertensive medications among patients receiving treatment, we calculated the population medians for the pretreatment SBP, the change in SBP from pretreatment baseline, and the posttreatment SBP. Third, we calculated the average median SBP on a monthly basis for the duration of the study. This was achieved by calculating the median value of all SBPs for an individual patient, then averaging across all patients in a given month. The average monthly median SBPs are displayed in the lower portion of the Figure.

To investigate whether the intervention was associated with negative patient outcomes, the proportions of several balancing outcomes were compared between pre- and postintervention periods, including ICU transfers, rapid response calls, and code blues (cardiopulmonary arrests).

Development and Implementation of an Intervention to Reduce Excessive IV Antihypertensive Use

After establishing the baseline prevalence of IV antihypertensive medication use at our institution, we developed a QI initiative with the goal of reducing IV antihypertensive medication utilization by the general medicine service for the treatment of asymptomatic patients. We hypothesized that potential contributors to overutilization might include lack of education, provider/nursing discomfort, and a system designed to mandate provider notification for even modestly elevated BPs. The QI initiative, which took place between October 2017 and December 2017, was designed to address these potential contributors and was comprised of a division-wide, two-tiered, bundled intervention. Our choice of a two-tiered approach was based on the fact that successful culture change is challenging, along with the existing evidence that multifaceted QI interventions are more often successful than single-tiered approaches.¹⁹

The first tier of the initiative included an educational campaign referred to colloquially as “NoIVForHighBP,” which targeted residents, hospitalists, and nursing staff. The campaign consisted of a series of presentations, best practice updates, handouts, and posters displayed prominently in shared workspaces. The educational content focused on alternative approaches to the management of asymptomatic elevated BP in the hospital, such as identification and treatment of pain, anxiety, volume overload, or other contributing factors (see supplemental materials). These educational outreaches occurred periodically between October 4, 2017 and November 20, 2017, with the bulk of the educational efforts taking place during November. Therefore, November 1, 2017 was designated the start date for the intervention period.

The second tier of the intervention included the liberalization of the EHR BP notification parameters on the standard inpatient admission order set from >160/90 mm Hg to >180/90 mm Hg. This change took effect on 12/6/2017. The decision to modify the BP notification parameters was based on the hypothesis that mandatory notifications for modestly elevated BPs may prompt providers to reflexively order IV antihypertensive medications, especially during times of cross-coverage or high clinical workload.

Statistical Analysis

All statistical analyses were performed using Stata software version 15 (StataCorp. 2017. Stata Statistical Software: Release 15. College Station, Texas: StataCorp LLC). Baseline patient characteristics were compared using nonparametric tests of significance. Population median SBPs were compared between pre- and postintervention periods using Mood’s Median Test, which was selected because the data were distributed nonnormally, and variances between samples were unequal.

 

Among patients treated with IV antihypertensive medications, we compared the proportion of pretreatment SBPs falling into each of three specified ranges (SBP <180 mm Hg, SBP 180-199 mm Hg, and SBP >200 mm Hg) between baseline and intervention periods using chi-squared tests.

Using aggregate data, we compared the unadjusted proportion of patients treated with IV antihypertensive medications between pre- and postintervention periods using a chi-squared test. Next, using patient-level data, a logistic regression analysis was performed to examine the association between receipt of IV antihypertensive medications and time (dichotomized between pre- and postintervention periods) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

Rates of balancing outcomes were compared using chi-squared tests. A logistic regression analysis using patient-level data was also performed to investigate the association between each of these outcomes and the intervention period (pre vs post) while adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight.

RESULTS

Baseline Period

We identified 2,306 patients with ≥1 episode of asymptomatic elevated BP during the 10-month preintervention period. Patients on average experienced 9 episodes of elevated BP per hospitalization, representing 21,207 potential opportunities for treatment. Baseline characteristics are summarized in Table 1. In general, this represents an older population that was medically complex and multiracial.

Of these patients, 251 (11%) received IV hydralazine and/or labetalol at least once during their hospitalization, with a total of 597 doses administered. Among those treated, a median of 2 doses were given per patient (IQR: 1-4), 64% of which were hydralazine. The majority (380 [64%]) were ordered on an “as needed” basis, while 217 (36%) were administered as a one-time dose. Three-quarters of all doses were ordered by the teaching service (456 [76%]), with the remaining 24% ordered by the direct-care (hospitalist) service.

During the baseline period among patients receiving IV antihypertensive medications, the median SBP of the population prior to treatment was 187 mm Hg (IQR 177-199; Table 2). Treatment was initiated in 30% of patients for an SBP <180 mm Hg and in 75% for an SBP <200 mm Hg. The median time to follow-up BP check was 34 minutes (IQR 15-58). The median decrement in SBP was 20 mm Hg (IQR 5-37); however, the response to treatment was highly variable, with 2% of patients experiencing no change and 14% experiencing an increase in SBP. Seventy-nine patients (14%) had a decrement in SBP >25% following treatment.

Description of Quality Improvement Results

Following the QI initiative, a total of 934 patients experienced 9,743 episodes of asymptomatic elevated blood pressure over a 4-month period (November 1, 2017 to February 28, 2018). As shown in Table 1, patients in the postintervention period had a slightly higher median age (67 [IQR 55-80] vs 69 [IQR 57-83]; P = .01), a higher median APR-DRG weight (1.34 [IQR 0.99-1.77] vs 1.48 [1.00-1.82]; P < .001), and a longer median length of stay (4.6 [2.8-8.0] days vs 5.1 [2.9-9.2] days; P = .004). There was also a higher proportion of nonEnglish speakers, fewer Black patients, and a lower proportion of preexisting HTN, in the postintervention period.

 

Of the 934 patients with ≥1 episode of asymptomatic elevated BP, 70 (7%) were treated with IV antihypertensive medications, with a total of 196 doses administered. The proportion of patients treated per month during the postintervention period ranged from 6% to 8%, which was the lowest of the entire study period and below the baseline average of 10% (Figure).

In a patient-level logistic regression pre-post analysis adjusting for age, sex, race, ethnicity, primary language, insurance status, preexisting HTN, length of stay, and APR-DRG weight, patients admitted to the general medicine service during the postintervention period had 38% lower odds of receiving IV antihypertensive medications than those admitted during the baseline period (OR = 0.62; 95% CI 0.47-0.83; P = .001). In this adjusted model, the following factors were independently associated with increased odds of receiving treatment: APR-DRG weight (OR 1.13; 95% CI 1.07-1.20; P < .001), Black race (OR 1.81; 95% CI 1.29-2.53; P = .001), length of stay (OR 1.02; 95% CI 1.01-1.03; P < .001), and preexisting HTN (OR 4.25; 95% CI 2.75-6.56; P < .001). Older age was associated with lower odds of treatment (Table 2).

Among patients who received treatment, there were no differences between pre- and postintervention periods in the proportion of pretreatment SBP <180 mm Hg (29% vs 32%; P = .40), 180-199 mm Hg (47% vs 40%; P = .10), or >200 mm Hg (25% vs 28%; P = .31; Table 3).

Population-level median SBP was similar between pre- and postintervention periods (167 mm Hg vs 168 mm Hg, P = .78), as were unadjusted rates of rapid response calls, ICU transfers, and code blues (Table 3). After adjustment for baseline characteristics and illness severity at the patient level, the odds of rapid response calls (OR 0.84; 95% CI 0.65-1.10; P = .21) and ICU transfers (OR 1.01; 95% CI 0.75-1.38; P = .93) did not differ between pre- and postintervention periods. A regression model was not fit for cardiopulmonary arrests due to the low absolute number of events.

CONCLUSIONS

Our results suggest that treatment of asymptomatic elevated BP using IV antihypertensive medications is common practice at our institution. We found that treatment is often initiated for only modestly elevated BPs and that the clinical response to these medications is highly variable. In the baseline period, one in seven patients experienced a decrement in BP >25% following treatment, which could potentially cause harm.¹¹ There is no evidence, neither are there any consensus guidelines, to support the rapid reduction of BP among asymptomatic patients, making this a potential valuable opportunity for reducing unnecessary treatment, minimizing waste, and avoiding harm.

While there are a few previously published studies with similar results, we add to the existing literature by studying a larger population of more than 3,000 total patients, which was uniquely multiracial, including a high proportion of non-English speakers. Furthermore, our cohort included patients in the ICU, which is reflected in the higher-than-average APR-DRG weights. Despite being critically ill, these patients arguably still do not warrant aggressive treatment of elevated BP when asymptomatic. By excluding symptomatic BP elevations using surrogate markers for end-organ damage in addition to discharge diagnosis codes indicative of conditions in which tight BP control may be warranted, we were able to study a more critically ill patient population. We were also able to describe which baseline patient characteristics convey higher adjusted odds of receiving treatment, such as preexisting HTN, younger age, illness severity, and black race.

Perhaps most significantly, our study is the first to demonstrate an effective QI intervention aimed at reducing unnecessary utilization of IV antihypertensives. We found that this can feasibly be accomplished through a combination of educational efforts and systems changes, which could easily be replicated at other institutions. While the absolute reduction in the number of patients receiving treatment was modest, if these findings were to be widely accepted and resulted in a wide-spread change in culture, there would be a potential for greater impact.

Despite the reduction in the proportion of patients receiving IV antihypertensive medications, we found no change in the median SBP compared with the baseline period, which seems to support that the intervention was well tolerated. We also found no difference in the number of ICU transfers, rapid response calls, and cardiopulmonary arrests between groups. While these findings are both reassuring, it is impossible to draw definitive conclusions about safety given the small absolute number of patients having received treatment in each group. Fortunately, current guidelines and literature support the safety of such an intervention, as there is no existing evidence to suggest that failing to rapidly lower BP among asymptomatic patients is potentially harmful.¹¹

There are several limitations to our study. First, by utilizing a large electronic dataset, the quality of our analyses was reliant on the accuracy of the recorded EHR data. Second, in the absence of a controlled trial or control group, we cannot say definitively that our QI initiative was the direct cause of the improved rates of IV antihypertensive utilization, though the effect did persist after adjusting for baseline characteristics in patient-level models. Third, our follow-up period was relatively short, with fewer than half as many patients as in the preintervention period. This is an important limitation, since the impact of QI interventions often diminishes over time. We plan to continually monitor IV antihypertensive use, feed those data back to our group, and revitalize educational efforts should rates begin to rise. Fourth, we were unable to directly measure which patients had true end-organ injury and instead used orders placed around the time of medication administration as a surrogate marker. While this is an imperfect measure, we feel that in cases where a provider was concerned enough to even test for end-organ injury, the use of IV antihypertensives was likely justified and was therefore appropriately excluded from the analysis. Lastly, we were limited in our ability to describe associations with true clinical outcomes, such as stroke or myocardial infarction, which could theoretically be propagated by either the use or the avoidance of IV antihypertensive medications. Fortunately, based on clinical guidelines and existing evidence, there is no reason to believe that reducing IV antihypertensive use would result in increased rates of these outcomes.

Our study reaffirms the fact that overutilization of IV antihypertensive medications among asymptomatic hospitalized patients is pervasive across hospital systems. This represents a potential target for a concerted change in culture, which we have demonstrated can be feasibly accomplished through education and systems changes.

 

Disclosures

Dr. Auerbach has current or pending grants from the CDC, PCORI, and FDA that are unrelated to this research manuscript. He also receives royalties from UpToDate, and received an honorarium for being editor of JHM. Dr. Jacobs received a $1,000 Resident/Fellow Travel Grant from the Society of Hospital Medicine to support the cost of travel to SHM, where he presented this research as a poster in 2018. Dr. Prasad receives money from EpiExcellence, LLC for consultation, which is unrelated to this research manuscript. All other authors have nothing to disclose.