Matthew M. Churpek, MD, MPH, PhD

METHODS

RESULTS

DISCUSSION

Acknowledgments

The authors thank Pamela McCall, BSN, OCN for her assistance with study implementation, Kevin Ig-Izevbekhai and Shivraj Grewal for assistance with data collection, UCM Clinical Engineering for technical support, and Timothy Holper, MS, Julie Johnson, MPH, RN, and Thomas Sutton for assistance with data abstraction.

Disclosure

Dr. Churpek is supported by a career development award from the National Heart, Lung, and Blood Institute (K08 HL121080) and has received honoraria from Chest for invited speaking engagements. Dr. Churpek and Dr. Edelson have a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients. In addition, Dr. Edelson has received research support from Philips Healthcare (Andover, MA), research support from the American Heart Association (Dallas, TX) and Laerdal Medical (Stavanger, Norway), and research support from EarlySense (Tel Aviv, Israel). She has ownership interest in Quant HC (Chicago, IL), which is developing products for risk stratification of hospitalized patients. This study was supported by a grant from Philips Healthcare in Andover, MA. The sponsor had no role in data collection, interpretation of results, or drafting of the manuscript.

METHODS

RESULTS

DISCUSSION

Acknowledgments

The authors thank Pamela McCall, BSN, OCN for her assistance with study implementation, Kevin Ig-Izevbekhai and Shivraj Grewal for assistance with data collection, UCM Clinical Engineering for technical support, and Timothy Holper, MS, Julie Johnson, MPH, RN, and Thomas Sutton for assistance with data abstraction.

Disclosure

Dr. Churpek is supported by a career development award from the National Heart, Lung, and Blood Institute (K08 HL121080) and has received honoraria from Chest for invited speaking engagements. Dr. Churpek and Dr. Edelson have a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients. In addition, Dr. Edelson has received research support from Philips Healthcare (Andover, MA), research support from the American Heart Association (Dallas, TX) and Laerdal Medical (Stavanger, Norway), and research support from EarlySense (Tel Aviv, Israel). She has ownership interest in Quant HC (Chicago, IL), which is developing products for risk stratification of hospitalized patients. This study was supported by a grant from Philips Healthcare in Andover, MA. The sponsor had no role in data collection, interpretation of results, or drafting of the manuscript.

METHODS

RESULTS

DISCUSSION

Acknowledgments

The authors thank Pamela McCall, BSN, OCN for her assistance with study implementation, Kevin Ig-Izevbekhai and Shivraj Grewal for assistance with data collection, UCM Clinical Engineering for technical support, and Timothy Holper, MS, Julie Johnson, MPH, RN, and Thomas Sutton for assistance with data abstraction.

Disclosure

Dr. Churpek is supported by a career development award from the National Heart, Lung, and Blood Institute (K08 HL121080) and has received honoraria from Chest for invited speaking engagements. Dr. Churpek and Dr. Edelson have a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients. In addition, Dr. Edelson has received research support from Philips Healthcare (Andover, MA), research support from the American Heart Association (Dallas, TX) and Laerdal Medical (Stavanger, Norway), and research support from EarlySense (Tel Aviv, Israel). She has ownership interest in Quant HC (Chicago, IL), which is developing products for risk stratification of hospitalized patients. This study was supported by a grant from Philips Healthcare in Andover, MA. The sponsor had no role in data collection, interpretation of results, or drafting of the manuscript.

MATERIALS AND METHODS

RESULTS

DISCUSSION

CONCLUSION

MATERIALS AND METHODS

RESULTS

DISCUSSION

CONCLUSION

MATERIALS AND METHODS

RESULTS

DISCUSSION

CONCLUSION

METHODS

RESULTS

DISCUSSION

METHODS

RESULTS

DISCUSSION

METHODS

RESULTS

DISCUSSION

Patients on hospital wards may become critically ill due to worsening of the underlying condition that was the cause of their admission or acquisition of a new hospital‐acquired illness. Once physiologic deterioration occurs, some patients are evaluated and quickly transferred to the intensive care unit (ICU), whereas others are left on the wards until further deterioration occurs. Because many critical illness syndromes benefit from early intervention, such as sepsis and respiratory failure, early transfer to the ICU for treatment may improve patient outcomes, and conversely, delays in ICU transfer may lead to increased mortality and length of stay (LOS) in critically ill ward patients.[1, 2] However, the timeliness of that transfer is dependent on numerous changing variables, such as ICU bed availability, clinician identification of the deterioration, and clinical judgment regarding the appropriate transfer thresholds.[2, 3, 4, 5, 6, 7] As a result, there is a large degree of heterogeneity in the severity of illness of patients at the time of ICU transfer and in patient outcomes.[6, 8]

Previous studies investigating the association between delayed ICU transfer and patient outcomes have typically utilized the time of consultation by the ICU team to denote the onset of critical illness.[5, 6, 9, 10] However, the decision to transfer a patient to the ICU is often subjective, and previous studies have found an alarmingly high rate of errors in diagnosis and management of critically ill ward patients, including the failure to call for help.[2, 11] Therefore, a more objective tool for quantifying critical illness is necessary for determining the onset of critical illness and quantifying the association of transfer delay with patient outcomes.

Early warning scores, which are designed to detect critical illness on the wards, represent objective measures of critical illness that can be easily calculated in ward patients.[12] The aim of this study was to utilize the electronic Cardiac Arrest Risk Triage (eCART) score, a previously published, statistically derived early warning score that utilizes demographic, vital sign, and laboratory data, as an objective measure of critical illness to estimate the effect of delayed ICU transfer on patient outcomes in a large, multicenter database.[13] We chose 6 hours as the cutoff for delay in this study a priori because it is a threshold noted to be an important time period in critical illness syndromes, such as sepsis.[14, 15]

METHODS

All patients admitted to the medical‐surgical wards at 5 hospitals between November 2008 and January 2013 were eligible for inclusion in this observational cohort study. Further details of the hospital populations have been previously described.[13] A waiver of consent was granted by NorthShore University HealthSystem (IRB #EH11‐258) and the University of Chicago Institutional Review Board (IRB #16995A) based on general impracticability and minimal harm. Collection of patient information was designed to comply with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) regulations.

RESULTS

A total of 269,999 admissions had documented vital signs on the hospital wards during the study period, including 11,995 patients who were either transferred from the wards to the ICU (n=11,636) or who suffered a cardiac arrest on the wards (n=359) during their initial ward stay. Of these patients, 3789 reached an eCART score at the 95% specificity cutoff (critical eCART score of 60) within 24 hours of transfer. The median time from first critical eCART value to ICU transfer was 5.4 hours (interquartile range (IQR), 214 hours; mean, 8 hours). Compared to patients without delayed ICU transfer, those with delayed transfer were slightly older (median age, 73 [IQR, 6083] years vs 71 [IQR, 5882] years; P=0.002), whereas all other characteristics were similar (Table 1). Table 2 shows comparisons of vital sign and laboratory results for delayed and nondelayed transfers at the time of ICU transfer. As shown, patients with delayed transfer had lower median respiratory rate, blood pressure, heart rate, and hemoglobin, but higher median white blood cell count and creatinine.

Delayed transfer occurred in 46% of patients (n=1734) and was associated with increased in‐hospital mortality (33.2% vs 24.5%, P<0.001). This relationship was linear, with each 1‐hour increase in transfer delay associated with a 3% increase in the odds of in‐hospital death (P<0.001) (Figure 1). The association between length of transfer delay and hospital mortality remained unchanged after controlling for age, sex, surgical status, initial eCART score on the wards, vital signs, laboratory values, and whether the ICU transfer was due to a cardiac arrest (3% increase per hour, P<0.001). This association did not vary based on the initial eCART score on the wards (P=0.71 for interaction). Additionally, despite having similar median hospital lengths of stay prior to first critical eCART score (1.6 vs 1.5 days, P=0.04), patients experiencing delayed ICU transfer who survived to discharge had a longer median hospital LOS by 2 days compared to those with nondelayed transfer who survived to discharge (median LOS, 13 (821) days vs 11 (719) days, P=0.01). The change in eCART score over time in the 12 hours before first reaching the critical eCART score until ICU transfer is shown in Figure 2 for patients with delayed and nondelayed transfer. As shown, patients transferred within 6 hours had a more rapid rise in eCART score prior to ICU transfer compared to those with a delayed transfer. This difference in trajectories between delayed and nondelayed patients was similar in patients with low (<13), intermediate (1359), and high (60) initial eCART scores on the wards. A regression model investigating the association between eCART slope prior to ICU transfer and time to ICU transfer demonstrated that a steeper slope was significantly associated with a decreased time to ICU transfer (P<0.01).

Figure 1

Figure 2

DISCUSSION

We found that a delay in transfer to the ICU after reaching a predefined objective threshold of critical illness was associated with a significant increase in hospital mortality and hospital LOS. We also discovered a significant association between critical illness trajectory and delays in transfer, suggesting that caregivers may not recognize more subtle trends in critical illness. This work highlights the importance of timely transfer to the ICU for critically ill ward patients, which can be affected by several factors such as ICU bed availability and caregiver recognition and triage decisions. Our findings have significant implications for patient safety on the wards and provide further evidence for implementing early warning scores into practice to aid with clinical decision making.

Our findings of increased mortality with delayed ICU transfer are consistent with previous studies.[1, 5, 9] For example, Young et al. compared ICU mortality between delayed and nondelayed transfers in 91 consecutive patients with noncardiac diagnoses at a community hospital.[1] They also used predefined criteria for critical illness, and found that delayed transfers had a higher ICU mortality than nondelayed patients (41% vs 11%). However, their criteria for critical illness only had a specificity of 13% for predicting ICU transfer, compared to 95% in our study, suggesting that our threshold is more consistent with critical illness. Another study, by Cardoso and colleagues, investigated the impact of delayed ICU admission due to bed shortages on ICU mortality in 401 patients at a university hospital.[9] Of those patients deemed appropriate for transfer to the ICU but who had to wait for a bed to become available, the median wait time for a bed was 18 hours. They found that each hour of waiting was associated with a 1.5% increase in ICU death. A similar study by Robert and colleagues investigated the impact of delayed or refused ICU admission due to a lack of bed availability.[5] Patients deemed too sick (or too well) to benefit from ICU transfer were excluded. Twenty‐eightday and 60‐day mortality were higher in the admitted group compared to those not admitted, although this finding was not statistically significant. In addition, patients later admitted to the ICU once a bed became available (median wait time, 6 hours; n=89) had higher 28‐day mortality than those admitted immediately (adjusted odds ratio, 1.78; P=0.05). Several other studies have investigated the impact of ICU refusal for reasons that included bed shortages, and found increased mortality in those not admitted to the ICU.[16, 17] However, many of these studies included patients deemed too sick or too well to be transferred to the ICU in the group of nonadmitted patients. Our study adds to this literature by utilizing a highly specific objective measure of critical illness and by including all patients on the wards who reached this threshold, rather than only those for whom a consult was requested.

There are several potential explanations for our finding of increased mortality with delayed ICU transfer. First, those with delayed transfer might be different in some way from those transferred immediately. For example, we found that those with delayed transfer were older. The finding that increasing age is associated with a delay in ICU transfer is interesting, and may reflect physiologic differences in older patients compared to younger ones. For example, older patients have a lower maximum heart rate and thus may not develop the same level of vital sign abnormalities that younger patients do, causing them to be inappropriately left on the wards for too long.[18] In addition, patients with delayed transfer had more deranged renal function and lower blood pressure. It is unknown whether these organ dysfunctions would have been prevented by earlier transfer and to what degree they were related to chronic conditions. However, delayed transfer was still associated with increased mortality even after controlling for age, vital sign and laboratory values, and eCART on ward admission. It may also be possible that patients with delayed transfer received early and appropriate treatment on the wards but failed to improve and thus required ICU transfer. We did not have access to orders in this large database, so this theory will need to be investigated in future work. Finally, the most likely explanation for our findings is that earlier identification and treatment improves outcomes of critically ill patients on the wards, which is consistent with the findings of previous studies.[1, 5, 9, 10] Our study demonstrates that early identification of critical illness is crucial, and that delayed treatment can rapidly lead to increased mortality and LOS.

Our comparison of eCART score trajectory showed that patients transferred within 6 hours of onset of critical illness had a more rapid rise in eCART score over the preceding time period, whereas patients who experienced transfer delay showed a slower increase in eCART score. One explanation for this finding is that patients who decompensate more rapidly are in turn more readily recognizable to providers, whereas patients who experience a more insidious clinical deterioration are recognized later in the process, which then leads to a delay in escalation of care. This hypothesis underlines the importance of utilizing an objective marker of illness that is calculated longitudinally and in real time, as opposed to relying upon provider recognition alone. In fact, we have recently demonstrated that eCART is more accurate and identifies patients earlier than standard rapid response team activation.[19]

There are several important implications of our findings. First, it highlights the potential impact that early warning scores, particular those that are evidence based, can have on the outcomes of hospitalized patients. Second, it suggests that it is important to include age in early warning scores. Previous studies have been mixed as to whether the inclusion of age improves detection of outcomes on the wards, although the method of inclusion of age has been variable in terms of its weighting.[20, 21, 22] Our study found that older patients were more likely to be left on the wards longer prior to ICU transfer after becoming critically ill. By incorporating age into early warning scores, both accuracy and early recognition of critical illness may be improved. Finally, our finding that the trends of the eCART score differed among patients who were immediately transferred to the ICU, and who had a delay in their transfer, suggests that adding vital sign trends to early warning scores may further improve their accuracy and ability to serve as clinical decision support tools.

Our study is unique in that we used an objective measure of critical illness and then examined outcomes after patients reached this threshold on the wards. This overcomes the subjectivity of using evaluation by the ICU team or rapid response team as the starting point, as previous studies have shown a failure to call for help when patients become critically ill on the wards.[2, 11, 23] By using the eCART score, which contains commonly collected electronic health record data and can be calculated electronically in real time, we were able to calculate the score for patients on the wards and in the ICU. This allowed us to examine trends in the eCART score over time to find clues as to why some patients are transferred late to the ICU and why these late transfers have worse outcomes than those transferred earlier. Another strength is the large multicenter database used for the analysis, which included an urban tertiary care hospital, suburban teaching hospitals, and a community nonteaching hospital.

Our study has several limitations. First, we utilized just 1 of many potential measures of critical illness and a cutoff that only included one‐third of patients ultimately transferred to the ICU. However, by using the eCART score, we were able to track a patient's physiologic status over time and remove the variability that comes with using subjective definitions of critical illness. Furthermore, we utilized a high‐specificity cutoff for eCART to ensure that transferred patients had significantly deranged physiology and to avoid including planned transfers to the ICU. It is likely that some patients who were critically ill with less deranged physiology that would have benefitted from earlier transfer were excluded from the study. Second, we were unable to determine the cause of physiologic deterioration for patients in our study due to the large number of included patients. In addition, we did not have code status, comorbidities, or reason for ICU admission available in the dataset. It is likely that the impact of delayed transfer varies by the indication for ICU admission and chronic disease burden. It is also possible that controlling for these unmeasured factors could negate the beneficial association seen for earlier ICU admission. However, our finding of such a strong relationship between time to transfer and mortality after controlling for several important variables suggests that early recognition of critical illness is beneficial to many patients on the wards. Third, due to its observational nature, our study cannot estimate the true impact of timely ICU transfer on critically ill ward patient outcomes. Future clinical trials will be needed to determine the impact of electronic early warning scores on patient outcomes.

In conclusion, delayed ICU transfer is associated with significantly increased hospital LOS and mortality. This association highlights the need for ongoing work toward both the implementation of an evidence‐based risk stratification tool as well as development of effective critical care outreach resources for patients decompensating on the wards. Real‐time use of a validated early warning score, such as eCART, could potentially lead to more timely ICU transfer for critically ill patients and reduced rates of preventable in‐hospital death.

Patients on hospital wards may become critically ill due to worsening of the underlying condition that was the cause of their admission or acquisition of a new hospital‐acquired illness. Once physiologic deterioration occurs, some patients are evaluated and quickly transferred to the intensive care unit (ICU), whereas others are left on the wards until further deterioration occurs. Because many critical illness syndromes benefit from early intervention, such as sepsis and respiratory failure, early transfer to the ICU for treatment may improve patient outcomes, and conversely, delays in ICU transfer may lead to increased mortality and length of stay (LOS) in critically ill ward patients.[1, 2] However, the timeliness of that transfer is dependent on numerous changing variables, such as ICU bed availability, clinician identification of the deterioration, and clinical judgment regarding the appropriate transfer thresholds.[2, 3, 4, 5, 6, 7] As a result, there is a large degree of heterogeneity in the severity of illness of patients at the time of ICU transfer and in patient outcomes.[6, 8]

Previous studies investigating the association between delayed ICU transfer and patient outcomes have typically utilized the time of consultation by the ICU team to denote the onset of critical illness.[5, 6, 9, 10] However, the decision to transfer a patient to the ICU is often subjective, and previous studies have found an alarmingly high rate of errors in diagnosis and management of critically ill ward patients, including the failure to call for help.[2, 11] Therefore, a more objective tool for quantifying critical illness is necessary for determining the onset of critical illness and quantifying the association of transfer delay with patient outcomes.

Early warning scores, which are designed to detect critical illness on the wards, represent objective measures of critical illness that can be easily calculated in ward patients.[12] The aim of this study was to utilize the electronic Cardiac Arrest Risk Triage (eCART) score, a previously published, statistically derived early warning score that utilizes demographic, vital sign, and laboratory data, as an objective measure of critical illness to estimate the effect of delayed ICU transfer on patient outcomes in a large, multicenter database.[13] We chose 6 hours as the cutoff for delay in this study a priori because it is a threshold noted to be an important time period in critical illness syndromes, such as sepsis.[14, 15]

METHODS

All patients admitted to the medical‐surgical wards at 5 hospitals between November 2008 and January 2013 were eligible for inclusion in this observational cohort study. Further details of the hospital populations have been previously described.[13] A waiver of consent was granted by NorthShore University HealthSystem (IRB #EH11‐258) and the University of Chicago Institutional Review Board (IRB #16995A) based on general impracticability and minimal harm. Collection of patient information was designed to comply with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) regulations.

RESULTS

A total of 269,999 admissions had documented vital signs on the hospital wards during the study period, including 11,995 patients who were either transferred from the wards to the ICU (n=11,636) or who suffered a cardiac arrest on the wards (n=359) during their initial ward stay. Of these patients, 3789 reached an eCART score at the 95% specificity cutoff (critical eCART score of 60) within 24 hours of transfer. The median time from first critical eCART value to ICU transfer was 5.4 hours (interquartile range (IQR), 214 hours; mean, 8 hours). Compared to patients without delayed ICU transfer, those with delayed transfer were slightly older (median age, 73 [IQR, 6083] years vs 71 [IQR, 5882] years; P=0.002), whereas all other characteristics were similar (Table 1). Table 2 shows comparisons of vital sign and laboratory results for delayed and nondelayed transfers at the time of ICU transfer. As shown, patients with delayed transfer had lower median respiratory rate, blood pressure, heart rate, and hemoglobin, but higher median white blood cell count and creatinine.

Delayed transfer occurred in 46% of patients (n=1734) and was associated with increased in‐hospital mortality (33.2% vs 24.5%, P<0.001). This relationship was linear, with each 1‐hour increase in transfer delay associated with a 3% increase in the odds of in‐hospital death (P<0.001) (Figure 1). The association between length of transfer delay and hospital mortality remained unchanged after controlling for age, sex, surgical status, initial eCART score on the wards, vital signs, laboratory values, and whether the ICU transfer was due to a cardiac arrest (3% increase per hour, P<0.001). This association did not vary based on the initial eCART score on the wards (P=0.71 for interaction). Additionally, despite having similar median hospital lengths of stay prior to first critical eCART score (1.6 vs 1.5 days, P=0.04), patients experiencing delayed ICU transfer who survived to discharge had a longer median hospital LOS by 2 days compared to those with nondelayed transfer who survived to discharge (median LOS, 13 (821) days vs 11 (719) days, P=0.01). The change in eCART score over time in the 12 hours before first reaching the critical eCART score until ICU transfer is shown in Figure 2 for patients with delayed and nondelayed transfer. As shown, patients transferred within 6 hours had a more rapid rise in eCART score prior to ICU transfer compared to those with a delayed transfer. This difference in trajectories between delayed and nondelayed patients was similar in patients with low (<13), intermediate (1359), and high (60) initial eCART scores on the wards. A regression model investigating the association between eCART slope prior to ICU transfer and time to ICU transfer demonstrated that a steeper slope was significantly associated with a decreased time to ICU transfer (P<0.01).

Figure 1

Figure 2

DISCUSSION

We found that a delay in transfer to the ICU after reaching a predefined objective threshold of critical illness was associated with a significant increase in hospital mortality and hospital LOS. We also discovered a significant association between critical illness trajectory and delays in transfer, suggesting that caregivers may not recognize more subtle trends in critical illness. This work highlights the importance of timely transfer to the ICU for critically ill ward patients, which can be affected by several factors such as ICU bed availability and caregiver recognition and triage decisions. Our findings have significant implications for patient safety on the wards and provide further evidence for implementing early warning scores into practice to aid with clinical decision making.

Our findings of increased mortality with delayed ICU transfer are consistent with previous studies.[1, 5, 9] For example, Young et al. compared ICU mortality between delayed and nondelayed transfers in 91 consecutive patients with noncardiac diagnoses at a community hospital.[1] They also used predefined criteria for critical illness, and found that delayed transfers had a higher ICU mortality than nondelayed patients (41% vs 11%). However, their criteria for critical illness only had a specificity of 13% for predicting ICU transfer, compared to 95% in our study, suggesting that our threshold is more consistent with critical illness. Another study, by Cardoso and colleagues, investigated the impact of delayed ICU admission due to bed shortages on ICU mortality in 401 patients at a university hospital.[9] Of those patients deemed appropriate for transfer to the ICU but who had to wait for a bed to become available, the median wait time for a bed was 18 hours. They found that each hour of waiting was associated with a 1.5% increase in ICU death. A similar study by Robert and colleagues investigated the impact of delayed or refused ICU admission due to a lack of bed availability.[5] Patients deemed too sick (or too well) to benefit from ICU transfer were excluded. Twenty‐eightday and 60‐day mortality were higher in the admitted group compared to those not admitted, although this finding was not statistically significant. In addition, patients later admitted to the ICU once a bed became available (median wait time, 6 hours; n=89) had higher 28‐day mortality than those admitted immediately (adjusted odds ratio, 1.78; P=0.05). Several other studies have investigated the impact of ICU refusal for reasons that included bed shortages, and found increased mortality in those not admitted to the ICU.[16, 17] However, many of these studies included patients deemed too sick or too well to be transferred to the ICU in the group of nonadmitted patients. Our study adds to this literature by utilizing a highly specific objective measure of critical illness and by including all patients on the wards who reached this threshold, rather than only those for whom a consult was requested.

There are several potential explanations for our finding of increased mortality with delayed ICU transfer. First, those with delayed transfer might be different in some way from those transferred immediately. For example, we found that those with delayed transfer were older. The finding that increasing age is associated with a delay in ICU transfer is interesting, and may reflect physiologic differences in older patients compared to younger ones. For example, older patients have a lower maximum heart rate and thus may not develop the same level of vital sign abnormalities that younger patients do, causing them to be inappropriately left on the wards for too long.[18] In addition, patients with delayed transfer had more deranged renal function and lower blood pressure. It is unknown whether these organ dysfunctions would have been prevented by earlier transfer and to what degree they were related to chronic conditions. However, delayed transfer was still associated with increased mortality even after controlling for age, vital sign and laboratory values, and eCART on ward admission. It may also be possible that patients with delayed transfer received early and appropriate treatment on the wards but failed to improve and thus required ICU transfer. We did not have access to orders in this large database, so this theory will need to be investigated in future work. Finally, the most likely explanation for our findings is that earlier identification and treatment improves outcomes of critically ill patients on the wards, which is consistent with the findings of previous studies.[1, 5, 9, 10] Our study demonstrates that early identification of critical illness is crucial, and that delayed treatment can rapidly lead to increased mortality and LOS.

Our comparison of eCART score trajectory showed that patients transferred within 6 hours of onset of critical illness had a more rapid rise in eCART score over the preceding time period, whereas patients who experienced transfer delay showed a slower increase in eCART score. One explanation for this finding is that patients who decompensate more rapidly are in turn more readily recognizable to providers, whereas patients who experience a more insidious clinical deterioration are recognized later in the process, which then leads to a delay in escalation of care. This hypothesis underlines the importance of utilizing an objective marker of illness that is calculated longitudinally and in real time, as opposed to relying upon provider recognition alone. In fact, we have recently demonstrated that eCART is more accurate and identifies patients earlier than standard rapid response team activation.[19]

There are several important implications of our findings. First, it highlights the potential impact that early warning scores, particular those that are evidence based, can have on the outcomes of hospitalized patients. Second, it suggests that it is important to include age in early warning scores. Previous studies have been mixed as to whether the inclusion of age improves detection of outcomes on the wards, although the method of inclusion of age has been variable in terms of its weighting.[20, 21, 22] Our study found that older patients were more likely to be left on the wards longer prior to ICU transfer after becoming critically ill. By incorporating age into early warning scores, both accuracy and early recognition of critical illness may be improved. Finally, our finding that the trends of the eCART score differed among patients who were immediately transferred to the ICU, and who had a delay in their transfer, suggests that adding vital sign trends to early warning scores may further improve their accuracy and ability to serve as clinical decision support tools.

Our study is unique in that we used an objective measure of critical illness and then examined outcomes after patients reached this threshold on the wards. This overcomes the subjectivity of using evaluation by the ICU team or rapid response team as the starting point, as previous studies have shown a failure to call for help when patients become critically ill on the wards.[2, 11, 23] By using the eCART score, which contains commonly collected electronic health record data and can be calculated electronically in real time, we were able to calculate the score for patients on the wards and in the ICU. This allowed us to examine trends in the eCART score over time to find clues as to why some patients are transferred late to the ICU and why these late transfers have worse outcomes than those transferred earlier. Another strength is the large multicenter database used for the analysis, which included an urban tertiary care hospital, suburban teaching hospitals, and a community nonteaching hospital.

Our study has several limitations. First, we utilized just 1 of many potential measures of critical illness and a cutoff that only included one‐third of patients ultimately transferred to the ICU. However, by using the eCART score, we were able to track a patient's physiologic status over time and remove the variability that comes with using subjective definitions of critical illness. Furthermore, we utilized a high‐specificity cutoff for eCART to ensure that transferred patients had significantly deranged physiology and to avoid including planned transfers to the ICU. It is likely that some patients who were critically ill with less deranged physiology that would have benefitted from earlier transfer were excluded from the study. Second, we were unable to determine the cause of physiologic deterioration for patients in our study due to the large number of included patients. In addition, we did not have code status, comorbidities, or reason for ICU admission available in the dataset. It is likely that the impact of delayed transfer varies by the indication for ICU admission and chronic disease burden. It is also possible that controlling for these unmeasured factors could negate the beneficial association seen for earlier ICU admission. However, our finding of such a strong relationship between time to transfer and mortality after controlling for several important variables suggests that early recognition of critical illness is beneficial to many patients on the wards. Third, due to its observational nature, our study cannot estimate the true impact of timely ICU transfer on critically ill ward patient outcomes. Future clinical trials will be needed to determine the impact of electronic early warning scores on patient outcomes.

In conclusion, delayed ICU transfer is associated with significantly increased hospital LOS and mortality. This association highlights the need for ongoing work toward both the implementation of an evidence‐based risk stratification tool as well as development of effective critical care outreach resources for patients decompensating on the wards. Real‐time use of a validated early warning score, such as eCART, could potentially lead to more timely ICU transfer for critically ill patients and reduced rates of preventable in‐hospital death.

Patients on hospital wards may become critically ill due to worsening of the underlying condition that was the cause of their admission or acquisition of a new hospital‐acquired illness. Once physiologic deterioration occurs, some patients are evaluated and quickly transferred to the intensive care unit (ICU), whereas others are left on the wards until further deterioration occurs. Because many critical illness syndromes benefit from early intervention, such as sepsis and respiratory failure, early transfer to the ICU for treatment may improve patient outcomes, and conversely, delays in ICU transfer may lead to increased mortality and length of stay (LOS) in critically ill ward patients.[1, 2] However, the timeliness of that transfer is dependent on numerous changing variables, such as ICU bed availability, clinician identification of the deterioration, and clinical judgment regarding the appropriate transfer thresholds.[2, 3, 4, 5, 6, 7] As a result, there is a large degree of heterogeneity in the severity of illness of patients at the time of ICU transfer and in patient outcomes.[6, 8]

Previous studies investigating the association between delayed ICU transfer and patient outcomes have typically utilized the time of consultation by the ICU team to denote the onset of critical illness.[5, 6, 9, 10] However, the decision to transfer a patient to the ICU is often subjective, and previous studies have found an alarmingly high rate of errors in diagnosis and management of critically ill ward patients, including the failure to call for help.[2, 11] Therefore, a more objective tool for quantifying critical illness is necessary for determining the onset of critical illness and quantifying the association of transfer delay with patient outcomes.

Early warning scores, which are designed to detect critical illness on the wards, represent objective measures of critical illness that can be easily calculated in ward patients.[12] The aim of this study was to utilize the electronic Cardiac Arrest Risk Triage (eCART) score, a previously published, statistically derived early warning score that utilizes demographic, vital sign, and laboratory data, as an objective measure of critical illness to estimate the effect of delayed ICU transfer on patient outcomes in a large, multicenter database.[13] We chose 6 hours as the cutoff for delay in this study a priori because it is a threshold noted to be an important time period in critical illness syndromes, such as sepsis.[14, 15]

METHODS

All patients admitted to the medical‐surgical wards at 5 hospitals between November 2008 and January 2013 were eligible for inclusion in this observational cohort study. Further details of the hospital populations have been previously described.[13] A waiver of consent was granted by NorthShore University HealthSystem (IRB #EH11‐258) and the University of Chicago Institutional Review Board (IRB #16995A) based on general impracticability and minimal harm. Collection of patient information was designed to comply with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) regulations.

RESULTS

A total of 269,999 admissions had documented vital signs on the hospital wards during the study period, including 11,995 patients who were either transferred from the wards to the ICU (n=11,636) or who suffered a cardiac arrest on the wards (n=359) during their initial ward stay. Of these patients, 3789 reached an eCART score at the 95% specificity cutoff (critical eCART score of 60) within 24 hours of transfer. The median time from first critical eCART value to ICU transfer was 5.4 hours (interquartile range (IQR), 214 hours; mean, 8 hours). Compared to patients without delayed ICU transfer, those with delayed transfer were slightly older (median age, 73 [IQR, 6083] years vs 71 [IQR, 5882] years; P=0.002), whereas all other characteristics were similar (Table 1). Table 2 shows comparisons of vital sign and laboratory results for delayed and nondelayed transfers at the time of ICU transfer. As shown, patients with delayed transfer had lower median respiratory rate, blood pressure, heart rate, and hemoglobin, but higher median white blood cell count and creatinine.

Delayed transfer occurred in 46% of patients (n=1734) and was associated with increased in‐hospital mortality (33.2% vs 24.5%, P<0.001). This relationship was linear, with each 1‐hour increase in transfer delay associated with a 3% increase in the odds of in‐hospital death (P<0.001) (Figure 1). The association between length of transfer delay and hospital mortality remained unchanged after controlling for age, sex, surgical status, initial eCART score on the wards, vital signs, laboratory values, and whether the ICU transfer was due to a cardiac arrest (3% increase per hour, P<0.001). This association did not vary based on the initial eCART score on the wards (P=0.71 for interaction). Additionally, despite having similar median hospital lengths of stay prior to first critical eCART score (1.6 vs 1.5 days, P=0.04), patients experiencing delayed ICU transfer who survived to discharge had a longer median hospital LOS by 2 days compared to those with nondelayed transfer who survived to discharge (median LOS, 13 (821) days vs 11 (719) days, P=0.01). The change in eCART score over time in the 12 hours before first reaching the critical eCART score until ICU transfer is shown in Figure 2 for patients with delayed and nondelayed transfer. As shown, patients transferred within 6 hours had a more rapid rise in eCART score prior to ICU transfer compared to those with a delayed transfer. This difference in trajectories between delayed and nondelayed patients was similar in patients with low (<13), intermediate (1359), and high (60) initial eCART scores on the wards. A regression model investigating the association between eCART slope prior to ICU transfer and time to ICU transfer demonstrated that a steeper slope was significantly associated with a decreased time to ICU transfer (P<0.01).

Figure 1

Figure 2

DISCUSSION

We found that a delay in transfer to the ICU after reaching a predefined objective threshold of critical illness was associated with a significant increase in hospital mortality and hospital LOS. We also discovered a significant association between critical illness trajectory and delays in transfer, suggesting that caregivers may not recognize more subtle trends in critical illness. This work highlights the importance of timely transfer to the ICU for critically ill ward patients, which can be affected by several factors such as ICU bed availability and caregiver recognition and triage decisions. Our findings have significant implications for patient safety on the wards and provide further evidence for implementing early warning scores into practice to aid with clinical decision making.

Our findings of increased mortality with delayed ICU transfer are consistent with previous studies.[1, 5, 9] For example, Young et al. compared ICU mortality between delayed and nondelayed transfers in 91 consecutive patients with noncardiac diagnoses at a community hospital.[1] They also used predefined criteria for critical illness, and found that delayed transfers had a higher ICU mortality than nondelayed patients (41% vs 11%). However, their criteria for critical illness only had a specificity of 13% for predicting ICU transfer, compared to 95% in our study, suggesting that our threshold is more consistent with critical illness. Another study, by Cardoso and colleagues, investigated the impact of delayed ICU admission due to bed shortages on ICU mortality in 401 patients at a university hospital.[9] Of those patients deemed appropriate for transfer to the ICU but who had to wait for a bed to become available, the median wait time for a bed was 18 hours. They found that each hour of waiting was associated with a 1.5% increase in ICU death. A similar study by Robert and colleagues investigated the impact of delayed or refused ICU admission due to a lack of bed availability.[5] Patients deemed too sick (or too well) to benefit from ICU transfer were excluded. Twenty‐eightday and 60‐day mortality were higher in the admitted group compared to those not admitted, although this finding was not statistically significant. In addition, patients later admitted to the ICU once a bed became available (median wait time, 6 hours; n=89) had higher 28‐day mortality than those admitted immediately (adjusted odds ratio, 1.78; P=0.05). Several other studies have investigated the impact of ICU refusal for reasons that included bed shortages, and found increased mortality in those not admitted to the ICU.[16, 17] However, many of these studies included patients deemed too sick or too well to be transferred to the ICU in the group of nonadmitted patients. Our study adds to this literature by utilizing a highly specific objective measure of critical illness and by including all patients on the wards who reached this threshold, rather than only those for whom a consult was requested.

There are several potential explanations for our finding of increased mortality with delayed ICU transfer. First, those with delayed transfer might be different in some way from those transferred immediately. For example, we found that those with delayed transfer were older. The finding that increasing age is associated with a delay in ICU transfer is interesting, and may reflect physiologic differences in older patients compared to younger ones. For example, older patients have a lower maximum heart rate and thus may not develop the same level of vital sign abnormalities that younger patients do, causing them to be inappropriately left on the wards for too long.[18] In addition, patients with delayed transfer had more deranged renal function and lower blood pressure. It is unknown whether these organ dysfunctions would have been prevented by earlier transfer and to what degree they were related to chronic conditions. However, delayed transfer was still associated with increased mortality even after controlling for age, vital sign and laboratory values, and eCART on ward admission. It may also be possible that patients with delayed transfer received early and appropriate treatment on the wards but failed to improve and thus required ICU transfer. We did not have access to orders in this large database, so this theory will need to be investigated in future work. Finally, the most likely explanation for our findings is that earlier identification and treatment improves outcomes of critically ill patients on the wards, which is consistent with the findings of previous studies.[1, 5, 9, 10] Our study demonstrates that early identification of critical illness is crucial, and that delayed treatment can rapidly lead to increased mortality and LOS.

Our comparison of eCART score trajectory showed that patients transferred within 6 hours of onset of critical illness had a more rapid rise in eCART score over the preceding time period, whereas patients who experienced transfer delay showed a slower increase in eCART score. One explanation for this finding is that patients who decompensate more rapidly are in turn more readily recognizable to providers, whereas patients who experience a more insidious clinical deterioration are recognized later in the process, which then leads to a delay in escalation of care. This hypothesis underlines the importance of utilizing an objective marker of illness that is calculated longitudinally and in real time, as opposed to relying upon provider recognition alone. In fact, we have recently demonstrated that eCART is more accurate and identifies patients earlier than standard rapid response team activation.[19]

There are several important implications of our findings. First, it highlights the potential impact that early warning scores, particular those that are evidence based, can have on the outcomes of hospitalized patients. Second, it suggests that it is important to include age in early warning scores. Previous studies have been mixed as to whether the inclusion of age improves detection of outcomes on the wards, although the method of inclusion of age has been variable in terms of its weighting.[20, 21, 22] Our study found that older patients were more likely to be left on the wards longer prior to ICU transfer after becoming critically ill. By incorporating age into early warning scores, both accuracy and early recognition of critical illness may be improved. Finally, our finding that the trends of the eCART score differed among patients who were immediately transferred to the ICU, and who had a delay in their transfer, suggests that adding vital sign trends to early warning scores may further improve their accuracy and ability to serve as clinical decision support tools.

Our study is unique in that we used an objective measure of critical illness and then examined outcomes after patients reached this threshold on the wards. This overcomes the subjectivity of using evaluation by the ICU team or rapid response team as the starting point, as previous studies have shown a failure to call for help when patients become critically ill on the wards.[2, 11, 23] By using the eCART score, which contains commonly collected electronic health record data and can be calculated electronically in real time, we were able to calculate the score for patients on the wards and in the ICU. This allowed us to examine trends in the eCART score over time to find clues as to why some patients are transferred late to the ICU and why these late transfers have worse outcomes than those transferred earlier. Another strength is the large multicenter database used for the analysis, which included an urban tertiary care hospital, suburban teaching hospitals, and a community nonteaching hospital.

Our study has several limitations. First, we utilized just 1 of many potential measures of critical illness and a cutoff that only included one‐third of patients ultimately transferred to the ICU. However, by using the eCART score, we were able to track a patient's physiologic status over time and remove the variability that comes with using subjective definitions of critical illness. Furthermore, we utilized a high‐specificity cutoff for eCART to ensure that transferred patients had significantly deranged physiology and to avoid including planned transfers to the ICU. It is likely that some patients who were critically ill with less deranged physiology that would have benefitted from earlier transfer were excluded from the study. Second, we were unable to determine the cause of physiologic deterioration for patients in our study due to the large number of included patients. In addition, we did not have code status, comorbidities, or reason for ICU admission available in the dataset. It is likely that the impact of delayed transfer varies by the indication for ICU admission and chronic disease burden. It is also possible that controlling for these unmeasured factors could negate the beneficial association seen for earlier ICU admission. However, our finding of such a strong relationship between time to transfer and mortality after controlling for several important variables suggests that early recognition of critical illness is beneficial to many patients on the wards. Third, due to its observational nature, our study cannot estimate the true impact of timely ICU transfer on critically ill ward patient outcomes. Future clinical trials will be needed to determine the impact of electronic early warning scores on patient outcomes.

In conclusion, delayed ICU transfer is associated with significantly increased hospital LOS and mortality. This association highlights the need for ongoing work toward both the implementation of an evidence‐based risk stratification tool as well as development of effective critical care outreach resources for patients decompensating on the wards. Real‐time use of a validated early warning score, such as eCART, could potentially lead to more timely ICU transfer for critically ill patients and reduced rates of preventable in‐hospital death.

Altered mental status (AMS) is a complex spectrum of cognitive deficits that includes orientation, memory, language, visuospatial ability, and perception.[1] The clinical definitions of both delirium and dementia include AMS as a hallmark clinical prerequisite. Regardless of etiology, this broader AMS definition is particularly salient in the hospital setting, where AMS is present in up to 60% of inpatients and is associated with longer hospital stay as well as increased morbidity and mortality.[2, 3] Not surprisingly, due to the complexity of identifying and assessing changes in mental status, clinically relevant AMS is often undetected among inpatients.[2] However, when detected, the most common causes of AMS (infection, polypharmacy, and pain) are treatable, suggesting that early AMS identification could alert clinicians to early signs of clinical decompensation, potentially improving clinical outcomes.[4]

Because rapid and systemic clinical detection of AMS is limited by the complexity of mental status, a number of assessments have been created, each with their own advantages, limitations, and target populations. These assessments are often limited by time‐intensive administration, subjectivity of mental status assessment, and lack of sensitivity in general medicine patients. Time‐intensive measures, such as the Short Portable Mental Status Questionnaire (SPMSQ) have utility in the research setting, whereas current common clinical risk stratification tools (eg, National Early Warning Score) utilize simpler measures such as the Alert, Voice, Pain, Unresponsive (AVPU) and Glasgow Coma Scale (GCS) as measures of mental status.[2, 5, 6, 7, 8, 9]

To address the need for a brief, clinically feasible, accurate tool in clinical detection of AMS, our group developed a mobile application for working memory testing, the Functional Assessment of Mentation (FAM^TM). In this study, we aimed to identify baseline scoring distributions of the FAM^TM in a nonhospitalized subgroup, as well as assess the correlation of the FAM^TM to discharge disposition and compare it to the SPMSQ in inpatients.

METHODS

FAM^TM Application

The FAM^TM application is a bedside tool for working memory assessment developed for the iPhone mobile operating system (Apple Inc., Cupertino, CA) and presented on an iPad mini (Apple). The application interface displays 4 colored rectangles individually labeled with a number (see Supporting Figure 1 in the online version of this article). The testing portion of the application presents a sequence of numbered rectangles, illuminated 1 at a time in random order. Subjects are prompted first to watch and remember the sequence and then repeat the sequence by touching the screen within each numbered rectangle. Successful reproduction of the sequence is followed by a distinct and longer sequence, whereas unsuccessful attempts are followed by a shorter sequence. The final FAM^TM score corresponds to the longest sequence of rectangles successfully repeated by the subject.

Figure 1

RESULTS

A total of 931 subjects were enrolled in the study. In the nonhospitalized subgroup, 651 consented to study participation and 612 were included in final analysis. Subjects were excluded if they started but did not complete the application (n = 36) or were under the age of 18 years (n = 3). Of the 363 hospitalized subjects approached for enrollment, 319 were included in the final analysis. Subjects were excluded if they refused to participate (n = 23), were under the age of 18 (n = 2), had technical failures (n = 5), or had physical or visual limitations that precluded them from participation (n = 14). Within the hospitalized subgroup, 268 subjects were discharged home (85%). The table displays demographics and score distributions by subgroup.1

The median FAM^TM score for the combined study population was 5 (interquartile range [IQR] 36), and median time to completion was 55 seconds (IQR 4567 seconds). A graded reduction was found in the FAM^TM score for all stepwise comparisons between nonhospitalized subjects, hospitalized subjects discharged home, and hospitalized subjects not discharged home (median 5 [IQR 47] vs 5 [IQR 36] vs 3 [IQR 15]; P < 0.001 for all pairwise comparisons). The AUC for the FAM^TM predicting discharge disposition (home vs not) was 0.66 (95% confidence interval [CI]: 0.58‐0.74]. After adjusting for confounders, higher FAM^TM scores were independently associated with not being hospitalized, being discharged home, higher levels of education, younger age, and white race (see Supporting Table 1 in the online version of this article). Additionally, in the hospitalized subgroup, decreasing FAM^TM score was significantly correlated with increasing errors on the SPMSQ (Spearman = 0.27, P < 0.001), whereas the GCS score was not correlated with the SPMSQ (Spearman = 0.05, P = 0.40) (Figure 1).

DISCUSSION

We demonstrated the utility of a rapid and accurate mobile application for assessment of mental status. The FAM^TM was able to be quickly administered with a median time to completion of approximately 1 minute. The ability to detect mild alterations in mental status was shown through concurrent validity by FAM^TM correlation with the SPMSQ and predictive validity with the association between the FAM^TM and discharge disposition. Our study highlights the potential for the FAM^TM to be used as a sensitive marker of AMS.

The novel design of the FAM^TM presents unique advantages compared to current mental status testing. First, the FAM^TM could allow patients with hearing impairment or language barriers to complete a mental status assessment. Additionally, the approximately 1‐minute median time to completion is much faster than other established mental status assessments including the SPMSQ (510 minutes). Compared to the SPMSQ taking 5 minutes, in a 400‐bed hospital, taken once per nursing shift, the FAM^TM would save approximately 20,000 hours and 10 nursing full‐time equivalents per year.[5] Finally, many current mental status tests such as the Confusion Assessment Model utilize subjective mental status assessments.[2] However, the FAM^TM is designed to be conducted through self‐assessment and, thus, could theoretically be free of observer bias. This potential for self‐administration expands beyond other proposed alternative testing mechanisms of the AMS such as ultrabrief assessments that include items such as asking subjects the months of the year backwards, and what is the day of the week?, and assessing arousal.[12, 13, 14]

In research settings and commonly in hospitals, the GCS and AVPU are used clinically for mental status assessment of hospitalized patients.[6, 15] However, similar to previous literature, our study found that the vast majority of hospitalized patients were defined as neurologically intact by the GCS, which is the more accurate predictor of the 2.[7] One major strength of the FAM^TM was that it identified an extensive gradation of scores for patients previously labeled as merely alert, providing greater resolution than the GCS in quantifying mental status.

One of the key benefits of the FAM^TM is that it can be measured longitudinally over the course of a patient's hospital stay. Therefore, once a baseline FAM^TM score is established, variation from the patient's personal baseline could indicate mental status deterioration, which would not be affected by the patient's demographics, health status, or underlying neurocognitive deficits.

There were important limitations to this study. First, limited generalizability of these data may exist due to the single‐center setting and patient population. However, this initial study provides pilot data for further expansion into the potential broad applicability of the FAM^TM to other patient populations and settings. Additionally, the cost of large‐scale implementation of the FAM^TM is unknown and was beyond the scope of this pilot study. However, to reduce costs, the FAM^TM technology could be integrated into existing hospital technology infrastructure. Finally, the scope of this study prevented a complete assessment of all validity measures or comparison to other mental status assessments such as the digit span or serial sevens tests. However, predictive and concurrent validity were assessed with comparison by discharge disposition, SPMSQ, and GCS scores.

In conclusion, this pilot study identifies the FAM^TM application as a potentially clinically useful, novel, rapid, and feasible assessment tool of mental status in a general medicine inpatient setting.

Altered mental status (AMS) is a complex spectrum of cognitive deficits that includes orientation, memory, language, visuospatial ability, and perception.[1] The clinical definitions of both delirium and dementia include AMS as a hallmark clinical prerequisite. Regardless of etiology, this broader AMS definition is particularly salient in the hospital setting, where AMS is present in up to 60% of inpatients and is associated with longer hospital stay as well as increased morbidity and mortality.[2, 3] Not surprisingly, due to the complexity of identifying and assessing changes in mental status, clinically relevant AMS is often undetected among inpatients.[2] However, when detected, the most common causes of AMS (infection, polypharmacy, and pain) are treatable, suggesting that early AMS identification could alert clinicians to early signs of clinical decompensation, potentially improving clinical outcomes.[4]

Because rapid and systemic clinical detection of AMS is limited by the complexity of mental status, a number of assessments have been created, each with their own advantages, limitations, and target populations. These assessments are often limited by time‐intensive administration, subjectivity of mental status assessment, and lack of sensitivity in general medicine patients. Time‐intensive measures, such as the Short Portable Mental Status Questionnaire (SPMSQ) have utility in the research setting, whereas current common clinical risk stratification tools (eg, National Early Warning Score) utilize simpler measures such as the Alert, Voice, Pain, Unresponsive (AVPU) and Glasgow Coma Scale (GCS) as measures of mental status.[2, 5, 6, 7, 8, 9]

To address the need for a brief, clinically feasible, accurate tool in clinical detection of AMS, our group developed a mobile application for working memory testing, the Functional Assessment of Mentation (FAM^TM). In this study, we aimed to identify baseline scoring distributions of the FAM^TM in a nonhospitalized subgroup, as well as assess the correlation of the FAM^TM to discharge disposition and compare it to the SPMSQ in inpatients.

METHODS

FAM^TM Application

The FAM^TM application is a bedside tool for working memory assessment developed for the iPhone mobile operating system (Apple Inc., Cupertino, CA) and presented on an iPad mini (Apple). The application interface displays 4 colored rectangles individually labeled with a number (see Supporting Figure 1 in the online version of this article). The testing portion of the application presents a sequence of numbered rectangles, illuminated 1 at a time in random order. Subjects are prompted first to watch and remember the sequence and then repeat the sequence by touching the screen within each numbered rectangle. Successful reproduction of the sequence is followed by a distinct and longer sequence, whereas unsuccessful attempts are followed by a shorter sequence. The final FAM^TM score corresponds to the longest sequence of rectangles successfully repeated by the subject.

Figure 1

RESULTS

A total of 931 subjects were enrolled in the study. In the nonhospitalized subgroup, 651 consented to study participation and 612 were included in final analysis. Subjects were excluded if they started but did not complete the application (n = 36) or were under the age of 18 years (n = 3). Of the 363 hospitalized subjects approached for enrollment, 319 were included in the final analysis. Subjects were excluded if they refused to participate (n = 23), were under the age of 18 (n = 2), had technical failures (n = 5), or had physical or visual limitations that precluded them from participation (n = 14). Within the hospitalized subgroup, 268 subjects were discharged home (85%). The table displays demographics and score distributions by subgroup.1

The median FAM^TM score for the combined study population was 5 (interquartile range [IQR] 36), and median time to completion was 55 seconds (IQR 4567 seconds). A graded reduction was found in the FAM^TM score for all stepwise comparisons between nonhospitalized subjects, hospitalized subjects discharged home, and hospitalized subjects not discharged home (median 5 [IQR 47] vs 5 [IQR 36] vs 3 [IQR 15]; P < 0.001 for all pairwise comparisons). The AUC for the FAM^TM predicting discharge disposition (home vs not) was 0.66 (95% confidence interval [CI]: 0.58‐0.74]. After adjusting for confounders, higher FAM^TM scores were independently associated with not being hospitalized, being discharged home, higher levels of education, younger age, and white race (see Supporting Table 1 in the online version of this article). Additionally, in the hospitalized subgroup, decreasing FAM^TM score was significantly correlated with increasing errors on the SPMSQ (Spearman = 0.27, P < 0.001), whereas the GCS score was not correlated with the SPMSQ (Spearman = 0.05, P = 0.40) (Figure 1).

DISCUSSION

We demonstrated the utility of a rapid and accurate mobile application for assessment of mental status. The FAM^TM was able to be quickly administered with a median time to completion of approximately 1 minute. The ability to detect mild alterations in mental status was shown through concurrent validity by FAM^TM correlation with the SPMSQ and predictive validity with the association between the FAM^TM and discharge disposition. Our study highlights the potential for the FAM^TM to be used as a sensitive marker of AMS.

The novel design of the FAM^TM presents unique advantages compared to current mental status testing. First, the FAM^TM could allow patients with hearing impairment or language barriers to complete a mental status assessment. Additionally, the approximately 1‐minute median time to completion is much faster than other established mental status assessments including the SPMSQ (510 minutes). Compared to the SPMSQ taking 5 minutes, in a 400‐bed hospital, taken once per nursing shift, the FAM^TM would save approximately 20,000 hours and 10 nursing full‐time equivalents per year.[5] Finally, many current mental status tests such as the Confusion Assessment Model utilize subjective mental status assessments.[2] However, the FAM^TM is designed to be conducted through self‐assessment and, thus, could theoretically be free of observer bias. This potential for self‐administration expands beyond other proposed alternative testing mechanisms of the AMS such as ultrabrief assessments that include items such as asking subjects the months of the year backwards, and what is the day of the week?, and assessing arousal.[12, 13, 14]

In research settings and commonly in hospitals, the GCS and AVPU are used clinically for mental status assessment of hospitalized patients.[6, 15] However, similar to previous literature, our study found that the vast majority of hospitalized patients were defined as neurologically intact by the GCS, which is the more accurate predictor of the 2.[7] One major strength of the FAM^TM was that it identified an extensive gradation of scores for patients previously labeled as merely alert, providing greater resolution than the GCS in quantifying mental status.

One of the key benefits of the FAM^TM is that it can be measured longitudinally over the course of a patient's hospital stay. Therefore, once a baseline FAM^TM score is established, variation from the patient's personal baseline could indicate mental status deterioration, which would not be affected by the patient's demographics, health status, or underlying neurocognitive deficits.

There were important limitations to this study. First, limited generalizability of these data may exist due to the single‐center setting and patient population. However, this initial study provides pilot data for further expansion into the potential broad applicability of the FAM^TM to other patient populations and settings. Additionally, the cost of large‐scale implementation of the FAM^TM is unknown and was beyond the scope of this pilot study. However, to reduce costs, the FAM^TM technology could be integrated into existing hospital technology infrastructure. Finally, the scope of this study prevented a complete assessment of all validity measures or comparison to other mental status assessments such as the digit span or serial sevens tests. However, predictive and concurrent validity were assessed with comparison by discharge disposition, SPMSQ, and GCS scores.

In conclusion, this pilot study identifies the FAM^TM application as a potentially clinically useful, novel, rapid, and feasible assessment tool of mental status in a general medicine inpatient setting.

Altered mental status (AMS) is a complex spectrum of cognitive deficits that includes orientation, memory, language, visuospatial ability, and perception.[1] The clinical definitions of both delirium and dementia include AMS as a hallmark clinical prerequisite. Regardless of etiology, this broader AMS definition is particularly salient in the hospital setting, where AMS is present in up to 60% of inpatients and is associated with longer hospital stay as well as increased morbidity and mortality.[2, 3] Not surprisingly, due to the complexity of identifying and assessing changes in mental status, clinically relevant AMS is often undetected among inpatients.[2] However, when detected, the most common causes of AMS (infection, polypharmacy, and pain) are treatable, suggesting that early AMS identification could alert clinicians to early signs of clinical decompensation, potentially improving clinical outcomes.[4]

Because rapid and systemic clinical detection of AMS is limited by the complexity of mental status, a number of assessments have been created, each with their own advantages, limitations, and target populations. These assessments are often limited by time‐intensive administration, subjectivity of mental status assessment, and lack of sensitivity in general medicine patients. Time‐intensive measures, such as the Short Portable Mental Status Questionnaire (SPMSQ) have utility in the research setting, whereas current common clinical risk stratification tools (eg, National Early Warning Score) utilize simpler measures such as the Alert, Voice, Pain, Unresponsive (AVPU) and Glasgow Coma Scale (GCS) as measures of mental status.[2, 5, 6, 7, 8, 9]

To address the need for a brief, clinically feasible, accurate tool in clinical detection of AMS, our group developed a mobile application for working memory testing, the Functional Assessment of Mentation (FAM^TM). In this study, we aimed to identify baseline scoring distributions of the FAM^TM in a nonhospitalized subgroup, as well as assess the correlation of the FAM^TM to discharge disposition and compare it to the SPMSQ in inpatients.

METHODS

FAM^TM Application

The FAM^TM application is a bedside tool for working memory assessment developed for the iPhone mobile operating system (Apple Inc., Cupertino, CA) and presented on an iPad mini (Apple). The application interface displays 4 colored rectangles individually labeled with a number (see Supporting Figure 1 in the online version of this article). The testing portion of the application presents a sequence of numbered rectangles, illuminated 1 at a time in random order. Subjects are prompted first to watch and remember the sequence and then repeat the sequence by touching the screen within each numbered rectangle. Successful reproduction of the sequence is followed by a distinct and longer sequence, whereas unsuccessful attempts are followed by a shorter sequence. The final FAM^TM score corresponds to the longest sequence of rectangles successfully repeated by the subject.

Figure 1

RESULTS

A total of 931 subjects were enrolled in the study. In the nonhospitalized subgroup, 651 consented to study participation and 612 were included in final analysis. Subjects were excluded if they started but did not complete the application (n = 36) or were under the age of 18 years (n = 3). Of the 363 hospitalized subjects approached for enrollment, 319 were included in the final analysis. Subjects were excluded if they refused to participate (n = 23), were under the age of 18 (n = 2), had technical failures (n = 5), or had physical or visual limitations that precluded them from participation (n = 14). Within the hospitalized subgroup, 268 subjects were discharged home (85%). The table displays demographics and score distributions by subgroup.1

The median FAM^TM score for the combined study population was 5 (interquartile range [IQR] 36), and median time to completion was 55 seconds (IQR 4567 seconds). A graded reduction was found in the FAM^TM score for all stepwise comparisons between nonhospitalized subjects, hospitalized subjects discharged home, and hospitalized subjects not discharged home (median 5 [IQR 47] vs 5 [IQR 36] vs 3 [IQR 15]; P < 0.001 for all pairwise comparisons). The AUC for the FAM^TM predicting discharge disposition (home vs not) was 0.66 (95% confidence interval [CI]: 0.58‐0.74]. After adjusting for confounders, higher FAM^TM scores were independently associated with not being hospitalized, being discharged home, higher levels of education, younger age, and white race (see Supporting Table 1 in the online version of this article). Additionally, in the hospitalized subgroup, decreasing FAM^TM score was significantly correlated with increasing errors on the SPMSQ (Spearman = 0.27, P < 0.001), whereas the GCS score was not correlated with the SPMSQ (Spearman = 0.05, P = 0.40) (Figure 1).

DISCUSSION

We demonstrated the utility of a rapid and accurate mobile application for assessment of mental status. The FAM^TM was able to be quickly administered with a median time to completion of approximately 1 minute. The ability to detect mild alterations in mental status was shown through concurrent validity by FAM^TM correlation with the SPMSQ and predictive validity with the association between the FAM^TM and discharge disposition. Our study highlights the potential for the FAM^TM to be used as a sensitive marker of AMS.

The novel design of the FAM^TM presents unique advantages compared to current mental status testing. First, the FAM^TM could allow patients with hearing impairment or language barriers to complete a mental status assessment. Additionally, the approximately 1‐minute median time to completion is much faster than other established mental status assessments including the SPMSQ (510 minutes). Compared to the SPMSQ taking 5 minutes, in a 400‐bed hospital, taken once per nursing shift, the FAM^TM would save approximately 20,000 hours and 10 nursing full‐time equivalents per year.[5] Finally, many current mental status tests such as the Confusion Assessment Model utilize subjective mental status assessments.[2] However, the FAM^TM is designed to be conducted through self‐assessment and, thus, could theoretically be free of observer bias. This potential for self‐administration expands beyond other proposed alternative testing mechanisms of the AMS such as ultrabrief assessments that include items such as asking subjects the months of the year backwards, and what is the day of the week?, and assessing arousal.[12, 13, 14]

In research settings and commonly in hospitals, the GCS and AVPU are used clinically for mental status assessment of hospitalized patients.[6, 15] However, similar to previous literature, our study found that the vast majority of hospitalized patients were defined as neurologically intact by the GCS, which is the more accurate predictor of the 2.[7] One major strength of the FAM^TM was that it identified an extensive gradation of scores for patients previously labeled as merely alert, providing greater resolution than the GCS in quantifying mental status.

One of the key benefits of the FAM^TM is that it can be measured longitudinally over the course of a patient's hospital stay. Therefore, once a baseline FAM^TM score is established, variation from the patient's personal baseline could indicate mental status deterioration, which would not be affected by the patient's demographics, health status, or underlying neurocognitive deficits.

There were important limitations to this study. First, limited generalizability of these data may exist due to the single‐center setting and patient population. However, this initial study provides pilot data for further expansion into the potential broad applicability of the FAM^TM to other patient populations and settings. Additionally, the cost of large‐scale implementation of the FAM^TM is unknown and was beyond the scope of this pilot study. However, to reduce costs, the FAM^TM technology could be integrated into existing hospital technology infrastructure. Finally, the scope of this study prevented a complete assessment of all validity measures or comparison to other mental status assessments such as the digit span or serial sevens tests. However, predictive and concurrent validity were assessed with comparison by discharge disposition, SPMSQ, and GCS scores.

In conclusion, this pilot study identifies the FAM^TM application as a potentially clinically useful, novel, rapid, and feasible assessment tool of mental status in a general medicine inpatient setting.

Altered mental status (AMS), characterized by abnormal changes in a patient's arousal and/or cognition, is a significant predictor of hospital mortality.[1, 2, 3] Yet despite its prevalence[3, 4, 5] and importance, up to three‐quarters of AMS events go unrecognized by caregivers.[6, 7, 8] Acute changes in mental status, often caused by delirium in the hospitalized patient,[3] can present nonspecifically, making it difficult to detect and distinguish from other diagnoses such as depression or dementia.[7, 9] Further complicating the recognition of AMS, numerous and imprecise qualitative descriptors such as confused and alert and oriented are used in clinical practice to describe the mental status of patients.[10] Thus, more objective measures may result in improved detection of altered mental status and in earlier diagnostic and therapeutic interventions.

In critically ill patients, several scales have been widely adopted for quantifying mental status. The Richmond Agitation and Sedation Scale (RASS) was created to optimize sedation.[11] The Glasgow Coma Scale (GCS) was developed for head‐trauma patients[12] and is now a standardized assessment tool in intensive care units,[13] the emergency department,[14] and the prehospital setting.[15] In addition, a simplified scale, AVPU (Alert, responsive to Verbal stimuli, responsive to Painful stimuli, and Unresponsive) was initially used in the primary survey of trauma patients[16] but is now a common component of early‐warning scores and rapid response activation criteria, such as the Modified Early Warning Score (MEWS).[17, 18] In fact, in a systematic review of 72 distinct early‐warning scores, 89% of the scores used AVPU as the measure of mentation.[17] However, the utility of these 3 scales is not well established in the general‐ward setting. Our aim was therefore to compare the accuracies of AVPU, GCS, and RASS for predicting mortality in hospitalized general‐ward patients to provide insight into the accuracy of these different scores for clinical deterioration.

METHODS

RESULTS

During the study period, 313,577 complete GCS and 305,177 RASS scores were recorded in the electronic health record by nursing staff. A total of 26,806 (17,603 GCS and 9203 RASS) observations were excluded due to nonsimultaneous measurement of the other score, resulting in 295,974 paired mental‐status observations. These observations were obtained from 26,873 admissions in 17,660 unique patients, with a median MEWS at ward admission of 1 (IQR 11). The mean patient age was 57 years (SD 17), and 23% were surgical patients (Table 1). Patients spent a median 63.9 hours (IQR 26.7118.6) on the wards per admission and contributed a median of 3 paired observations (IQR 24) per day, with 91% of patients having at least 2 observations per day. A total of 417 (1.6%) general‐ward admissions resulted in death during the hospitalization, with 354 mental‐status observations occurring within 24 hours of a death. In addition, 26,618 (99.9%) admissions had at least 1 paired mental‐status observation within the last 24 hours of their ward stay.

AVPU was moderately correlated with GCS (Spearman's =0.56) (Figure 1a) and weakly correlated with RASS (Spearman's =0.28) (Figure 1b). GCS scores were also weakly correlated to RASS (Spearman's =0.13, P<0.001). Notably, AVPU mapped to distinct levels of GCS, with Alert associated with a median GCS total score of 15, Voice a score of 12, Pain a score of 8, and Unresponsive a score of 5. Abnormal mental‐status scores on any scale were associated with significantly higher odds of death within 24 hours than normal mental‐status scores (Table 2). This association was consistent within the 3 subscales of GCS and for scores in both the sedation (<0) and agitation (>0) ranges of RASS.

Figure 1

AVPU was the least accurate predictor of mortality (AUC 0.73 [95% confidence interval {CI}: 0.710.76]), whereas simultaneous use of GCS and RASS was the most accurate predictor (AUC 0.85 [95% CI: 0.820.87] (Figure 2). The accuracies of GCS and RASS were not significantly different from one another in the total study population (AUC 0.80 [95% CI: 0.770.83] and 0.82 [0.790.84], respectively, P=0.13). Ten‐fold cross‐validation to estimate the internal validity of the RASS model resulted in a lower AUC (0.78 [95% CI: 0.750.81]) for RASS as a predictor of 24‐hour mortality. Subgroup analysis indicated that RASS was more accurate than GCS in younger patients (<57 years old) and in surgical patients (Figure 3).

Figure 2

Figure 3

Removal of the 255 admissions missing a paired mental‐status observation within the last 24 hours of their ward stay resulted in no change in the AUC values. A sensitivity analysis for prediction of a combined secondary outcome of 24‐hour intensive care unit ICU transfer or cardiac arrest yielded lower AUCs for each mental‐status scale, with no change in the association among scales.

DISCUSSION

To our knowledge, this study is the first to compare the accuracies of AVPU, GCS, and RASS for predicting mortality in the general‐ward setting. Similar to McNarry and Goldhill, we demonstrated that AVPU scores mapped to distinct levels of GCS. Although our study reports the same median GCS scores of 15 and 8 for AVPU levels of Alert and Pain, respectively, we indicate slightly lower corresponding median GCS scores for AVPU scores of Voice (12 vs 13) and Unresponsive (5 vs 6) than their previous work.[21] We found that AVPU was the least accurate predictor of mortality within 24 hours of an observation, and the combination of GCS and RASS was the most accurate. RASS was at least as accurate a predictor for 24‐hour mortality in comparison to GCS total in the overall study population. However, the RASS score was the most accurate individual score in surgical and younger patients. These findings suggest that changing from the commonly used AVPU scale to the RASS and/or GCS would improve the prognostic ability of mental‐status assessments on the general wards.

Buist and colleagues have previously demonstrated altered mental status to be one of the strongest predictors of death on the wards. In that study, a GCS score of 3 and a decrease in GCS score by more than 2 points were independently associated with mortality (odds ratio 6.1 [95% CI: 3.111.8] and 5.5 [95% CI: 2.611.9], respectively).[22] We have also previously shown that after adjusting for vital signs, being unresponsive to pain was associated with a 4.5‐fold increase in the odds of death within 24 hours,[23]whereas Subbe and colleagues showed a relative risk ratio of 5.2 (95% CI: 1.518.1) for the combined endpoint of cardiac arrest, death at 60 days, or admission to the intensive care/high dependency unit.[19] In the current study, the magnitude of these associations was even stronger, with a GCS score <13 correlating with a 55‐fold increase in the odds of death, compared to a normal GCS, and not being alert being associated with a 33.8‐fold increase in the odds of death. This difference in magnitude is likely a product of the univariate nature of the current analysis, compared to both the Buist et al. and Churpek et al. studies, which adjusted for vital signs, thereby lessening the impact of any single predictor. Because this study was designed to compare mental‐status variables to one another for future model inclusion, and all the analyses were paired, confounding by additional predictors of death was not a concern.

One of the potential strengths of RASS over GCS and AVPU is its ability to measure agitation levels, in addition to depressed mentation, a feature that has been shown to be present in up to 60% of delirium episodes.[24] This may also explain why RASS was the most accurate predictor of mortality in our subset of younger patients and surgical patients, because hyperactive delirium is more common in younger and healthier patients, which surgical patients tend to be as compared to medical patients.[25, 26] In this study, we found negative RASS scores portending a worse prognosis than positive ones, which supports previous findings that hypoactive delirium had a higher association with mortality than hyperactive delirium at 6 months (hazard ratio 1.90 vs 1.37) and at 1 year (hazard ratio 1.60 vs 1.30) in elderly patients at postacute‐care facilities in 2 separate studies.[27, 28] However, a study of patients undergoing surgery for hip fracture found that patients with hyperactive delirium were more likely to die or be placed in a nursing home at 1 month follow‐up when compared to patients with purely hypoactive delirium (79% vs 32%, P=0.003).[29]

We found the assessment of RASS and GCS by ward nurses to be highly feasible. During the study period, nurses assessed mental status with the GCS and RASS scales at least once per 12‐hour shift in 91% of patients. GCS has been shown to be reliably and accurately recorded by experienced nurses (reliability coefficient=0.944 with 96.4% agreement with expert ratings).[30] RASS can take <30 seconds to administer, and in previous studies of the ICU setting has been shown to have over 94% nurse compliance for administration,[31] and good inter‐rater reliability (weighted kappa 0.66 and 0.89, respectively).[31, 32] Further, in a prior survey of 55 critical care nurses, 82% agreed that RASS was easy to score and clinically relevant.[31]

This study has several limitations. First, it was conducted in a single academic institution, which may limit generalizability to other hospitals. Second, baseline cognition and comorbidities were not available in the dataset, so we were unable to conduct additional subgroup analyses by these categories. However, we used age and hospital admission type as proxies. Third, the AVPU scores in this study were extracted from the Eye subset of the GCS scale, as AVPU was not directly assessed on our wards during the study period. Clinical assessment of mental status on the AVPU scale notes the presence of any active patient response (eg, eye opening, grunting, moaning, movement) to increasingly noxious stimuli. As such, our adaptation of AVPU using only eye‐opening criteria may underestimate the true number of patients correctly classified as alert, or responding to vocal/painful stimuli. However, a sensitivity analysis comparing directly assessed AVPU during a 3‐year period prior to the study implementation at our institution, and AVPU derived from the GCS Eye subscale for the study period, indicated no difference in predictive value for 24‐hour mortality. Fourth, we did not perform trend analyses for change from baseline mental status or evolution of AMS, which may more accurately predict 24‐hour mortality than discrete mental‐status observations. Finally, the 3 scales we compared differ in length, which may bias the AUC against AVPU, a 4‐point scale with a trapezoidal ROC curve compared to the smoother curve generated by the 15‐point GCS scale, for example. However, the lack of discrimination of the AVPU is the likely source of its lesser accuracy.

CONCLUSION

In the general‐ward setting, routine collection of GCS and RASS is feasible, and both are significantly more accurate for predicting mortality than the more commonly used AVPU scale. In addition, the combination of GCS and RASS has greater accuracy than any of the 3 individual scales. RASS may be particularly beneficial in the assessment of younger and/or surgical patients. Routine documentation and tracking of GCS and/or RASS by nurses may improve the detection of clinical deterioration in general‐ward patients. In addition, future early‐warning scores may benefit from the inclusion of GCS and/or RASS in lieu of AVPU.

Altered mental status (AMS), characterized by abnormal changes in a patient's arousal and/or cognition, is a significant predictor of hospital mortality.[1, 2, 3] Yet despite its prevalence[3, 4, 5] and importance, up to three‐quarters of AMS events go unrecognized by caregivers.[6, 7, 8] Acute changes in mental status, often caused by delirium in the hospitalized patient,[3] can present nonspecifically, making it difficult to detect and distinguish from other diagnoses such as depression or dementia.[7, 9] Further complicating the recognition of AMS, numerous and imprecise qualitative descriptors such as confused and alert and oriented are used in clinical practice to describe the mental status of patients.[10] Thus, more objective measures may result in improved detection of altered mental status and in earlier diagnostic and therapeutic interventions.

In critically ill patients, several scales have been widely adopted for quantifying mental status. The Richmond Agitation and Sedation Scale (RASS) was created to optimize sedation.[11] The Glasgow Coma Scale (GCS) was developed for head‐trauma patients[12] and is now a standardized assessment tool in intensive care units,[13] the emergency department,[14] and the prehospital setting.[15] In addition, a simplified scale, AVPU (Alert, responsive to Verbal stimuli, responsive to Painful stimuli, and Unresponsive) was initially used in the primary survey of trauma patients[16] but is now a common component of early‐warning scores and rapid response activation criteria, such as the Modified Early Warning Score (MEWS).[17, 18] In fact, in a systematic review of 72 distinct early‐warning scores, 89% of the scores used AVPU as the measure of mentation.[17] However, the utility of these 3 scales is not well established in the general‐ward setting. Our aim was therefore to compare the accuracies of AVPU, GCS, and RASS for predicting mortality in hospitalized general‐ward patients to provide insight into the accuracy of these different scores for clinical deterioration.

METHODS

RESULTS

During the study period, 313,577 complete GCS and 305,177 RASS scores were recorded in the electronic health record by nursing staff. A total of 26,806 (17,603 GCS and 9203 RASS) observations were excluded due to nonsimultaneous measurement of the other score, resulting in 295,974 paired mental‐status observations. These observations were obtained from 26,873 admissions in 17,660 unique patients, with a median MEWS at ward admission of 1 (IQR 11). The mean patient age was 57 years (SD 17), and 23% were surgical patients (Table 1). Patients spent a median 63.9 hours (IQR 26.7118.6) on the wards per admission and contributed a median of 3 paired observations (IQR 24) per day, with 91% of patients having at least 2 observations per day. A total of 417 (1.6%) general‐ward admissions resulted in death during the hospitalization, with 354 mental‐status observations occurring within 24 hours of a death. In addition, 26,618 (99.9%) admissions had at least 1 paired mental‐status observation within the last 24 hours of their ward stay.

AVPU was moderately correlated with GCS (Spearman's =0.56) (Figure 1a) and weakly correlated with RASS (Spearman's =0.28) (Figure 1b). GCS scores were also weakly correlated to RASS (Spearman's =0.13, P<0.001). Notably, AVPU mapped to distinct levels of GCS, with Alert associated with a median GCS total score of 15, Voice a score of 12, Pain a score of 8, and Unresponsive a score of 5. Abnormal mental‐status scores on any scale were associated with significantly higher odds of death within 24 hours than normal mental‐status scores (Table 2). This association was consistent within the 3 subscales of GCS and for scores in both the sedation (<0) and agitation (>0) ranges of RASS.

Figure 1

AVPU was the least accurate predictor of mortality (AUC 0.73 [95% confidence interval {CI}: 0.710.76]), whereas simultaneous use of GCS and RASS was the most accurate predictor (AUC 0.85 [95% CI: 0.820.87] (Figure 2). The accuracies of GCS and RASS were not significantly different from one another in the total study population (AUC 0.80 [95% CI: 0.770.83] and 0.82 [0.790.84], respectively, P=0.13). Ten‐fold cross‐validation to estimate the internal validity of the RASS model resulted in a lower AUC (0.78 [95% CI: 0.750.81]) for RASS as a predictor of 24‐hour mortality. Subgroup analysis indicated that RASS was more accurate than GCS in younger patients (<57 years old) and in surgical patients (Figure 3).

Figure 2

Figure 3

Removal of the 255 admissions missing a paired mental‐status observation within the last 24 hours of their ward stay resulted in no change in the AUC values. A sensitivity analysis for prediction of a combined secondary outcome of 24‐hour intensive care unit ICU transfer or cardiac arrest yielded lower AUCs for each mental‐status scale, with no change in the association among scales.

DISCUSSION

To our knowledge, this study is the first to compare the accuracies of AVPU, GCS, and RASS for predicting mortality in the general‐ward setting. Similar to McNarry and Goldhill, we demonstrated that AVPU scores mapped to distinct levels of GCS. Although our study reports the same median GCS scores of 15 and 8 for AVPU levels of Alert and Pain, respectively, we indicate slightly lower corresponding median GCS scores for AVPU scores of Voice (12 vs 13) and Unresponsive (5 vs 6) than their previous work.[21] We found that AVPU was the least accurate predictor of mortality within 24 hours of an observation, and the combination of GCS and RASS was the most accurate. RASS was at least as accurate a predictor for 24‐hour mortality in comparison to GCS total in the overall study population. However, the RASS score was the most accurate individual score in surgical and younger patients. These findings suggest that changing from the commonly used AVPU scale to the RASS and/or GCS would improve the prognostic ability of mental‐status assessments on the general wards.

Buist and colleagues have previously demonstrated altered mental status to be one of the strongest predictors of death on the wards. In that study, a GCS score of 3 and a decrease in GCS score by more than 2 points were independently associated with mortality (odds ratio 6.1 [95% CI: 3.111.8] and 5.5 [95% CI: 2.611.9], respectively).[22] We have also previously shown that after adjusting for vital signs, being unresponsive to pain was associated with a 4.5‐fold increase in the odds of death within 24 hours,[23]whereas Subbe and colleagues showed a relative risk ratio of 5.2 (95% CI: 1.518.1) for the combined endpoint of cardiac arrest, death at 60 days, or admission to the intensive care/high dependency unit.[19] In the current study, the magnitude of these associations was even stronger, with a GCS score <13 correlating with a 55‐fold increase in the odds of death, compared to a normal GCS, and not being alert being associated with a 33.8‐fold increase in the odds of death. This difference in magnitude is likely a product of the univariate nature of the current analysis, compared to both the Buist et al. and Churpek et al. studies, which adjusted for vital signs, thereby lessening the impact of any single predictor. Because this study was designed to compare mental‐status variables to one another for future model inclusion, and all the analyses were paired, confounding by additional predictors of death was not a concern.

One of the potential strengths of RASS over GCS and AVPU is its ability to measure agitation levels, in addition to depressed mentation, a feature that has been shown to be present in up to 60% of delirium episodes.[24] This may also explain why RASS was the most accurate predictor of mortality in our subset of younger patients and surgical patients, because hyperactive delirium is more common in younger and healthier patients, which surgical patients tend to be as compared to medical patients.[25, 26] In this study, we found negative RASS scores portending a worse prognosis than positive ones, which supports previous findings that hypoactive delirium had a higher association with mortality than hyperactive delirium at 6 months (hazard ratio 1.90 vs 1.37) and at 1 year (hazard ratio 1.60 vs 1.30) in elderly patients at postacute‐care facilities in 2 separate studies.[27, 28] However, a study of patients undergoing surgery for hip fracture found that patients with hyperactive delirium were more likely to die or be placed in a nursing home at 1 month follow‐up when compared to patients with purely hypoactive delirium (79% vs 32%, P=0.003).[29]

We found the assessment of RASS and GCS by ward nurses to be highly feasible. During the study period, nurses assessed mental status with the GCS and RASS scales at least once per 12‐hour shift in 91% of patients. GCS has been shown to be reliably and accurately recorded by experienced nurses (reliability coefficient=0.944 with 96.4% agreement with expert ratings).[30] RASS can take <30 seconds to administer, and in previous studies of the ICU setting has been shown to have over 94% nurse compliance for administration,[31] and good inter‐rater reliability (weighted kappa 0.66 and 0.89, respectively).[31, 32] Further, in a prior survey of 55 critical care nurses, 82% agreed that RASS was easy to score and clinically relevant.[31]

This study has several limitations. First, it was conducted in a single academic institution, which may limit generalizability to other hospitals. Second, baseline cognition and comorbidities were not available in the dataset, so we were unable to conduct additional subgroup analyses by these categories. However, we used age and hospital admission type as proxies. Third, the AVPU scores in this study were extracted from the Eye subset of the GCS scale, as AVPU was not directly assessed on our wards during the study period. Clinical assessment of mental status on the AVPU scale notes the presence of any active patient response (eg, eye opening, grunting, moaning, movement) to increasingly noxious stimuli. As such, our adaptation of AVPU using only eye‐opening criteria may underestimate the true number of patients correctly classified as alert, or responding to vocal/painful stimuli. However, a sensitivity analysis comparing directly assessed AVPU during a 3‐year period prior to the study implementation at our institution, and AVPU derived from the GCS Eye subscale for the study period, indicated no difference in predictive value for 24‐hour mortality. Fourth, we did not perform trend analyses for change from baseline mental status or evolution of AMS, which may more accurately predict 24‐hour mortality than discrete mental‐status observations. Finally, the 3 scales we compared differ in length, which may bias the AUC against AVPU, a 4‐point scale with a trapezoidal ROC curve compared to the smoother curve generated by the 15‐point GCS scale, for example. However, the lack of discrimination of the AVPU is the likely source of its lesser accuracy.

CONCLUSION

In the general‐ward setting, routine collection of GCS and RASS is feasible, and both are significantly more accurate for predicting mortality than the more commonly used AVPU scale. In addition, the combination of GCS and RASS has greater accuracy than any of the 3 individual scales. RASS may be particularly beneficial in the assessment of younger and/or surgical patients. Routine documentation and tracking of GCS and/or RASS by nurses may improve the detection of clinical deterioration in general‐ward patients. In addition, future early‐warning scores may benefit from the inclusion of GCS and/or RASS in lieu of AVPU.

Altered mental status (AMS), characterized by abnormal changes in a patient's arousal and/or cognition, is a significant predictor of hospital mortality.[1, 2, 3] Yet despite its prevalence[3, 4, 5] and importance, up to three‐quarters of AMS events go unrecognized by caregivers.[6, 7, 8] Acute changes in mental status, often caused by delirium in the hospitalized patient,[3] can present nonspecifically, making it difficult to detect and distinguish from other diagnoses such as depression or dementia.[7, 9] Further complicating the recognition of AMS, numerous and imprecise qualitative descriptors such as confused and alert and oriented are used in clinical practice to describe the mental status of patients.[10] Thus, more objective measures may result in improved detection of altered mental status and in earlier diagnostic and therapeutic interventions.

In critically ill patients, several scales have been widely adopted for quantifying mental status. The Richmond Agitation and Sedation Scale (RASS) was created to optimize sedation.[11] The Glasgow Coma Scale (GCS) was developed for head‐trauma patients[12] and is now a standardized assessment tool in intensive care units,[13] the emergency department,[14] and the prehospital setting.[15] In addition, a simplified scale, AVPU (Alert, responsive to Verbal stimuli, responsive to Painful stimuli, and Unresponsive) was initially used in the primary survey of trauma patients[16] but is now a common component of early‐warning scores and rapid response activation criteria, such as the Modified Early Warning Score (MEWS).[17, 18] In fact, in a systematic review of 72 distinct early‐warning scores, 89% of the scores used AVPU as the measure of mentation.[17] However, the utility of these 3 scales is not well established in the general‐ward setting. Our aim was therefore to compare the accuracies of AVPU, GCS, and RASS for predicting mortality in hospitalized general‐ward patients to provide insight into the accuracy of these different scores for clinical deterioration.

METHODS

RESULTS

During the study period, 313,577 complete GCS and 305,177 RASS scores were recorded in the electronic health record by nursing staff. A total of 26,806 (17,603 GCS and 9203 RASS) observations were excluded due to nonsimultaneous measurement of the other score, resulting in 295,974 paired mental‐status observations. These observations were obtained from 26,873 admissions in 17,660 unique patients, with a median MEWS at ward admission of 1 (IQR 11). The mean patient age was 57 years (SD 17), and 23% were surgical patients (Table 1). Patients spent a median 63.9 hours (IQR 26.7118.6) on the wards per admission and contributed a median of 3 paired observations (IQR 24) per day, with 91% of patients having at least 2 observations per day. A total of 417 (1.6%) general‐ward admissions resulted in death during the hospitalization, with 354 mental‐status observations occurring within 24 hours of a death. In addition, 26,618 (99.9%) admissions had at least 1 paired mental‐status observation within the last 24 hours of their ward stay.

AVPU was moderately correlated with GCS (Spearman's =0.56) (Figure 1a) and weakly correlated with RASS (Spearman's =0.28) (Figure 1b). GCS scores were also weakly correlated to RASS (Spearman's =0.13, P<0.001). Notably, AVPU mapped to distinct levels of GCS, with Alert associated with a median GCS total score of 15, Voice a score of 12, Pain a score of 8, and Unresponsive a score of 5. Abnormal mental‐status scores on any scale were associated with significantly higher odds of death within 24 hours than normal mental‐status scores (Table 2). This association was consistent within the 3 subscales of GCS and for scores in both the sedation (<0) and agitation (>0) ranges of RASS.

Figure 1

AVPU was the least accurate predictor of mortality (AUC 0.73 [95% confidence interval {CI}: 0.710.76]), whereas simultaneous use of GCS and RASS was the most accurate predictor (AUC 0.85 [95% CI: 0.820.87] (Figure 2). The accuracies of GCS and RASS were not significantly different from one another in the total study population (AUC 0.80 [95% CI: 0.770.83] and 0.82 [0.790.84], respectively, P=0.13). Ten‐fold cross‐validation to estimate the internal validity of the RASS model resulted in a lower AUC (0.78 [95% CI: 0.750.81]) for RASS as a predictor of 24‐hour mortality. Subgroup analysis indicated that RASS was more accurate than GCS in younger patients (<57 years old) and in surgical patients (Figure 3).

Figure 2

Figure 3

Removal of the 255 admissions missing a paired mental‐status observation within the last 24 hours of their ward stay resulted in no change in the AUC values. A sensitivity analysis for prediction of a combined secondary outcome of 24‐hour intensive care unit ICU transfer or cardiac arrest yielded lower AUCs for each mental‐status scale, with no change in the association among scales.

DISCUSSION

To our knowledge, this study is the first to compare the accuracies of AVPU, GCS, and RASS for predicting mortality in the general‐ward setting. Similar to McNarry and Goldhill, we demonstrated that AVPU scores mapped to distinct levels of GCS. Although our study reports the same median GCS scores of 15 and 8 for AVPU levels of Alert and Pain, respectively, we indicate slightly lower corresponding median GCS scores for AVPU scores of Voice (12 vs 13) and Unresponsive (5 vs 6) than their previous work.[21] We found that AVPU was the least accurate predictor of mortality within 24 hours of an observation, and the combination of GCS and RASS was the most accurate. RASS was at least as accurate a predictor for 24‐hour mortality in comparison to GCS total in the overall study population. However, the RASS score was the most accurate individual score in surgical and younger patients. These findings suggest that changing from the commonly used AVPU scale to the RASS and/or GCS would improve the prognostic ability of mental‐status assessments on the general wards.

Buist and colleagues have previously demonstrated altered mental status to be one of the strongest predictors of death on the wards. In that study, a GCS score of 3 and a decrease in GCS score by more than 2 points were independently associated with mortality (odds ratio 6.1 [95% CI: 3.111.8] and 5.5 [95% CI: 2.611.9], respectively).[22] We have also previously shown that after adjusting for vital signs, being unresponsive to pain was associated with a 4.5‐fold increase in the odds of death within 24 hours,[23]whereas Subbe and colleagues showed a relative risk ratio of 5.2 (95% CI: 1.518.1) for the combined endpoint of cardiac arrest, death at 60 days, or admission to the intensive care/high dependency unit.[19] In the current study, the magnitude of these associations was even stronger, with a GCS score <13 correlating with a 55‐fold increase in the odds of death, compared to a normal GCS, and not being alert being associated with a 33.8‐fold increase in the odds of death. This difference in magnitude is likely a product of the univariate nature of the current analysis, compared to both the Buist et al. and Churpek et al. studies, which adjusted for vital signs, thereby lessening the impact of any single predictor. Because this study was designed to compare mental‐status variables to one another for future model inclusion, and all the analyses were paired, confounding by additional predictors of death was not a concern.

One of the potential strengths of RASS over GCS and AVPU is its ability to measure agitation levels, in addition to depressed mentation, a feature that has been shown to be present in up to 60% of delirium episodes.[24] This may also explain why RASS was the most accurate predictor of mortality in our subset of younger patients and surgical patients, because hyperactive delirium is more common in younger and healthier patients, which surgical patients tend to be as compared to medical patients.[25, 26] In this study, we found negative RASS scores portending a worse prognosis than positive ones, which supports previous findings that hypoactive delirium had a higher association with mortality than hyperactive delirium at 6 months (hazard ratio 1.90 vs 1.37) and at 1 year (hazard ratio 1.60 vs 1.30) in elderly patients at postacute‐care facilities in 2 separate studies.[27, 28] However, a study of patients undergoing surgery for hip fracture found that patients with hyperactive delirium were more likely to die or be placed in a nursing home at 1 month follow‐up when compared to patients with purely hypoactive delirium (79% vs 32%, P=0.003).[29]

We found the assessment of RASS and GCS by ward nurses to be highly feasible. During the study period, nurses assessed mental status with the GCS and RASS scales at least once per 12‐hour shift in 91% of patients. GCS has been shown to be reliably and accurately recorded by experienced nurses (reliability coefficient=0.944 with 96.4% agreement with expert ratings).[30] RASS can take <30 seconds to administer, and in previous studies of the ICU setting has been shown to have over 94% nurse compliance for administration,[31] and good inter‐rater reliability (weighted kappa 0.66 and 0.89, respectively).[31, 32] Further, in a prior survey of 55 critical care nurses, 82% agreed that RASS was easy to score and clinically relevant.[31]

This study has several limitations. First, it was conducted in a single academic institution, which may limit generalizability to other hospitals. Second, baseline cognition and comorbidities were not available in the dataset, so we were unable to conduct additional subgroup analyses by these categories. However, we used age and hospital admission type as proxies. Third, the AVPU scores in this study were extracted from the Eye subset of the GCS scale, as AVPU was not directly assessed on our wards during the study period. Clinical assessment of mental status on the AVPU scale notes the presence of any active patient response (eg, eye opening, grunting, moaning, movement) to increasingly noxious stimuli. As such, our adaptation of AVPU using only eye‐opening criteria may underestimate the true number of patients correctly classified as alert, or responding to vocal/painful stimuli. However, a sensitivity analysis comparing directly assessed AVPU during a 3‐year period prior to the study implementation at our institution, and AVPU derived from the GCS Eye subscale for the study period, indicated no difference in predictive value for 24‐hour mortality. Fourth, we did not perform trend analyses for change from baseline mental status or evolution of AMS, which may more accurately predict 24‐hour mortality than discrete mental‐status observations. Finally, the 3 scales we compared differ in length, which may bias the AUC against AVPU, a 4‐point scale with a trapezoidal ROC curve compared to the smoother curve generated by the 15‐point GCS scale, for example. However, the lack of discrimination of the AVPU is the likely source of its lesser accuracy.

CONCLUSION

In the general‐ward setting, routine collection of GCS and RASS is feasible, and both are significantly more accurate for predicting mortality than the more commonly used AVPU scale. In addition, the combination of GCS and RASS has greater accuracy than any of the 3 individual scales. RASS may be particularly beneficial in the assessment of younger and/or surgical patients. Routine documentation and tracking of GCS and/or RASS by nurses may improve the detection of clinical deterioration in general‐ward patients. In addition, future early‐warning scores may benefit from the inclusion of GCS and/or RASS in lieu of AVPU.