User login
Deterioration Alerts on Medical Wards
Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]
The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.
METHODS
Study Location
The study was conducted at Barnes‐Jewish Hospital, a 1250‐bed academic medical center in St. Louis, Missouri. Eight adult medicine wards were assessed from July 2007 through December 2011. The medicine wards are closed areas with patient care delivered by dedicated house staff physicians under the supervision of a board‐certified attending physician. The study was approved by the Washington University School of Medicine Human Studies Committee.
Study Period
The period from July 2007 through January 2010 was used to train and retrospectively test the prediction model. The period from January 2011 through December 2011 was used to prospectively validate the model during a randomized trial using alerts generated from the prediction model.
Patients
Electronically captured clinical data were housed in a centralized clinical data repository. This repository cataloged 28,927 hospital visits from 19,116 distinct patients between July 2007 and January 2010. It contained a rich set of demographic and medical data for each of the visits, such as patient age, manually collected vital‐sign data, pharmacy data, laboratory data, and intensive care unit (ICU) transfer. This study served as a proof of concept for our vision of using machine learning to identify at‐risk patients and ultimately to perform real‐time event detection and interventions.
Algorithm Overview
Details regarding the predictive model development have been previously described.[11] To predict ICU transfer for patients housed on general medical wards, we used logistic regression, employing a novel framework to analyze the data stream from each patient, assigning scores to reflect the probability of ICU transfer to each patient.
Before building the model, several preprocessing steps were applied to eliminate outliers and find an appropriate representation of patients' states. For each of 36 input variables we specified acceptable ranges based on the domain knowledge of the medical experts on our team. For any value that was outside of the medically conceivable range, we replaced it by the mean value for that patient, if available. Values for every continuous parameter were scaled so that all measurements lay in the interval [0, 1] and were normalized by the minimum and maximum of the parameter. To capture the temporal effects in our data, we retained a sliding window of all the collected data points within the last 24 hours. We then subdivided these data into a series of 6 sequential buckets of 4 hours each.
To capture variations within a bucket, we computed 3 values for each feature in the bucket: the minimum, maximum, and mean data points. Each of the resulting 3n values was input to the logistic regression equation as separate variables. To deal with missing data points within the buckets, we used the patients' most recent reading from any time earlier in the hospital stay, if available. If no prior values existed, we used mean values calculated over the entire historical dataset. Bucket 6 max/min/mean represents the most recent 4‐hour window from the preceding 24‐hour time period for the maximum, minimum, and mean values, respectively. By itself, logistic regression does not operate on time‐series data. That is, each variable input to the logistic equation corresponds to exactly 1 data point (eg, a blood‐pressure variable would consist of a single blood‐pressure reading). In a clinical application, however, it is important to capture unusual changes in vital‐sign data over time. Such changes may precede clinical deterioration by hours, providing a chance to intervene if detected early enough. In addition, not all readings in time‐series data should be treated equally; the value of some kinds of data may change depending on their age. For example, a patient's condition may be better reflected by a blood‐oxygenation reading collected 1 hour ago than a reading collected 12 hours ago. This is the rationale for our use of a sliding window of all collected data points within the last 24 hours performed in a real‐time basis.
The algorithm was first implemented in MATLAB (Natick, MA). For the purposes of training, we used a single 24‐hour window of data from each patient. For patients admitted to ICU, this window was 26 hours to 2 hours prior to ICU admission; for all other patients, this window consisted of the first 24 hours of their hospital stay. The dataset's 36 input variables were divided into buckets and min/mean/max features wherever applicable, resulting in 398 variables. The first half of the dataset was used to train the model. We then used the second half of the dataset as the validation dataset. We generated a predicted outcome for each case in the validation data, using the model parameter coefficients derived from the training data. We also employed bootstrap aggregation to improve classification accuracy and to address overfitting. We then applied various threshold cut‐points to convert these predictions into binary values and compared the results against the ICU transfer outcome. A threshold of 0.9760 for specificity was chosen to achieve a sensitivity of approximately 40%. These operating characteristics were chosen in turn to generate a manageable number of alerts per hospital nursing unit per day (estimated at 12 per nursing unit per day). At this cut‐point the C‐statistic was 0.8834, with an overall accuracy of 0.9292.
In order to train the logistic model, we used a single 24‐hour window of data for each patient. However, in a system that predicts patients' outcomes in real time, scores are recomputed each time new data are entered into the database. Hence, patients have a series of scores over the length of their hospital stay, and an alert is triggered when any one of these scores is above the chosen threshold.
Once the model was developed, we implemented it in an internally developed, Java‐based clinical decision support rules engine, which identified when new data relevant to the model were available in a real‐time central data repository. The rules engine queried the data repository to acquire all data needed to evaluate the model. The score was calculated with each relevant new data point, and an alert was generated when the score exceeded the cut‐point threshold. We then prospectively validated these alerts on patients on 8 general medical wards at Barnes Jewish Hospital. Details regarding the architecture of our clinical decision support system have been previously published.[12] The sensitivity and positive predictive values for ICU transfer for these alerts were tracked during an intervention trial that ran from January 24, 2011, through December 31, 2011. Four general medical wards were randomized to the intervention group and 4 wards were randomized to the control group. The 8 general medical wards were ordered according to their alert rates based upon the historical data from July 2007 through January 2010, creating 4 pairs of wards in ascending order of alert rate. Within each of the 4 pairs, 1 member of the pair was randomized to the intervention group and the other to the control group using a random number generator.
Real‐time automated alerts generated 24 hours per day, 7 days per week from the predictive algorithm were sent to the charge‐nurse pager on the intervention units. Alerts were also generated and stored in the database on the control units, but these alerts were not sent to the charge nurse on those units. The alerts were sent to the charge nurses on the respective wards, as these individuals were thought to be in the best position to perform the initial assessment of the alerted patients, especially during evening hours when physician staffing was reduced. The charge nurses assessed the intervention‐group patients and were instructed to contact the responsible physician (hospitalist or internal medicine house officer) to inform them of the alert, or to call the rapid response team (RRT) if the patient's condition already appeared to be significantly deteriorating.
Descriptive statistics for algorithm sensitivity and positive predictive value and for patient outcomes were performed. Associations between alerts and the primary outcome, ICU transfer, were determined, as well as the impact of alerts in the intervention group compared with the control group, using [2] tests. The same analyses were performed for patient death. Differences in length of stay (LOS) were assessed using the Wilcoxon rank sum test.
RESULTS
Predictive Model
The variables with the greatest coefficients contributing to the PT model included respiratory rate, oxygen saturation, shock index, systolic blood pressure, anticoagulation use, heart rate, and diastolic blood pressure. A complete list of variables is provided in the Appendix (see Supporting Information in the online version of this article). All but 1 are routinely collected vital‐sign measures, and all but 1 occur in the 4‐hour period immediately prior to the alert (bucket 6).
Prospective Trial
Patient characteristics are presented in Table 1. Patients were well matched for race, sex, age, and underlying diagnoses. All alerts reported to the charge nurses were to be associated with a call from the charge nurse to the responsible physician caring for the alerted patient. The mean number of alerts per alerted patient was 1.8 (standard deviation=1.7). Patients meeting the alert threshold were at nearly 5.3‐fold greater risk of ICU transfer (95% confidence interval [CI]: 4.6‐6.0) than those not satisfying the alert threshold (358 of 2353 [15.2%; 95% CI: 13.8%‐16.7%] vs 512 of 17678 [2.9%; 95% CI: 2.7%‐3.2%], respectively; P<0.0001). Patients with alerts were at 8.9‐fold greater risk of death (95% CI: 7.4‐10.7) than those without alerts (244 of 2353 [10.4%; 95% CI: 9.2%‐11.7%] vs 206 of 17678 [1.2%; 95% CI: 1.0%‐1.3%], respectively; P<0.0001). Operating characteristics of the PT from the prospective trial are shown in Table 2. Alerts occurred a median of 25.5 hours prior to ICU transfer (interquartile range, 7.00‐81.75) and 8 hours prior to death (interquartile range, 4.09‐15.66).
| Study Group | ||||||
|---|---|---|---|---|---|---|
| Control (N=10,120) | Intervention (N=9911) | |||||
| ||||||
| Race | N | % | N | % | ||
| White | 5,062 | 50 | 4,934 | 50 | ||
| Black | 4,864 | 48 | 4,790 | 48 | ||
| Other | 194 | 2 | 187 | 2 | ||
| Sex | ||||||
| F | 5,355 | 53 | 5,308 | 54 | ||
| M | 4,765 | 47 | 4,603 | 46 | ||
| Age at discharge, median (IQR), y | 57 (4469) | 57 (4470) | ||||
| Top 10 ICD‐9 descriptions and counts, n (%) | ||||||
| 1 | Diseases of the digestive system | 1,774 (17.5) | Diseases of the digestive system | 1,664 (16.7) | ||
| 2 | Diseases of the circulatory system | 1,252 (12.4) | Diseases of the circulatory system | 1,253 (12.6) | ||
| 3 | Diseases of the respiratory system | 1,236 (12.2) | Diseases of the respiratory system | 1,210 (12.2) | ||
| 4 | Injury and poisoning | 864 (8.5) | Injury and poisoning | 849 (8.6) | ||
| 5 | Endocrine, nutritional, and metabolic diseases, and immunity disorders | 797 (7.9) | Diseases of the genitourinary system | 795 (8.0) | ||
| 6 | Diseases of the genitourinary system | 762 (7.5) | Endocrine, nutritional, and metabolic diseases, and immunity disorders | 780 (7.9) | ||
| 7 | Infectious and parasitic diseases | 555 (5.5) | Infectious and parasitic diseases | 549 (5.5) | ||
| 8 | Neoplasms | 547 (5.4) | Neoplasms | 465 (4.7) | ||
| 9 | Diseases of the blood and blood‐forming organs | 426 (4.2) | Diseases of the blood and blood‐forming organs | 429 (4.3) | ||
| 10 | Symptoms, signs, and ill‐defined conditions and factors influencing health status | 410 (4.1) | Diseases of the musculoskeletal system and connective tissue | 399 (4.0) | ||
| Sensitivity, % | Specificity, % | PPV, % | NPV, % | Positive Likelihood Ratio | Negative Likelihood Ratio | |||
|---|---|---|---|---|---|---|---|---|
| ||||||||
| ICU Transfer | Yes (N=870) | No (N=19,161) | ||||||
| Alert | 358 | 1,995 | 41.1 (95% CI: 37.944.5) | 89.6 (95% CI: 89.290.0) | 15.2 (95% CI: 13.816.7) | 97.1 (95% CI: 96.897.3) | 3.95 (95% CI: 3.614.30) | 0.66 (95% CI: 0.620.70) |
| No Alert | 512 | 17,166 | ||||||
| Death | Yes (N=450) | No (N=19,581) | ||||||
| Alert | 244 | 2109 | 54.2 (95% CI: 49.658.8) | 89.2 (95% CI: 88.889.7) | 10.4 (95% CI: 9.211.7) | 98.8 (95% CI: 98.799.0) | 5.03 (95% CI: 4.585.53) | 0.51 (95% CI: 0.460.57) |
| No Alert | 206 | 17,472 | ||||||
Among patients identified by the PT, there were no differences in the proportion of patients who were transferred to the ICU or who died in the intervention group as compared with the control group (Table 3). In addition, although there was no difference in LOS in the intervention group compared with the control group, identification by the PT was associated with a significantly longer median LOS (7.01 days vs 2.94 days, P<0.001). The largest numbers of patients who were transferred to the ICU or died did so in the first hospital day, and 60% of patients who were transferred to the ICU did so in the first 4 days, whereas deaths were more evenly distributed across the hospital stay.
| Outcomes by Alert Statusa | ||||||||
|---|---|---|---|---|---|---|---|---|
| Alert Study Group | No‐Alert Study Group | |||||||
| Intervention, N=1194 | Control, N=1159 | Intervention, N=8717 | Control, N=8961 | |||||
| N | % | N | % | N | % | N | % | |
| ||||||||
| ICU Transfer | ||||||||
| Yes | 192 | 16 | 166 | 14 | 252 | 3 | 260 | 3 |
| No | 1002 | 84 | 993 | 86 | 8465 | 97 | 8701 | 97 |
| Death | ||||||||
| Yes | 127 | 11 | 117 | 10 | 96 | 1 | 110 | 1 |
| No | 1067 | 89 | 1042 | 90 | 8621 | 99 | 8851 | 99 |
| LOS from admit to discharge, median (IQR), da | 7.07 (3.9912.15) | 6.92 (3.8212.67) | 2.97 (1.775.33) | 2.91 (1.745.19) | ||||
DISCUSSION
We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.
Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).
Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.
We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.
As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.
The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]
Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).
An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]
In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.
Acknowledgments
Disclosures: This study was funded in part by the Barnes‐Jewish Hospital Foundation and by Grant No. UL1 RR024992 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and the NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH.
- , , , et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299:2294–2303.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368–1377.
- , , , et al. Before‐after study of a standardized hospital order set for the management of septic shock. Crit Care Med. 2007;34:2707–2713.
- , , . Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Crit Care Med. 2008;36:296–327.
- , , , et al. Septic shock: an analysis of outcomes for patients with onset on hospital wards versus intensive care units. Crit Care Med. 1998;26:1020–1024.
- , , , , . Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity. J Gen Intern Med. 2003;18:77–83.
- , , , , , . Causes and effects of surgical delay in patients with hip fracture: a cohort study. Ann Intern Med. 2011;155:226–233.
- , , , . Comprehensive stroke centers overcome the weekend versus weekday gap in stroke treatment and mortality. Stroke. 2011;42:2403–2409.
- , , , , , . Hospital‐wide impact of a standardized order set for the management of bacteremic severe sepsis. Crit Care Med. 2009;37:819–824.
- , , , et al. Implementation of a real‐time computerized sepsis alert in non–intensive care unit patients. Crit Care Med. 2011;39:469–473.
- , , , et al. Toward a two‐tier clinical warning system for hospitalized patients. AMIA Annu Symp Proc. 2011;2011:511–519.
- , , , , , . Migrating toward a next‐generation clinical decision support application: the BJC HealthCare experience. AMIA Annu Symp Proc. 2007;344–348.
- , , , . Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system. J Hosp Med. 2011;7:224–230.
- , , , . Unplanned transfers to a medical intensive care unit: causes and relationship to preventable errors in care. J Hosp Med. 2011;6:68–72.
- , , , , , . Unplanned transfers to the intensive care unit: the role of the shock index. J Hosp Med. 2010;5:460–465.
- Institute for Healthcare Improvement. Early warning systems: the next level of rapid response. Available at: http://www.ihi.org/IHI/Programs/AudioAndWebPrograms/ExpeditionEarlyWarningSystemsTheNextLevelofRapidResponse.htmplayerwmp. Accessed April 6, 2011.
- , , , , , . Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7:388–395.
- , , . Early warning systems: the next level of rapid response. Nursing. 2012;42:38–44.
- , , , et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at‐risk patients on the ward. Intensive Care Med. 2007;33:667–679.
- , , , . Timing and teamwork—an observational pilot study of patients referred to a Rapid Response Team with the aim of identifying factors amenable to re‐design of a Rapid Response System. Resuscitation. 2012;83:782–787.
- , , , . Rapid response: a quality improvement conundrum. J Hosp Med. 2009;4:255–257.
- , , , et al. Introducing critical care outreach: a ward‐randomised trial of phased introduction in a general hospital. Intensive Care Med. 2004;30:1398–1404.
- . Out of our reach? Assessing the impact of introducing critical care outreach service. Anaesthesiology. 2003;58:882–885.
- , , . Effect of the critical care outreach team on patient survival to discharge from hospital and readmission to critical care: non‐randomised population based study. BMJ. 2003;327:1014–1016.
- , , . Reducing mortality and avoiding preventable ICU utilization: analysis of a successful rapid response program using APR DRGs [published online ahead of print March 10, 2010]. J Healthc Qual. doi: 10.1111/j.1945‐1474.2010.00084.x.
- , , , et al. Introduction of the medical emergency team (MET) system: a cluster‐randomised control trial. Lancet. 2005;365:2091–2097.
- , , , , , . The impact of the introduction of critical care outreach services in England: a multicentre interrupted time‐series analysis. Crit Care. 2007;11:R113.
- . Rapid response systems now established at 2,900 hospitals. Hospitalist News. March 2010;3:1.
- , , , , . Rapid response teams: a systematic review and meta‐analysis. Arch Intern Med. 2010;170:18–26.
- , , , et al. Outreach and early warning systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev. 2007;3:CD005529.
- , , . Rapid‐response teams. N Engl J Med. 2011;365:139–146.
- , , . Early warning systems. Hosp Chron. 2012;7(suppl 1):37–43.
- , , , et al. Grand challenges in clinical decision support. J Biomed Inform. 2008;41(2):387–392.
- , , , et al. Utility of commonly captured data from an EHR to identify hospitalized patients at risk for clinical deterioration. AMIA Annu Symp Proc. 2007;404–408.
- , , , , , . Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125(4)e763–e769.
- , , , . In‐hospital mortality after serious adverse events on medical and surgical nursing units: a mixed methods study [published online ahead of print July 24, 2012]. J Clin Nurs. doi: 10.1111/j.1365‐2702.2012.04154.x.
- , , , et al. Managing deteriorating patients: registered nurses' performance in a simulated setting. Open Nurs J. 2011;5:120–126.
Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]
The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.
METHODS
Study Location
The study was conducted at Barnes‐Jewish Hospital, a 1250‐bed academic medical center in St. Louis, Missouri. Eight adult medicine wards were assessed from July 2007 through December 2011. The medicine wards are closed areas with patient care delivered by dedicated house staff physicians under the supervision of a board‐certified attending physician. The study was approved by the Washington University School of Medicine Human Studies Committee.
Study Period
The period from July 2007 through January 2010 was used to train and retrospectively test the prediction model. The period from January 2011 through December 2011 was used to prospectively validate the model during a randomized trial using alerts generated from the prediction model.
Patients
Electronically captured clinical data were housed in a centralized clinical data repository. This repository cataloged 28,927 hospital visits from 19,116 distinct patients between July 2007 and January 2010. It contained a rich set of demographic and medical data for each of the visits, such as patient age, manually collected vital‐sign data, pharmacy data, laboratory data, and intensive care unit (ICU) transfer. This study served as a proof of concept for our vision of using machine learning to identify at‐risk patients and ultimately to perform real‐time event detection and interventions.
Algorithm Overview
Details regarding the predictive model development have been previously described.[11] To predict ICU transfer for patients housed on general medical wards, we used logistic regression, employing a novel framework to analyze the data stream from each patient, assigning scores to reflect the probability of ICU transfer to each patient.
Before building the model, several preprocessing steps were applied to eliminate outliers and find an appropriate representation of patients' states. For each of 36 input variables we specified acceptable ranges based on the domain knowledge of the medical experts on our team. For any value that was outside of the medically conceivable range, we replaced it by the mean value for that patient, if available. Values for every continuous parameter were scaled so that all measurements lay in the interval [0, 1] and were normalized by the minimum and maximum of the parameter. To capture the temporal effects in our data, we retained a sliding window of all the collected data points within the last 24 hours. We then subdivided these data into a series of 6 sequential buckets of 4 hours each.
To capture variations within a bucket, we computed 3 values for each feature in the bucket: the minimum, maximum, and mean data points. Each of the resulting 3n values was input to the logistic regression equation as separate variables. To deal with missing data points within the buckets, we used the patients' most recent reading from any time earlier in the hospital stay, if available. If no prior values existed, we used mean values calculated over the entire historical dataset. Bucket 6 max/min/mean represents the most recent 4‐hour window from the preceding 24‐hour time period for the maximum, minimum, and mean values, respectively. By itself, logistic regression does not operate on time‐series data. That is, each variable input to the logistic equation corresponds to exactly 1 data point (eg, a blood‐pressure variable would consist of a single blood‐pressure reading). In a clinical application, however, it is important to capture unusual changes in vital‐sign data over time. Such changes may precede clinical deterioration by hours, providing a chance to intervene if detected early enough. In addition, not all readings in time‐series data should be treated equally; the value of some kinds of data may change depending on their age. For example, a patient's condition may be better reflected by a blood‐oxygenation reading collected 1 hour ago than a reading collected 12 hours ago. This is the rationale for our use of a sliding window of all collected data points within the last 24 hours performed in a real‐time basis.
The algorithm was first implemented in MATLAB (Natick, MA). For the purposes of training, we used a single 24‐hour window of data from each patient. For patients admitted to ICU, this window was 26 hours to 2 hours prior to ICU admission; for all other patients, this window consisted of the first 24 hours of their hospital stay. The dataset's 36 input variables were divided into buckets and min/mean/max features wherever applicable, resulting in 398 variables. The first half of the dataset was used to train the model. We then used the second half of the dataset as the validation dataset. We generated a predicted outcome for each case in the validation data, using the model parameter coefficients derived from the training data. We also employed bootstrap aggregation to improve classification accuracy and to address overfitting. We then applied various threshold cut‐points to convert these predictions into binary values and compared the results against the ICU transfer outcome. A threshold of 0.9760 for specificity was chosen to achieve a sensitivity of approximately 40%. These operating characteristics were chosen in turn to generate a manageable number of alerts per hospital nursing unit per day (estimated at 12 per nursing unit per day). At this cut‐point the C‐statistic was 0.8834, with an overall accuracy of 0.9292.
In order to train the logistic model, we used a single 24‐hour window of data for each patient. However, in a system that predicts patients' outcomes in real time, scores are recomputed each time new data are entered into the database. Hence, patients have a series of scores over the length of their hospital stay, and an alert is triggered when any one of these scores is above the chosen threshold.
Once the model was developed, we implemented it in an internally developed, Java‐based clinical decision support rules engine, which identified when new data relevant to the model were available in a real‐time central data repository. The rules engine queried the data repository to acquire all data needed to evaluate the model. The score was calculated with each relevant new data point, and an alert was generated when the score exceeded the cut‐point threshold. We then prospectively validated these alerts on patients on 8 general medical wards at Barnes Jewish Hospital. Details regarding the architecture of our clinical decision support system have been previously published.[12] The sensitivity and positive predictive values for ICU transfer for these alerts were tracked during an intervention trial that ran from January 24, 2011, through December 31, 2011. Four general medical wards were randomized to the intervention group and 4 wards were randomized to the control group. The 8 general medical wards were ordered according to their alert rates based upon the historical data from July 2007 through January 2010, creating 4 pairs of wards in ascending order of alert rate. Within each of the 4 pairs, 1 member of the pair was randomized to the intervention group and the other to the control group using a random number generator.
Real‐time automated alerts generated 24 hours per day, 7 days per week from the predictive algorithm were sent to the charge‐nurse pager on the intervention units. Alerts were also generated and stored in the database on the control units, but these alerts were not sent to the charge nurse on those units. The alerts were sent to the charge nurses on the respective wards, as these individuals were thought to be in the best position to perform the initial assessment of the alerted patients, especially during evening hours when physician staffing was reduced. The charge nurses assessed the intervention‐group patients and were instructed to contact the responsible physician (hospitalist or internal medicine house officer) to inform them of the alert, or to call the rapid response team (RRT) if the patient's condition already appeared to be significantly deteriorating.
Descriptive statistics for algorithm sensitivity and positive predictive value and for patient outcomes were performed. Associations between alerts and the primary outcome, ICU transfer, were determined, as well as the impact of alerts in the intervention group compared with the control group, using [2] tests. The same analyses were performed for patient death. Differences in length of stay (LOS) were assessed using the Wilcoxon rank sum test.
RESULTS
Predictive Model
The variables with the greatest coefficients contributing to the PT model included respiratory rate, oxygen saturation, shock index, systolic blood pressure, anticoagulation use, heart rate, and diastolic blood pressure. A complete list of variables is provided in the Appendix (see Supporting Information in the online version of this article). All but 1 are routinely collected vital‐sign measures, and all but 1 occur in the 4‐hour period immediately prior to the alert (bucket 6).
Prospective Trial
Patient characteristics are presented in Table 1. Patients were well matched for race, sex, age, and underlying diagnoses. All alerts reported to the charge nurses were to be associated with a call from the charge nurse to the responsible physician caring for the alerted patient. The mean number of alerts per alerted patient was 1.8 (standard deviation=1.7). Patients meeting the alert threshold were at nearly 5.3‐fold greater risk of ICU transfer (95% confidence interval [CI]: 4.6‐6.0) than those not satisfying the alert threshold (358 of 2353 [15.2%; 95% CI: 13.8%‐16.7%] vs 512 of 17678 [2.9%; 95% CI: 2.7%‐3.2%], respectively; P<0.0001). Patients with alerts were at 8.9‐fold greater risk of death (95% CI: 7.4‐10.7) than those without alerts (244 of 2353 [10.4%; 95% CI: 9.2%‐11.7%] vs 206 of 17678 [1.2%; 95% CI: 1.0%‐1.3%], respectively; P<0.0001). Operating characteristics of the PT from the prospective trial are shown in Table 2. Alerts occurred a median of 25.5 hours prior to ICU transfer (interquartile range, 7.00‐81.75) and 8 hours prior to death (interquartile range, 4.09‐15.66).
| Study Group | ||||||
|---|---|---|---|---|---|---|
| Control (N=10,120) | Intervention (N=9911) | |||||
| ||||||
| Race | N | % | N | % | ||
| White | 5,062 | 50 | 4,934 | 50 | ||
| Black | 4,864 | 48 | 4,790 | 48 | ||
| Other | 194 | 2 | 187 | 2 | ||
| Sex | ||||||
| F | 5,355 | 53 | 5,308 | 54 | ||
| M | 4,765 | 47 | 4,603 | 46 | ||
| Age at discharge, median (IQR), y | 57 (4469) | 57 (4470) | ||||
| Top 10 ICD‐9 descriptions and counts, n (%) | ||||||
| 1 | Diseases of the digestive system | 1,774 (17.5) | Diseases of the digestive system | 1,664 (16.7) | ||
| 2 | Diseases of the circulatory system | 1,252 (12.4) | Diseases of the circulatory system | 1,253 (12.6) | ||
| 3 | Diseases of the respiratory system | 1,236 (12.2) | Diseases of the respiratory system | 1,210 (12.2) | ||
| 4 | Injury and poisoning | 864 (8.5) | Injury and poisoning | 849 (8.6) | ||
| 5 | Endocrine, nutritional, and metabolic diseases, and immunity disorders | 797 (7.9) | Diseases of the genitourinary system | 795 (8.0) | ||
| 6 | Diseases of the genitourinary system | 762 (7.5) | Endocrine, nutritional, and metabolic diseases, and immunity disorders | 780 (7.9) | ||
| 7 | Infectious and parasitic diseases | 555 (5.5) | Infectious and parasitic diseases | 549 (5.5) | ||
| 8 | Neoplasms | 547 (5.4) | Neoplasms | 465 (4.7) | ||
| 9 | Diseases of the blood and blood‐forming organs | 426 (4.2) | Diseases of the blood and blood‐forming organs | 429 (4.3) | ||
| 10 | Symptoms, signs, and ill‐defined conditions and factors influencing health status | 410 (4.1) | Diseases of the musculoskeletal system and connective tissue | 399 (4.0) | ||
| Sensitivity, % | Specificity, % | PPV, % | NPV, % | Positive Likelihood Ratio | Negative Likelihood Ratio | |||
|---|---|---|---|---|---|---|---|---|
| ||||||||
| ICU Transfer | Yes (N=870) | No (N=19,161) | ||||||
| Alert | 358 | 1,995 | 41.1 (95% CI: 37.944.5) | 89.6 (95% CI: 89.290.0) | 15.2 (95% CI: 13.816.7) | 97.1 (95% CI: 96.897.3) | 3.95 (95% CI: 3.614.30) | 0.66 (95% CI: 0.620.70) |
| No Alert | 512 | 17,166 | ||||||
| Death | Yes (N=450) | No (N=19,581) | ||||||
| Alert | 244 | 2109 | 54.2 (95% CI: 49.658.8) | 89.2 (95% CI: 88.889.7) | 10.4 (95% CI: 9.211.7) | 98.8 (95% CI: 98.799.0) | 5.03 (95% CI: 4.585.53) | 0.51 (95% CI: 0.460.57) |
| No Alert | 206 | 17,472 | ||||||
Among patients identified by the PT, there were no differences in the proportion of patients who were transferred to the ICU or who died in the intervention group as compared with the control group (Table 3). In addition, although there was no difference in LOS in the intervention group compared with the control group, identification by the PT was associated with a significantly longer median LOS (7.01 days vs 2.94 days, P<0.001). The largest numbers of patients who were transferred to the ICU or died did so in the first hospital day, and 60% of patients who were transferred to the ICU did so in the first 4 days, whereas deaths were more evenly distributed across the hospital stay.
| Outcomes by Alert Statusa | ||||||||
|---|---|---|---|---|---|---|---|---|
| Alert Study Group | No‐Alert Study Group | |||||||
| Intervention, N=1194 | Control, N=1159 | Intervention, N=8717 | Control, N=8961 | |||||
| N | % | N | % | N | % | N | % | |
| ||||||||
| ICU Transfer | ||||||||
| Yes | 192 | 16 | 166 | 14 | 252 | 3 | 260 | 3 |
| No | 1002 | 84 | 993 | 86 | 8465 | 97 | 8701 | 97 |
| Death | ||||||||
| Yes | 127 | 11 | 117 | 10 | 96 | 1 | 110 | 1 |
| No | 1067 | 89 | 1042 | 90 | 8621 | 99 | 8851 | 99 |
| LOS from admit to discharge, median (IQR), da | 7.07 (3.9912.15) | 6.92 (3.8212.67) | 2.97 (1.775.33) | 2.91 (1.745.19) | ||||
DISCUSSION
We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.
Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).
Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.
We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.
As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.
The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]
Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).
An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]
In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.
Acknowledgments
Disclosures: This study was funded in part by the Barnes‐Jewish Hospital Foundation and by Grant No. UL1 RR024992 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and the NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH.
Timely interventions are essential in the management of complex medical conditions such as new‐onset sepsis in order to prevent rapid progression to severe sepsis and septic shock.[1, 2, 3, 4, 5] Similarly, rapid identification and appropriate treatment of other medical and surgical conditions have been associated with improved outcomes.[6, 7, 8] We previously developed a real‐time, computerized prediction tool (PT) using recursive partitioning regression tree analysis for the identification of impending sepsis for use on general hospital wards.[9] We also showed that implementation of a real‐time computerized sepsis alert on hospital wards based on the PT resulted in increased use of early interventions, including antibiotic escalation, intravenous fluids, oxygen therapy, and diagnostics in patients identified as at risk.[10]
The first goal of this study was to develop an updated PT for use on hospital wards that could be used to predict subsequent global clinical deterioration and the need for a higher level of care. The second goal was to determine whether simply providing a real‐time alert to nursing staff based on the updated PT resulted in any demonstrable changes in patient outcomes.
METHODS
Study Location
The study was conducted at Barnes‐Jewish Hospital, a 1250‐bed academic medical center in St. Louis, Missouri. Eight adult medicine wards were assessed from July 2007 through December 2011. The medicine wards are closed areas with patient care delivered by dedicated house staff physicians under the supervision of a board‐certified attending physician. The study was approved by the Washington University School of Medicine Human Studies Committee.
Study Period
The period from July 2007 through January 2010 was used to train and retrospectively test the prediction model. The period from January 2011 through December 2011 was used to prospectively validate the model during a randomized trial using alerts generated from the prediction model.
Patients
Electronically captured clinical data were housed in a centralized clinical data repository. This repository cataloged 28,927 hospital visits from 19,116 distinct patients between July 2007 and January 2010. It contained a rich set of demographic and medical data for each of the visits, such as patient age, manually collected vital‐sign data, pharmacy data, laboratory data, and intensive care unit (ICU) transfer. This study served as a proof of concept for our vision of using machine learning to identify at‐risk patients and ultimately to perform real‐time event detection and interventions.
Algorithm Overview
Details regarding the predictive model development have been previously described.[11] To predict ICU transfer for patients housed on general medical wards, we used logistic regression, employing a novel framework to analyze the data stream from each patient, assigning scores to reflect the probability of ICU transfer to each patient.
Before building the model, several preprocessing steps were applied to eliminate outliers and find an appropriate representation of patients' states. For each of 36 input variables we specified acceptable ranges based on the domain knowledge of the medical experts on our team. For any value that was outside of the medically conceivable range, we replaced it by the mean value for that patient, if available. Values for every continuous parameter were scaled so that all measurements lay in the interval [0, 1] and were normalized by the minimum and maximum of the parameter. To capture the temporal effects in our data, we retained a sliding window of all the collected data points within the last 24 hours. We then subdivided these data into a series of 6 sequential buckets of 4 hours each.
To capture variations within a bucket, we computed 3 values for each feature in the bucket: the minimum, maximum, and mean data points. Each of the resulting 3n values was input to the logistic regression equation as separate variables. To deal with missing data points within the buckets, we used the patients' most recent reading from any time earlier in the hospital stay, if available. If no prior values existed, we used mean values calculated over the entire historical dataset. Bucket 6 max/min/mean represents the most recent 4‐hour window from the preceding 24‐hour time period for the maximum, minimum, and mean values, respectively. By itself, logistic regression does not operate on time‐series data. That is, each variable input to the logistic equation corresponds to exactly 1 data point (eg, a blood‐pressure variable would consist of a single blood‐pressure reading). In a clinical application, however, it is important to capture unusual changes in vital‐sign data over time. Such changes may precede clinical deterioration by hours, providing a chance to intervene if detected early enough. In addition, not all readings in time‐series data should be treated equally; the value of some kinds of data may change depending on their age. For example, a patient's condition may be better reflected by a blood‐oxygenation reading collected 1 hour ago than a reading collected 12 hours ago. This is the rationale for our use of a sliding window of all collected data points within the last 24 hours performed in a real‐time basis.
The algorithm was first implemented in MATLAB (Natick, MA). For the purposes of training, we used a single 24‐hour window of data from each patient. For patients admitted to ICU, this window was 26 hours to 2 hours prior to ICU admission; for all other patients, this window consisted of the first 24 hours of their hospital stay. The dataset's 36 input variables were divided into buckets and min/mean/max features wherever applicable, resulting in 398 variables. The first half of the dataset was used to train the model. We then used the second half of the dataset as the validation dataset. We generated a predicted outcome for each case in the validation data, using the model parameter coefficients derived from the training data. We also employed bootstrap aggregation to improve classification accuracy and to address overfitting. We then applied various threshold cut‐points to convert these predictions into binary values and compared the results against the ICU transfer outcome. A threshold of 0.9760 for specificity was chosen to achieve a sensitivity of approximately 40%. These operating characteristics were chosen in turn to generate a manageable number of alerts per hospital nursing unit per day (estimated at 12 per nursing unit per day). At this cut‐point the C‐statistic was 0.8834, with an overall accuracy of 0.9292.
In order to train the logistic model, we used a single 24‐hour window of data for each patient. However, in a system that predicts patients' outcomes in real time, scores are recomputed each time new data are entered into the database. Hence, patients have a series of scores over the length of their hospital stay, and an alert is triggered when any one of these scores is above the chosen threshold.
Once the model was developed, we implemented it in an internally developed, Java‐based clinical decision support rules engine, which identified when new data relevant to the model were available in a real‐time central data repository. The rules engine queried the data repository to acquire all data needed to evaluate the model. The score was calculated with each relevant new data point, and an alert was generated when the score exceeded the cut‐point threshold. We then prospectively validated these alerts on patients on 8 general medical wards at Barnes Jewish Hospital. Details regarding the architecture of our clinical decision support system have been previously published.[12] The sensitivity and positive predictive values for ICU transfer for these alerts were tracked during an intervention trial that ran from January 24, 2011, through December 31, 2011. Four general medical wards were randomized to the intervention group and 4 wards were randomized to the control group. The 8 general medical wards were ordered according to their alert rates based upon the historical data from July 2007 through January 2010, creating 4 pairs of wards in ascending order of alert rate. Within each of the 4 pairs, 1 member of the pair was randomized to the intervention group and the other to the control group using a random number generator.
Real‐time automated alerts generated 24 hours per day, 7 days per week from the predictive algorithm were sent to the charge‐nurse pager on the intervention units. Alerts were also generated and stored in the database on the control units, but these alerts were not sent to the charge nurse on those units. The alerts were sent to the charge nurses on the respective wards, as these individuals were thought to be in the best position to perform the initial assessment of the alerted patients, especially during evening hours when physician staffing was reduced. The charge nurses assessed the intervention‐group patients and were instructed to contact the responsible physician (hospitalist or internal medicine house officer) to inform them of the alert, or to call the rapid response team (RRT) if the patient's condition already appeared to be significantly deteriorating.
Descriptive statistics for algorithm sensitivity and positive predictive value and for patient outcomes were performed. Associations between alerts and the primary outcome, ICU transfer, were determined, as well as the impact of alerts in the intervention group compared with the control group, using [2] tests. The same analyses were performed for patient death. Differences in length of stay (LOS) were assessed using the Wilcoxon rank sum test.
RESULTS
Predictive Model
The variables with the greatest coefficients contributing to the PT model included respiratory rate, oxygen saturation, shock index, systolic blood pressure, anticoagulation use, heart rate, and diastolic blood pressure. A complete list of variables is provided in the Appendix (see Supporting Information in the online version of this article). All but 1 are routinely collected vital‐sign measures, and all but 1 occur in the 4‐hour period immediately prior to the alert (bucket 6).
Prospective Trial
Patient characteristics are presented in Table 1. Patients were well matched for race, sex, age, and underlying diagnoses. All alerts reported to the charge nurses were to be associated with a call from the charge nurse to the responsible physician caring for the alerted patient. The mean number of alerts per alerted patient was 1.8 (standard deviation=1.7). Patients meeting the alert threshold were at nearly 5.3‐fold greater risk of ICU transfer (95% confidence interval [CI]: 4.6‐6.0) than those not satisfying the alert threshold (358 of 2353 [15.2%; 95% CI: 13.8%‐16.7%] vs 512 of 17678 [2.9%; 95% CI: 2.7%‐3.2%], respectively; P<0.0001). Patients with alerts were at 8.9‐fold greater risk of death (95% CI: 7.4‐10.7) than those without alerts (244 of 2353 [10.4%; 95% CI: 9.2%‐11.7%] vs 206 of 17678 [1.2%; 95% CI: 1.0%‐1.3%], respectively; P<0.0001). Operating characteristics of the PT from the prospective trial are shown in Table 2. Alerts occurred a median of 25.5 hours prior to ICU transfer (interquartile range, 7.00‐81.75) and 8 hours prior to death (interquartile range, 4.09‐15.66).
| Study Group | ||||||
|---|---|---|---|---|---|---|
| Control (N=10,120) | Intervention (N=9911) | |||||
| ||||||
| Race | N | % | N | % | ||
| White | 5,062 | 50 | 4,934 | 50 | ||
| Black | 4,864 | 48 | 4,790 | 48 | ||
| Other | 194 | 2 | 187 | 2 | ||
| Sex | ||||||
| F | 5,355 | 53 | 5,308 | 54 | ||
| M | 4,765 | 47 | 4,603 | 46 | ||
| Age at discharge, median (IQR), y | 57 (4469) | 57 (4470) | ||||
| Top 10 ICD‐9 descriptions and counts, n (%) | ||||||
| 1 | Diseases of the digestive system | 1,774 (17.5) | Diseases of the digestive system | 1,664 (16.7) | ||
| 2 | Diseases of the circulatory system | 1,252 (12.4) | Diseases of the circulatory system | 1,253 (12.6) | ||
| 3 | Diseases of the respiratory system | 1,236 (12.2) | Diseases of the respiratory system | 1,210 (12.2) | ||
| 4 | Injury and poisoning | 864 (8.5) | Injury and poisoning | 849 (8.6) | ||
| 5 | Endocrine, nutritional, and metabolic diseases, and immunity disorders | 797 (7.9) | Diseases of the genitourinary system | 795 (8.0) | ||
| 6 | Diseases of the genitourinary system | 762 (7.5) | Endocrine, nutritional, and metabolic diseases, and immunity disorders | 780 (7.9) | ||
| 7 | Infectious and parasitic diseases | 555 (5.5) | Infectious and parasitic diseases | 549 (5.5) | ||
| 8 | Neoplasms | 547 (5.4) | Neoplasms | 465 (4.7) | ||
| 9 | Diseases of the blood and blood‐forming organs | 426 (4.2) | Diseases of the blood and blood‐forming organs | 429 (4.3) | ||
| 10 | Symptoms, signs, and ill‐defined conditions and factors influencing health status | 410 (4.1) | Diseases of the musculoskeletal system and connective tissue | 399 (4.0) | ||
| Sensitivity, % | Specificity, % | PPV, % | NPV, % | Positive Likelihood Ratio | Negative Likelihood Ratio | |||
|---|---|---|---|---|---|---|---|---|
| ||||||||
| ICU Transfer | Yes (N=870) | No (N=19,161) | ||||||
| Alert | 358 | 1,995 | 41.1 (95% CI: 37.944.5) | 89.6 (95% CI: 89.290.0) | 15.2 (95% CI: 13.816.7) | 97.1 (95% CI: 96.897.3) | 3.95 (95% CI: 3.614.30) | 0.66 (95% CI: 0.620.70) |
| No Alert | 512 | 17,166 | ||||||
| Death | Yes (N=450) | No (N=19,581) | ||||||
| Alert | 244 | 2109 | 54.2 (95% CI: 49.658.8) | 89.2 (95% CI: 88.889.7) | 10.4 (95% CI: 9.211.7) | 98.8 (95% CI: 98.799.0) | 5.03 (95% CI: 4.585.53) | 0.51 (95% CI: 0.460.57) |
| No Alert | 206 | 17,472 | ||||||
Among patients identified by the PT, there were no differences in the proportion of patients who were transferred to the ICU or who died in the intervention group as compared with the control group (Table 3). In addition, although there was no difference in LOS in the intervention group compared with the control group, identification by the PT was associated with a significantly longer median LOS (7.01 days vs 2.94 days, P<0.001). The largest numbers of patients who were transferred to the ICU or died did so in the first hospital day, and 60% of patients who were transferred to the ICU did so in the first 4 days, whereas deaths were more evenly distributed across the hospital stay.
| Outcomes by Alert Statusa | ||||||||
|---|---|---|---|---|---|---|---|---|
| Alert Study Group | No‐Alert Study Group | |||||||
| Intervention, N=1194 | Control, N=1159 | Intervention, N=8717 | Control, N=8961 | |||||
| N | % | N | % | N | % | N | % | |
| ||||||||
| ICU Transfer | ||||||||
| Yes | 192 | 16 | 166 | 14 | 252 | 3 | 260 | 3 |
| No | 1002 | 84 | 993 | 86 | 8465 | 97 | 8701 | 97 |
| Death | ||||||||
| Yes | 127 | 11 | 117 | 10 | 96 | 1 | 110 | 1 |
| No | 1067 | 89 | 1042 | 90 | 8621 | 99 | 8851 | 99 |
| LOS from admit to discharge, median (IQR), da | 7.07 (3.9912.15) | 6.92 (3.8212.67) | 2.97 (1.775.33) | 2.91 (1.745.19) | ||||
DISCUSSION
We have demonstrated that a relatively simple hospital‐specific method for generating a PT derived from routine laboratory and hemodynamic values is capable of predicting clinical deterioration and the need for ICU transfer, as well as hospital mortality, in non‐ICU patients admitted to general hospital wards. We also found that the PT identified a sicker patient population as manifest by longer hospital LOS. The methods used in generating this real‐time PT are relatively simple and easily executed with the use of an electronic medical record (EMR) system. However, our data also showed that simply providing an alert to nursing units based on the PT did not result in any demonstrable improvement in patient outcomes. Moreover, our PT and intervention in their current form have substantial limitations, including low sensitivity and positive predictive value, high possibility of alert fatigue, and no clear clinical impact. These limitations suggest that this approach has limited applicability in its current form.
Unplanned ICU transfers occurring as early as within 8 hours of hospitalization are relatively common and associated with increased mortality.[13] Bapoje et al evaluated a total of 152 patients over 1 year who had unplanned ICU transfers.[14] The most common reason was worsening of the problem for which the patient was admitted (48%). Other investigators have also attempted to identify predictors for clinical deterioration resulting in unplanned ICU transfer that could be employed in a PT or early warning system (EWS). Keller et al evaluated 50 consecutive general medical patients with unplanned ICU transfers between 2003 and 2004.[15] Using a case‐control methodology, these investigators found shock index values>0.85 to be the best predictor for subsequent unplanned ICU transfer (P<0.02; odds ratio: 3.0).
Organizations such as the Institute for Healthcare Improvement have called for the development and implementation of EWSs in order to direct the activities of RRTs and improve outcomes.[16] Escobar et al carried out a retrospective case‐control study using as the unit of analysis 12‐hour patient shifts on hospital wards.[17] Using logistic regression and split validation, they developed a PT for ICU transfer from clinical variables available in their EMR. The EMR derived PT had a C‐statistic of 0.845 in the derivation dataset and 0.775 in the validation dataset, concluding that EMR‐based detection of impending deterioration outside the ICU is feasible in integrated healthcare delivery systems.
We found that simply providing an alert to nursing units did not result in any demonstrable improvements in the outcomes of high‐risk patients identified by our PT. This may have been due to simply relying on the alerted nursing staff to make phone calls to physicians and not linking a specific and effective patient‐directed intervention to the PT. Other investigators have similarly observed that the use of an EWS or PT may not result in outcome improvements.[18] Gao et al performed an analysis of 31 studies describing hospital track and trigger EWSs.[19] They found little evidence of reliability, validity, and utility of these systems. Peebles et al showed that even when high‐risk non‐ICU patients are identified, delays in providing appropriate therapies occur, which may explain the lack of efficacy of EWSs and RRTs.[20] These observations suggest that there is currently a paucity of validated interventions available to improve outcome in deteriorating patients, despite our ability to identify patients who are at risk for such deterioration.
As a result of mandates from quality‐improvement organizations, most US hospitals currently employ RRTs for emergent mobilization of resources when a clinically deteriorating patient is identified on a hospital ward.[21] However, as noted above, there is limited evidence to suggest that RRTs contribute to improved patient outcomes.[22, 23, 24, 25, 26, 27] The potential importance of this is reflected in a recent report suggesting that 2900 US hospitals now have rapid‐response systems in place without clear demonstration of their overall efficacy.[28] Linking rapid‐response interventions with a validated real‐time alert may represent a way of improving the effectiveness of such interventions.[29, 30, 31, 32, 33, 34] Our data showed that hospital LOS was statistically longer among alerted patients compared with nonalerted patients. This supports the conclusion that the alerts helped identify a sicker group of patients, but the nursing alerts did not appear to change outcomes. This finding also seems to refute the hypothesis that simply linking an intervention to a PT will improve outcomes, albeit the intervention we employed may not have been robust enough to influence patient outcomes.
The development of accurate real‐time EWSs holds the potential to identify patients at risk for clinical deterioration at an earlier point in time when rescue interventions can be implemented in a potentially more effective manner in both adults and children.[35] Unfortunately, the ideal intervention to be applied in this situation is unknown. Our experience suggests that successful interventions will require a more integrated approach than simply providing an alert with general management principles. As a result of our experience, we are undertaking a randomized clinical trial in 2013 to determine whether linking a patient‐specific intervention to a PT will result in improved outcomes. The intervention we will be testing is to have the RRT immediately notified about alerted patients so as to formally evaluate them and to determine the need for therapeutic interventions, and to administer such interventions as needed and/or transfer the alerted patients to a higher level of care as deemed necessary. Additionally, we are updating our PT with more temporal data to determine if this will improve its accuracy. One of these updates will include linking the PT to wirelessly obtained continuous oximetry and heart‐rate data, using minimally intrusive sensors, to establish a 2‐tiered EWS.[11]
Our study has several important limitations. First, the PT was developed using local data, and thus the results may not be applicable to other settings. However, our model shares many of the characteristics identified in other clinical‐deterioration PTs.[15, 17] Second, the positive prediction value of 15.2% for ICU transfer may not be clinically useful due to the large number of false‐positive results. Moreover, the large number of false positives could result in alert fatigue, causing alerts to be ignored. Third, although the charge nurses were supposed to call the responsible physicians for the alerted patients, we did not determine whether all these calls occurred or whether they resulted in any meaningful changes in monitoring or patient treatment. This is important because lack of an effective intervention or treatment would make the intervention group much more like our control group. Future studies are needed to assess the impact of an integrated intervention (eg, notification of experienced RRT members with adequate resource access) to determine if patient outcomes can be impacted by the use of an EWS. Finally, we did not compare the performance of our PT to other models such as the modified early warning score (MEWS).
An additional limitation to consider is that our PT offered no new information to the nurse manager, or the PT did not change the opinions of the charge nurses. This is supported by a recent study of 63 serious adverse outcomes in a Belgian teaching hospital where death was the final outcome.[36] Survey results revealed that nurses were often unaware that their patients were deteriorating before the crisis. Nurses also reported threshold levels for concern for abnormal vital signs that suggested they would call for assistance relatively late in clinical crises. The limited ability of nursing staff to identify deteriorating patients is also supported by a recent simulation study demonstrating that nurses did identify that patients were deteriorating, but as each patient deteriorated staff performance declined, with a reduction in all observational records and actions.[37]
In summary, we have demonstrated that a relatively simple hospital‐specific PT could accurately identify patients on general medicine wards who subsequently developed clinical deterioration and the need for ICU transfer, as well as hospital mortality. However, no improvements in patient outcomes were found from reporting this information to nursing wards on a real‐time basis. The low positive predictive value of the alerts, local development of the EWS, and absence of improved outcomes substantially limits the broader application of this system in its current form. Continued efforts are needed to identify and implement systems that will not only accurately identify high‐risk patients on general wards but also intervene to improve their outcomes.
Acknowledgments
Disclosures: This study was funded in part by the Barnes‐Jewish Hospital Foundation and by Grant No. UL1 RR024992 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and the NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH.
- , , , et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299:2294–2303.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368–1377.
- , , , et al. Before‐after study of a standardized hospital order set for the management of septic shock. Crit Care Med. 2007;34:2707–2713.
- , , . Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Crit Care Med. 2008;36:296–327.
- , , , et al. Septic shock: an analysis of outcomes for patients with onset on hospital wards versus intensive care units. Crit Care Med. 1998;26:1020–1024.
- , , , , . Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity. J Gen Intern Med. 2003;18:77–83.
- , , , , , . Causes and effects of surgical delay in patients with hip fracture: a cohort study. Ann Intern Med. 2011;155:226–233.
- , , , . Comprehensive stroke centers overcome the weekend versus weekday gap in stroke treatment and mortality. Stroke. 2011;42:2403–2409.
- , , , , , . Hospital‐wide impact of a standardized order set for the management of bacteremic severe sepsis. Crit Care Med. 2009;37:819–824.
- , , , et al. Implementation of a real‐time computerized sepsis alert in non–intensive care unit patients. Crit Care Med. 2011;39:469–473.
- , , , et al. Toward a two‐tier clinical warning system for hospitalized patients. AMIA Annu Symp Proc. 2011;2011:511–519.
- , , , , , . Migrating toward a next‐generation clinical decision support application: the BJC HealthCare experience. AMIA Annu Symp Proc. 2007;344–348.
- , , , . Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system. J Hosp Med. 2011;7:224–230.
- , , , . Unplanned transfers to a medical intensive care unit: causes and relationship to preventable errors in care. J Hosp Med. 2011;6:68–72.
- , , , , , . Unplanned transfers to the intensive care unit: the role of the shock index. J Hosp Med. 2010;5:460–465.
- Institute for Healthcare Improvement. Early warning systems: the next level of rapid response. Available at: http://www.ihi.org/IHI/Programs/AudioAndWebPrograms/ExpeditionEarlyWarningSystemsTheNextLevelofRapidResponse.htmplayerwmp. Accessed April 6, 2011.
- , , , , , . Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7:388–395.
- , , . Early warning systems: the next level of rapid response. Nursing. 2012;42:38–44.
- , , , et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at‐risk patients on the ward. Intensive Care Med. 2007;33:667–679.
- , , , . Timing and teamwork—an observational pilot study of patients referred to a Rapid Response Team with the aim of identifying factors amenable to re‐design of a Rapid Response System. Resuscitation. 2012;83:782–787.
- , , , . Rapid response: a quality improvement conundrum. J Hosp Med. 2009;4:255–257.
- , , , et al. Introducing critical care outreach: a ward‐randomised trial of phased introduction in a general hospital. Intensive Care Med. 2004;30:1398–1404.
- . Out of our reach? Assessing the impact of introducing critical care outreach service. Anaesthesiology. 2003;58:882–885.
- , , . Effect of the critical care outreach team on patient survival to discharge from hospital and readmission to critical care: non‐randomised population based study. BMJ. 2003;327:1014–1016.
- , , . Reducing mortality and avoiding preventable ICU utilization: analysis of a successful rapid response program using APR DRGs [published online ahead of print March 10, 2010]. J Healthc Qual. doi: 10.1111/j.1945‐1474.2010.00084.x.
- , , , et al. Introduction of the medical emergency team (MET) system: a cluster‐randomised control trial. Lancet. 2005;365:2091–2097.
- , , , , , . The impact of the introduction of critical care outreach services in England: a multicentre interrupted time‐series analysis. Crit Care. 2007;11:R113.
- . Rapid response systems now established at 2,900 hospitals. Hospitalist News. March 2010;3:1.
- , , , , . Rapid response teams: a systematic review and meta‐analysis. Arch Intern Med. 2010;170:18–26.
- , , , et al. Outreach and early warning systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev. 2007;3:CD005529.
- , , . Rapid‐response teams. N Engl J Med. 2011;365:139–146.
- , , . Early warning systems. Hosp Chron. 2012;7(suppl 1):37–43.
- , , , et al. Grand challenges in clinical decision support. J Biomed Inform. 2008;41(2):387–392.
- , , , et al. Utility of commonly captured data from an EHR to identify hospitalized patients at risk for clinical deterioration. AMIA Annu Symp Proc. 2007;404–408.
- , , , , , . Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125(4)e763–e769.
- , , , . In‐hospital mortality after serious adverse events on medical and surgical nursing units: a mixed methods study [published online ahead of print July 24, 2012]. J Clin Nurs. doi: 10.1111/j.1365‐2702.2012.04154.x.
- , , , et al. Managing deteriorating patients: registered nurses' performance in a simulated setting. Open Nurs J. 2011;5:120–126.
- , , , et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299:2294–2303.
- , , , et al. Early goal‐directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345:1368–1377.
- , , , et al. Before‐after study of a standardized hospital order set for the management of septic shock. Crit Care Med. 2007;34:2707–2713.
- , , . Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Crit Care Med. 2008;36:296–327.
- , , , et al. Septic shock: an analysis of outcomes for patients with onset on hospital wards versus intensive care units. Crit Care Med. 1998;26:1020–1024.
- , , , , . Inpatient transfers to the intensive care unit: delays are associated with increased mortality and morbidity. J Gen Intern Med. 2003;18:77–83.
- , , , , , . Causes and effects of surgical delay in patients with hip fracture: a cohort study. Ann Intern Med. 2011;155:226–233.
- , , , . Comprehensive stroke centers overcome the weekend versus weekday gap in stroke treatment and mortality. Stroke. 2011;42:2403–2409.
- , , , , , . Hospital‐wide impact of a standardized order set for the management of bacteremic severe sepsis. Crit Care Med. 2009;37:819–824.
- , , , et al. Implementation of a real‐time computerized sepsis alert in non–intensive care unit patients. Crit Care Med. 2011;39:469–473.
- , , , et al. Toward a two‐tier clinical warning system for hospitalized patients. AMIA Annu Symp Proc. 2011;2011:511–519.
- , , , , , . Migrating toward a next‐generation clinical decision support application: the BJC HealthCare experience. AMIA Annu Symp Proc. 2007;344–348.
- , , , . Adverse outcomes associated with delayed intensive care unit transfers in an integrated healthcare system. J Hosp Med. 2011;7:224–230.
- , , , . Unplanned transfers to a medical intensive care unit: causes and relationship to preventable errors in care. J Hosp Med. 2011;6:68–72.
- , , , , , . Unplanned transfers to the intensive care unit: the role of the shock index. J Hosp Med. 2010;5:460–465.
- Institute for Healthcare Improvement. Early warning systems: the next level of rapid response. Available at: http://www.ihi.org/IHI/Programs/AudioAndWebPrograms/ExpeditionEarlyWarningSystemsTheNextLevelofRapidResponse.htmplayerwmp. Accessed April 6, 2011.
- , , , , , . Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med. 2012;7:388–395.
- , , . Early warning systems: the next level of rapid response. Nursing. 2012;42:38–44.
- , , , et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at‐risk patients on the ward. Intensive Care Med. 2007;33:667–679.
- , , , . Timing and teamwork—an observational pilot study of patients referred to a Rapid Response Team with the aim of identifying factors amenable to re‐design of a Rapid Response System. Resuscitation. 2012;83:782–787.
- , , , . Rapid response: a quality improvement conundrum. J Hosp Med. 2009;4:255–257.
- , , , et al. Introducing critical care outreach: a ward‐randomised trial of phased introduction in a general hospital. Intensive Care Med. 2004;30:1398–1404.
- . Out of our reach? Assessing the impact of introducing critical care outreach service. Anaesthesiology. 2003;58:882–885.
- , , . Effect of the critical care outreach team on patient survival to discharge from hospital and readmission to critical care: non‐randomised population based study. BMJ. 2003;327:1014–1016.
- , , . Reducing mortality and avoiding preventable ICU utilization: analysis of a successful rapid response program using APR DRGs [published online ahead of print March 10, 2010]. J Healthc Qual. doi: 10.1111/j.1945‐1474.2010.00084.x.
- , , , et al. Introduction of the medical emergency team (MET) system: a cluster‐randomised control trial. Lancet. 2005;365:2091–2097.
- , , , , , . The impact of the introduction of critical care outreach services in England: a multicentre interrupted time‐series analysis. Crit Care. 2007;11:R113.
- . Rapid response systems now established at 2,900 hospitals. Hospitalist News. March 2010;3:1.
- , , , , . Rapid response teams: a systematic review and meta‐analysis. Arch Intern Med. 2010;170:18–26.
- , , , et al. Outreach and early warning systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev. 2007;3:CD005529.
- , , . Rapid‐response teams. N Engl J Med. 2011;365:139–146.
- , , . Early warning systems. Hosp Chron. 2012;7(suppl 1):37–43.
- , , , et al. Grand challenges in clinical decision support. J Biomed Inform. 2008;41(2):387–392.
- , , , et al. Utility of commonly captured data from an EHR to identify hospitalized patients at risk for clinical deterioration. AMIA Annu Symp Proc. 2007;404–408.
- , , , , , . Sensitivity of the pediatric early warning score to identify patient deterioration. Pediatrics. 2010;125(4)e763–e769.
- , , , . In‐hospital mortality after serious adverse events on medical and surgical nursing units: a mixed methods study [published online ahead of print July 24, 2012]. J Clin Nurs. doi: 10.1111/j.1365‐2702.2012.04154.x.
- , , , et al. Managing deteriorating patients: registered nurses' performance in a simulated setting. Open Nurs J. 2011;5:120–126.
Copyright © 2013 Society of Hospital Medicine