Santiago Romero-Brufau, MD

Patients typically show signs and symptoms of deterioration hours to days prior to cardiorespiratory arrest.^1,2 The rate of inhospital cardiorespiratory arrest (CRA) requiring cardiopulmonary resuscitation is estimated to be 0.174 per bed per year in the United States.³ After CRA, survival to discharge is estimated to be as low as 18%.^3,4 Efforts to predict and prevent arrest could prove beneficial.^1,2

Rapid response systems (RRS) have been proposed as a means of identifying clinical deterioration and facilitating a timely response. These systems were designed to bring clinicians with critical care expertise to the bedside to prevent unnecessary deaths. They typically include an afferent limb (detects deteriorating patients), an efferent limb (responds to calls and acts to avoid further deterioration), and administrative and data analysis limbs.^5,6 Automatic provision of recommendations and computer-based systems are desirable components of the afferent limb of the detection system.⁶ Both are independent predictors of improved clinical practices for clinical decision support systems.⁷ However, the existing early warning scores (EWS) may not be ready for automation due to low positive predictive values (PPV) and sensitivities.⁸

It is possible that the low discriminatory accuracy of the published EWS may be secondary to the use of aggregate patient populations for derivation of scores. We hypothesized that these EWS perform differently in medical and in surgical subpopulations. Also, the EWS need to be tested in a time-dependent manner to serve as a realistic clinical support tool for hospitalized patients.

STUDY AIM

The aim of this study was to evaluate the differential performance of widely used EWS in medical vs surgical patients.

METHODS

Site

The study was conducted in an academic center with 2 hospitals in Southeastern Minnesota totaling approximately 1500 general care nonintensive care unit (ICU) beds. The Mayo Clinic Institutional Review Board approved the research proposal.

Subjects

Our retrospective cohort was comprised of all adult inpatients discharged from 2 academic hospitals between January 1, 2011 and December 31, 2011 who spent any time in a general care (non-ICU) unit. We excluded patients younger than 18 years, psychiatric or rehabilitation inpatients, those without research authorization, and patients admitted for research purposes.

Study patients were divided into medical and surgical cohorts. Hospitalizations were considered surgical if patients had surgery at any time during their hospital stay according to billing data. A trigger was an instance in which a patient met the conditions of a specific rule (score/vital sign exceeded the published/defined threshold).

A resuscitation call was defined as a call for cardiopulmonary resuscitation when a patient has a CRA.

An event was an occurrence of 1 of the following in a general care setting: unplanned transfer to the ICU, resuscitation call, or RRS activation.

The RRS activation criteria consisted of an “acute and persistent change” in any 1 or more of the following: oxygen saturations less than 90%, heart rate less than 40 or greater than 130 beats/minute, systolic blood pressure less than 90 mm Hg, or respiratory rate less than 10 or greater than 28 breaths/minute. The RRS activation requires health provider action; they are not electronically generated. Nurses and physicians may also activate the RRS if they are concerned about a patient, even if calling criteria are not met. This is in contrast to the EWS analyzed, which are aggregate composites of multiple parameters. However, whether or not a derangement in vital signs is considered an “acute and persistent change” still involves clinical judgment. Any movement from a general care bed to an ICU bed, or from a general care bed to a procedure area, and from there to an ICU, was considered unplanned. Transfers to the ICU directly from the emergency department or operating room (OR) were not considered as an unplanned transfer and were not included in the analyses.

Coverage time was the period observed for events after a rule was triggered. In this analysis, a coverage time of 24 hours was considered, with a 1-hour look-back. A trigger was counted as a true positive if an event occurred during the following 24 hours. The 1-hour look-back was included to take into account the nursing clinical process of prioritizing a call to the RRS followed by documentation of the altered vital signs that prompted the call.

An episode was the continuous time on the general care floor within a hospitalization, excluding times when a patient was in the OR or ICU. For example, if a patient was admitted to a general bed on a surgery floor, subsequently went to the OR, and then returned to the surgery floor, the 2 episodes were considered separate: the time on the floor before surgery, and the time on the floor after surgery.

Assessment of implementation of RRS in our hospitals showed a significant drop in the failure-to-rescue rate (issues considered related to delay or failure to identify or intervene appropriately when a patient was deteriorating, as identified through mortality review) and a decrease in non-ICU mortality.^9,10 This suggests that our current process captures many of the relevant episodes of acute deterioration when a rapid response team is needed and supports using RRS activation as outcomes.

 

Data Sources

We developed a time-stamped longitudinal database of patient data from the electronic health record, including vital signs, laboratory test results, demographics (age, sex), administrative data (including length of stay), comorbidities, resuscitation code status, location in hospital, and at the minute level throughout each patient’s hospital stay. Physiologically impossible values (eg, blood pressures of 1200 mm Hg) were considered entered in error and eliminated from the database. Time spent in the OR or ICU was excluded because RRS activation would not be applied in these already highly monitored areas. SAS Statistical software (SAS Institute Inc. Cary, North Carolina) was used for database creation.

We applied the current RRS calling criteria in our institution and calculated the Kirkland score,¹¹ along with some of the most widely used early warning scores:¹² Modified Early Warning System (MEWS),¹³ Standardized Early Warning Scoring System (SEWS),¹⁴ Global Modified Early Warning Score (GMEWS),¹⁵ Worthing physiologic scoring system,¹⁶ National Early Warning Score (NEWS),¹⁷ and VitaPAC Early Warning Score (ViEWS).¹⁸ Published thresholds for these scores were used to create rule triggers in the data. Once a trigger was created to calculate the number of false positives and true positives, all subsequent triggers were ignored until the end of the episode or until 24 hours elapsed. We calculated triggers in a rolling fashion throughout the episodes of care. The EWS score was updated every time a new parameter was entered into the analytical electronic health record, and the most recent value for each was used to calculate the score. SAS statistical software was used for calculation of scores and identification of outcomes.

For our analysis, events were treated as dependent variables, and triggers were independent variables. We calculated the score for each EWS to the minute level throughout our retrospective database. If the score for a specific EWS was higher than the published/recommended threshold for that EWS, an alert was considered to have been issued, and the patient was followed for 24 hours. If the patient had an event in the subsequent 24 hours, or 1 hour before (1-hour look-back), the alert was considered a true positive; if not, a false positive. Events that were not preceded by an alert were false negatives, and 24-hour intervals without either an alert or an event were considered true negatives. This simulation exercise was performed for each EWS in both subcohorts (medical and surgical). Clusters of RRS calls followed by transfers to the ICU within 3 hours were considered as a single adverse event (RRS calls, as it was the first event to occur) to avoid double counting. We have described how well this simulation methodology,⁸ correlates with results from prospective studies.¹⁹

Statistical Analysis

To calculate whether results were statistically significant for subgroups, a jackknife method of calculating variance²⁰ was used. The jackknife method calculates variance by repeating the calculations of the statistic leaving out 1 sample at a time. In our case, we repeated the calculation of sensitivity and PPV leaving out 1 patient at a time. Once the simulation method had been run and the false/true positives/negatives had been assigned, calculation of each metric (PPV and sensitivity) was repeated for n subsamples, each leaving out 1 patient. The variance was calculated and 2 Student t tests were performed for each EWS: 1 for PPV and another for sensitivity. SAS statistical software v 9.3 was used for the simulation analysis; R statistical software v 3.0.2 (The R Foundation, Vienna, Austria) was used for the calculation of the statistical significance of results. A univariable analysis was also performed to assess the sensitivity and PPVs for the published thresholds of the most common variables in each EWS: respiratory rate, systolic blood pressure, heart rate, temperature, and mental status as measured by the modified Richmond Agitation Sedation Score.²¹

RESULTS

The initial cohort included 60,020 hospitalizations, of which the following were excluded: 2751 because of a lack of appropriate research authorization; 6433 because the patients were younger than 18 years; 2129 as psychiatric admissions; 284 as rehabilitation admissions; 872 as research purposes-only admissions; and 1185 because the patient was never in a general care bed (eg, they were either admitted directly to the ICU, or they were admitted for an outpatient surgical procedure and spent time in the postanesthesia care unit).

Table 1 summarizes patient and trigger characteristics, overall and by subgroup. The final cohort included 75,240 total episodes in 46,366 hospitalizations, from 34,898 unique patients, of which 48.7% were male. There were 23,831 medical and 22,535 surgical hospitalizations. Median length of episode was 2 days both for medical and surgical patients. Median length of stay was 3 days, both for medical and for surgical patients.

There were 3332 events in total, of which 1709 were RRS calls, 185 were resuscitation calls, and 1438 were unscheduled transfers to the ICU. The rate of events was 4.67 events per 100 episodes in the aggregate adult population. There were 3.93 events per 100 episodes for surgical hospitalizations, and 5.86 events per 100 episodes for medical hospitalizations (P < .001). The number of CRAs in our cohort was 0.27 per 100 episodes, 0.128 per hospital bed per year, or 4.37 per 1000 hospital admissions, similar to other reported numbers in the literature.^{3, 22,23}

The total number of EWS triggers varied greatly between EWS rules, with the volume ranging during the study year from 1363 triggers with the GMEWS rule to 77,711 triggers with the ViEWS score.

All scores had PPVs less than 25%. As seen in Table 2 and shown graphically in the Figure, all scores performed better on medical patients (blue) than on surgical patients (yellow). The P value was < .0001 for both PPV and sensitivity. The Worthing score had the highest sensitivity (0.78 for medical and 0.68 for surgical) but a very low PPV (0.04 for medical and 0.03 for surgical), while GMEWS was the opposite: low sensitivity (0.10 and 0.07) but the highest PPV (0.22 and 0.18).

The results of the univariable analysis can be seen in Table 3. Most of the criteria performed better (higher sensitivity and PPV) as predictors in the medical hospitalizations than in the surgical hospitalizations.

 

DISCUSSION

We hypothesized that EWS may perform differently when applied to medical rather than surgical patients. Studies had not analyzed this in a time-dependent manner,^24-26 which limited the applicability of the results.⁸

All analyzed scores performed better in medical patients than in surgical patients (Figure). This could reflect a behavioral difference by the teams on surgical and medical floors in the decision to activate the RRS, or a bias of the clinicians who designed the scores (mostly nonsurgeons). The difference could also mean that physiological deteriorations are intrinsically different in patients who have undergone anesthesia and surgery. For example, in surgical patients, a bleeding episode is more likely to be the cause of their physiological deterioration, or the lingering effects of anesthesia could mask underlying deterioration. Such patients would benefit from scores where variables such as heart rate, blood pressure, or hemoglobin had more influence.

When comparing the different scores, it was much easier for a patient to meet the alerting score with the Worthing score than with GMEWS. In the Worthing score, a respiratory rate greater than 22 breaths per minute, or a systolic blood pressure less than 100 mm Hg, already meet alerting criteria. Similar vital signs result in 0 and 1 points (respectively) in GMEWS, far from its alerting score of 5. This reflects the intrinsic tradeoff of EWS: as the threshold for considering a patient “at risk” drops, not only does the number of true positives (and the sensitivity) increase, but also the number of false positives, thus lowering the PPV.

However, none of the scores analyzed were considered to perform well based on their PPV and sensitivity, particularly in the surgical subpopulation. Focusing on another metric, the area under the receiver operator curve can give misleadingly optimistic results.^24,27 However, the extremely low prevalence of acute physiological deterioration can produce low PPVs even when specificity seems acceptable, which is why it is important to evaluate PPV directly.²⁸

To use EWS effectively to activate RRS, they need to be combined with clinical judgment to avoid high levels of false alerts, particularly in surgical patients. It has been reported that RRS is activated only 30% of the time a patient meets RRS calling criteria.²⁹ While there may be cultural characteristics inhibiting the decision to call,³⁰ our study hints at another explanation: if RRS was activated every time a patient met calling criteria based on the scores analyzed, the number of RRS calls would be very high and difficult to manage. So health providers may be doing the right thing when “filtering” RRS calls and not applying the criteria strictly, but in conjunction with clinical judgment.

A limitation of any study like this is how to define “acute physiological deterioration.” We defined an event as recognized episodes of acute physiological deterioration that are signaled by escalations of care (eg, RRS, resuscitation calls, or transfers to an ICU) or unexpected death. By definition, our calculated PPV is affected by clinicians’ recognition of clinical deteriorations. This definition, common in the literature, has the limitation of potentially underestimating EWS’ performance by missing some events that are resolved by the primary care team without an escalation of care. However, we believe our interpretation is not unreasonable since the purpose of EWS is to trigger escalations of care in a timely fashion. Prospective studies could define an event in a way that is less affected by the clinicians’ judgment.

Regarding patient demographics, age was similar between the 2 groups (average, 58.2 years for medical vs 58.9 years for surgical), and there was only a small difference in gender ratios (45.1% male in the medical vs 51.4% in the surgical group). These differences are unlikely to have affected the results significantly, but unknown differences in demographics or other patient characteristics between groups may account for differences in score performance between surgical and medical patients.

Several of the EWS analyzed had overlapping trigger criteria with our own RRS activation criteria (although as single-parameter triggers and not as aggregate). To test how these potential biases could affect our results, we performed a post hoc sensitivity analysis eliminating calls to the RRS as an outcome (so using the alternative outcome of unexpected transfers to the ICU and resuscitation calls). The results are similar to those of our main analysis, with all analyzed scores having lower sensitivity and PPV in surgical hospitalizations when compared to medical hospitalizations.

Our study suggests that, to optimize detection of physiological deterioration events, EWS should try to take into account different patient types, with the most basic distinction being surgical vs medical. This tailoring will make EWS more complex, and less suited for paper-based calculation, but new electronic health records are increasingly able to incorporate decision support, and some EWS have been developed for electronic calculation only. Of particular interest in this regard is the score developed by Escobar et al,³¹ which groups patients into categories according to the reason for admission, and calculates a different subscore based on that category. While the score by Escobar et al. does not split patients based on medical or surgical status, a more general interpretation of our results suggests that a score may be more accurate if it classifies patients into subgroups with different subscores. This seems to be confirmed by the fact that the score by Escobar et al performs better than MEWS.²⁸ Unfortunately, the paper describing it does not provide enough detail to use it in our database.

A recent systematic review showed increasing evidence that RRS may be effective in reducing CRAs occurring in a non-ICU setting and, more important, overall inhospital mortality.³² While differing implementation strategies (eg, different length of the educational effort, changes in the frequency of vital signs monitoring) can impact the success of such an initiative, it has been speculated that the afferent limb (which often includes an EWS) might be the most critical part of the system.³³ Our results show that the most widely used EWS perform significantly worse on surgical patients, and suggest that a way to improve the accuracy of EWS would be to tailor the risk calculation to different patient subgroups (eg, medical and surgical patients). Plausible next steps would be to demonstrate that tailoring risk calculation to medical and surgical patients separately can improve risk predictions and accuracy of EWS.

 

Disclosure

The authors report no financial conflicts of interest.

Patients typically show signs and symptoms of deterioration hours to days prior to cardiorespiratory arrest.^1,2 The rate of inhospital cardiorespiratory arrest (CRA) requiring cardiopulmonary resuscitation is estimated to be 0.174 per bed per year in the United States.³ After CRA, survival to discharge is estimated to be as low as 18%.^3,4 Efforts to predict and prevent arrest could prove beneficial.^1,2

Rapid response systems (RRS) have been proposed as a means of identifying clinical deterioration and facilitating a timely response. These systems were designed to bring clinicians with critical care expertise to the bedside to prevent unnecessary deaths. They typically include an afferent limb (detects deteriorating patients), an efferent limb (responds to calls and acts to avoid further deterioration), and administrative and data analysis limbs.^5,6 Automatic provision of recommendations and computer-based systems are desirable components of the afferent limb of the detection system.⁶ Both are independent predictors of improved clinical practices for clinical decision support systems.⁷ However, the existing early warning scores (EWS) may not be ready for automation due to low positive predictive values (PPV) and sensitivities.⁸

It is possible that the low discriminatory accuracy of the published EWS may be secondary to the use of aggregate patient populations for derivation of scores. We hypothesized that these EWS perform differently in medical and in surgical subpopulations. Also, the EWS need to be tested in a time-dependent manner to serve as a realistic clinical support tool for hospitalized patients.

STUDY AIM

The aim of this study was to evaluate the differential performance of widely used EWS in medical vs surgical patients.

METHODS

Site

The study was conducted in an academic center with 2 hospitals in Southeastern Minnesota totaling approximately 1500 general care nonintensive care unit (ICU) beds. The Mayo Clinic Institutional Review Board approved the research proposal.

Subjects

Our retrospective cohort was comprised of all adult inpatients discharged from 2 academic hospitals between January 1, 2011 and December 31, 2011 who spent any time in a general care (non-ICU) unit. We excluded patients younger than 18 years, psychiatric or rehabilitation inpatients, those without research authorization, and patients admitted for research purposes.

Study patients were divided into medical and surgical cohorts. Hospitalizations were considered surgical if patients had surgery at any time during their hospital stay according to billing data. A trigger was an instance in which a patient met the conditions of a specific rule (score/vital sign exceeded the published/defined threshold).

A resuscitation call was defined as a call for cardiopulmonary resuscitation when a patient has a CRA.

An event was an occurrence of 1 of the following in a general care setting: unplanned transfer to the ICU, resuscitation call, or RRS activation.

The RRS activation criteria consisted of an “acute and persistent change” in any 1 or more of the following: oxygen saturations less than 90%, heart rate less than 40 or greater than 130 beats/minute, systolic blood pressure less than 90 mm Hg, or respiratory rate less than 10 or greater than 28 breaths/minute. The RRS activation requires health provider action; they are not electronically generated. Nurses and physicians may also activate the RRS if they are concerned about a patient, even if calling criteria are not met. This is in contrast to the EWS analyzed, which are aggregate composites of multiple parameters. However, whether or not a derangement in vital signs is considered an “acute and persistent change” still involves clinical judgment. Any movement from a general care bed to an ICU bed, or from a general care bed to a procedure area, and from there to an ICU, was considered unplanned. Transfers to the ICU directly from the emergency department or operating room (OR) were not considered as an unplanned transfer and were not included in the analyses.

Coverage time was the period observed for events after a rule was triggered. In this analysis, a coverage time of 24 hours was considered, with a 1-hour look-back. A trigger was counted as a true positive if an event occurred during the following 24 hours. The 1-hour look-back was included to take into account the nursing clinical process of prioritizing a call to the RRS followed by documentation of the altered vital signs that prompted the call.

An episode was the continuous time on the general care floor within a hospitalization, excluding times when a patient was in the OR or ICU. For example, if a patient was admitted to a general bed on a surgery floor, subsequently went to the OR, and then returned to the surgery floor, the 2 episodes were considered separate: the time on the floor before surgery, and the time on the floor after surgery.

Assessment of implementation of RRS in our hospitals showed a significant drop in the failure-to-rescue rate (issues considered related to delay or failure to identify or intervene appropriately when a patient was deteriorating, as identified through mortality review) and a decrease in non-ICU mortality.^9,10 This suggests that our current process captures many of the relevant episodes of acute deterioration when a rapid response team is needed and supports using RRS activation as outcomes.

 

Data Sources

We developed a time-stamped longitudinal database of patient data from the electronic health record, including vital signs, laboratory test results, demographics (age, sex), administrative data (including length of stay), comorbidities, resuscitation code status, location in hospital, and at the minute level throughout each patient’s hospital stay. Physiologically impossible values (eg, blood pressures of 1200 mm Hg) were considered entered in error and eliminated from the database. Time spent in the OR or ICU was excluded because RRS activation would not be applied in these already highly monitored areas. SAS Statistical software (SAS Institute Inc. Cary, North Carolina) was used for database creation.

We applied the current RRS calling criteria in our institution and calculated the Kirkland score,¹¹ along with some of the most widely used early warning scores:¹² Modified Early Warning System (MEWS),¹³ Standardized Early Warning Scoring System (SEWS),¹⁴ Global Modified Early Warning Score (GMEWS),¹⁵ Worthing physiologic scoring system,¹⁶ National Early Warning Score (NEWS),¹⁷ and VitaPAC Early Warning Score (ViEWS).¹⁸ Published thresholds for these scores were used to create rule triggers in the data. Once a trigger was created to calculate the number of false positives and true positives, all subsequent triggers were ignored until the end of the episode or until 24 hours elapsed. We calculated triggers in a rolling fashion throughout the episodes of care. The EWS score was updated every time a new parameter was entered into the analytical electronic health record, and the most recent value for each was used to calculate the score. SAS statistical software was used for calculation of scores and identification of outcomes.

For our analysis, events were treated as dependent variables, and triggers were independent variables. We calculated the score for each EWS to the minute level throughout our retrospective database. If the score for a specific EWS was higher than the published/recommended threshold for that EWS, an alert was considered to have been issued, and the patient was followed for 24 hours. If the patient had an event in the subsequent 24 hours, or 1 hour before (1-hour look-back), the alert was considered a true positive; if not, a false positive. Events that were not preceded by an alert were false negatives, and 24-hour intervals without either an alert or an event were considered true negatives. This simulation exercise was performed for each EWS in both subcohorts (medical and surgical). Clusters of RRS calls followed by transfers to the ICU within 3 hours were considered as a single adverse event (RRS calls, as it was the first event to occur) to avoid double counting. We have described how well this simulation methodology,⁸ correlates with results from prospective studies.¹⁹

Statistical Analysis

To calculate whether results were statistically significant for subgroups, a jackknife method of calculating variance²⁰ was used. The jackknife method calculates variance by repeating the calculations of the statistic leaving out 1 sample at a time. In our case, we repeated the calculation of sensitivity and PPV leaving out 1 patient at a time. Once the simulation method had been run and the false/true positives/negatives had been assigned, calculation of each metric (PPV and sensitivity) was repeated for n subsamples, each leaving out 1 patient. The variance was calculated and 2 Student t tests were performed for each EWS: 1 for PPV and another for sensitivity. SAS statistical software v 9.3 was used for the simulation analysis; R statistical software v 3.0.2 (The R Foundation, Vienna, Austria) was used for the calculation of the statistical significance of results. A univariable analysis was also performed to assess the sensitivity and PPVs for the published thresholds of the most common variables in each EWS: respiratory rate, systolic blood pressure, heart rate, temperature, and mental status as measured by the modified Richmond Agitation Sedation Score.²¹

RESULTS

The initial cohort included 60,020 hospitalizations, of which the following were excluded: 2751 because of a lack of appropriate research authorization; 6433 because the patients were younger than 18 years; 2129 as psychiatric admissions; 284 as rehabilitation admissions; 872 as research purposes-only admissions; and 1185 because the patient was never in a general care bed (eg, they were either admitted directly to the ICU, or they were admitted for an outpatient surgical procedure and spent time in the postanesthesia care unit).

Table 1 summarizes patient and trigger characteristics, overall and by subgroup. The final cohort included 75,240 total episodes in 46,366 hospitalizations, from 34,898 unique patients, of which 48.7% were male. There were 23,831 medical and 22,535 surgical hospitalizations. Median length of episode was 2 days both for medical and surgical patients. Median length of stay was 3 days, both for medical and for surgical patients.

There were 3332 events in total, of which 1709 were RRS calls, 185 were resuscitation calls, and 1438 were unscheduled transfers to the ICU. The rate of events was 4.67 events per 100 episodes in the aggregate adult population. There were 3.93 events per 100 episodes for surgical hospitalizations, and 5.86 events per 100 episodes for medical hospitalizations (P < .001). The number of CRAs in our cohort was 0.27 per 100 episodes, 0.128 per hospital bed per year, or 4.37 per 1000 hospital admissions, similar to other reported numbers in the literature.^{3, 22,23}

The total number of EWS triggers varied greatly between EWS rules, with the volume ranging during the study year from 1363 triggers with the GMEWS rule to 77,711 triggers with the ViEWS score.

All scores had PPVs less than 25%. As seen in Table 2 and shown graphically in the Figure, all scores performed better on medical patients (blue) than on surgical patients (yellow). The P value was < .0001 for both PPV and sensitivity. The Worthing score had the highest sensitivity (0.78 for medical and 0.68 for surgical) but a very low PPV (0.04 for medical and 0.03 for surgical), while GMEWS was the opposite: low sensitivity (0.10 and 0.07) but the highest PPV (0.22 and 0.18).

The results of the univariable analysis can be seen in Table 3. Most of the criteria performed better (higher sensitivity and PPV) as predictors in the medical hospitalizations than in the surgical hospitalizations.

 

DISCUSSION

We hypothesized that EWS may perform differently when applied to medical rather than surgical patients. Studies had not analyzed this in a time-dependent manner,^24-26 which limited the applicability of the results.⁸

All analyzed scores performed better in medical patients than in surgical patients (Figure). This could reflect a behavioral difference by the teams on surgical and medical floors in the decision to activate the RRS, or a bias of the clinicians who designed the scores (mostly nonsurgeons). The difference could also mean that physiological deteriorations are intrinsically different in patients who have undergone anesthesia and surgery. For example, in surgical patients, a bleeding episode is more likely to be the cause of their physiological deterioration, or the lingering effects of anesthesia could mask underlying deterioration. Such patients would benefit from scores where variables such as heart rate, blood pressure, or hemoglobin had more influence.

When comparing the different scores, it was much easier for a patient to meet the alerting score with the Worthing score than with GMEWS. In the Worthing score, a respiratory rate greater than 22 breaths per minute, or a systolic blood pressure less than 100 mm Hg, already meet alerting criteria. Similar vital signs result in 0 and 1 points (respectively) in GMEWS, far from its alerting score of 5. This reflects the intrinsic tradeoff of EWS: as the threshold for considering a patient “at risk” drops, not only does the number of true positives (and the sensitivity) increase, but also the number of false positives, thus lowering the PPV.

However, none of the scores analyzed were considered to perform well based on their PPV and sensitivity, particularly in the surgical subpopulation. Focusing on another metric, the area under the receiver operator curve can give misleadingly optimistic results.^24,27 However, the extremely low prevalence of acute physiological deterioration can produce low PPVs even when specificity seems acceptable, which is why it is important to evaluate PPV directly.²⁸

To use EWS effectively to activate RRS, they need to be combined with clinical judgment to avoid high levels of false alerts, particularly in surgical patients. It has been reported that RRS is activated only 30% of the time a patient meets RRS calling criteria.²⁹ While there may be cultural characteristics inhibiting the decision to call,³⁰ our study hints at another explanation: if RRS was activated every time a patient met calling criteria based on the scores analyzed, the number of RRS calls would be very high and difficult to manage. So health providers may be doing the right thing when “filtering” RRS calls and not applying the criteria strictly, but in conjunction with clinical judgment.

A limitation of any study like this is how to define “acute physiological deterioration.” We defined an event as recognized episodes of acute physiological deterioration that are signaled by escalations of care (eg, RRS, resuscitation calls, or transfers to an ICU) or unexpected death. By definition, our calculated PPV is affected by clinicians’ recognition of clinical deteriorations. This definition, common in the literature, has the limitation of potentially underestimating EWS’ performance by missing some events that are resolved by the primary care team without an escalation of care. However, we believe our interpretation is not unreasonable since the purpose of EWS is to trigger escalations of care in a timely fashion. Prospective studies could define an event in a way that is less affected by the clinicians’ judgment.

Regarding patient demographics, age was similar between the 2 groups (average, 58.2 years for medical vs 58.9 years for surgical), and there was only a small difference in gender ratios (45.1% male in the medical vs 51.4% in the surgical group). These differences are unlikely to have affected the results significantly, but unknown differences in demographics or other patient characteristics between groups may account for differences in score performance between surgical and medical patients.

Several of the EWS analyzed had overlapping trigger criteria with our own RRS activation criteria (although as single-parameter triggers and not as aggregate). To test how these potential biases could affect our results, we performed a post hoc sensitivity analysis eliminating calls to the RRS as an outcome (so using the alternative outcome of unexpected transfers to the ICU and resuscitation calls). The results are similar to those of our main analysis, with all analyzed scores having lower sensitivity and PPV in surgical hospitalizations when compared to medical hospitalizations.

Our study suggests that, to optimize detection of physiological deterioration events, EWS should try to take into account different patient types, with the most basic distinction being surgical vs medical. This tailoring will make EWS more complex, and less suited for paper-based calculation, but new electronic health records are increasingly able to incorporate decision support, and some EWS have been developed for electronic calculation only. Of particular interest in this regard is the score developed by Escobar et al,³¹ which groups patients into categories according to the reason for admission, and calculates a different subscore based on that category. While the score by Escobar et al. does not split patients based on medical or surgical status, a more general interpretation of our results suggests that a score may be more accurate if it classifies patients into subgroups with different subscores. This seems to be confirmed by the fact that the score by Escobar et al performs better than MEWS.²⁸ Unfortunately, the paper describing it does not provide enough detail to use it in our database.

A recent systematic review showed increasing evidence that RRS may be effective in reducing CRAs occurring in a non-ICU setting and, more important, overall inhospital mortality.³² While differing implementation strategies (eg, different length of the educational effort, changes in the frequency of vital signs monitoring) can impact the success of such an initiative, it has been speculated that the afferent limb (which often includes an EWS) might be the most critical part of the system.³³ Our results show that the most widely used EWS perform significantly worse on surgical patients, and suggest that a way to improve the accuracy of EWS would be to tailor the risk calculation to different patient subgroups (eg, medical and surgical patients). Plausible next steps would be to demonstrate that tailoring risk calculation to medical and surgical patients separately can improve risk predictions and accuracy of EWS.

 

Disclosure

The authors report no financial conflicts of interest.

Patients typically show signs and symptoms of deterioration hours to days prior to cardiorespiratory arrest.^1,2 The rate of inhospital cardiorespiratory arrest (CRA) requiring cardiopulmonary resuscitation is estimated to be 0.174 per bed per year in the United States.³ After CRA, survival to discharge is estimated to be as low as 18%.^3,4 Efforts to predict and prevent arrest could prove beneficial.^1,2

Rapid response systems (RRS) have been proposed as a means of identifying clinical deterioration and facilitating a timely response. These systems were designed to bring clinicians with critical care expertise to the bedside to prevent unnecessary deaths. They typically include an afferent limb (detects deteriorating patients), an efferent limb (responds to calls and acts to avoid further deterioration), and administrative and data analysis limbs.^5,6 Automatic provision of recommendations and computer-based systems are desirable components of the afferent limb of the detection system.⁶ Both are independent predictors of improved clinical practices for clinical decision support systems.⁷ However, the existing early warning scores (EWS) may not be ready for automation due to low positive predictive values (PPV) and sensitivities.⁸

It is possible that the low discriminatory accuracy of the published EWS may be secondary to the use of aggregate patient populations for derivation of scores. We hypothesized that these EWS perform differently in medical and in surgical subpopulations. Also, the EWS need to be tested in a time-dependent manner to serve as a realistic clinical support tool for hospitalized patients.

STUDY AIM

The aim of this study was to evaluate the differential performance of widely used EWS in medical vs surgical patients.

METHODS

Site

The study was conducted in an academic center with 2 hospitals in Southeastern Minnesota totaling approximately 1500 general care nonintensive care unit (ICU) beds. The Mayo Clinic Institutional Review Board approved the research proposal.

Subjects

Our retrospective cohort was comprised of all adult inpatients discharged from 2 academic hospitals between January 1, 2011 and December 31, 2011 who spent any time in a general care (non-ICU) unit. We excluded patients younger than 18 years, psychiatric or rehabilitation inpatients, those without research authorization, and patients admitted for research purposes.

Study patients were divided into medical and surgical cohorts. Hospitalizations were considered surgical if patients had surgery at any time during their hospital stay according to billing data. A trigger was an instance in which a patient met the conditions of a specific rule (score/vital sign exceeded the published/defined threshold).

A resuscitation call was defined as a call for cardiopulmonary resuscitation when a patient has a CRA.

An event was an occurrence of 1 of the following in a general care setting: unplanned transfer to the ICU, resuscitation call, or RRS activation.

The RRS activation criteria consisted of an “acute and persistent change” in any 1 or more of the following: oxygen saturations less than 90%, heart rate less than 40 or greater than 130 beats/minute, systolic blood pressure less than 90 mm Hg, or respiratory rate less than 10 or greater than 28 breaths/minute. The RRS activation requires health provider action; they are not electronically generated. Nurses and physicians may also activate the RRS if they are concerned about a patient, even if calling criteria are not met. This is in contrast to the EWS analyzed, which are aggregate composites of multiple parameters. However, whether or not a derangement in vital signs is considered an “acute and persistent change” still involves clinical judgment. Any movement from a general care bed to an ICU bed, or from a general care bed to a procedure area, and from there to an ICU, was considered unplanned. Transfers to the ICU directly from the emergency department or operating room (OR) were not considered as an unplanned transfer and were not included in the analyses.

Coverage time was the period observed for events after a rule was triggered. In this analysis, a coverage time of 24 hours was considered, with a 1-hour look-back. A trigger was counted as a true positive if an event occurred during the following 24 hours. The 1-hour look-back was included to take into account the nursing clinical process of prioritizing a call to the RRS followed by documentation of the altered vital signs that prompted the call.

An episode was the continuous time on the general care floor within a hospitalization, excluding times when a patient was in the OR or ICU. For example, if a patient was admitted to a general bed on a surgery floor, subsequently went to the OR, and then returned to the surgery floor, the 2 episodes were considered separate: the time on the floor before surgery, and the time on the floor after surgery.

Assessment of implementation of RRS in our hospitals showed a significant drop in the failure-to-rescue rate (issues considered related to delay or failure to identify or intervene appropriately when a patient was deteriorating, as identified through mortality review) and a decrease in non-ICU mortality.^9,10 This suggests that our current process captures many of the relevant episodes of acute deterioration when a rapid response team is needed and supports using RRS activation as outcomes.

 

Data Sources

We developed a time-stamped longitudinal database of patient data from the electronic health record, including vital signs, laboratory test results, demographics (age, sex), administrative data (including length of stay), comorbidities, resuscitation code status, location in hospital, and at the minute level throughout each patient’s hospital stay. Physiologically impossible values (eg, blood pressures of 1200 mm Hg) were considered entered in error and eliminated from the database. Time spent in the OR or ICU was excluded because RRS activation would not be applied in these already highly monitored areas. SAS Statistical software (SAS Institute Inc. Cary, North Carolina) was used for database creation.

We applied the current RRS calling criteria in our institution and calculated the Kirkland score,¹¹ along with some of the most widely used early warning scores:¹² Modified Early Warning System (MEWS),¹³ Standardized Early Warning Scoring System (SEWS),¹⁴ Global Modified Early Warning Score (GMEWS),¹⁵ Worthing physiologic scoring system,¹⁶ National Early Warning Score (NEWS),¹⁷ and VitaPAC Early Warning Score (ViEWS).¹⁸ Published thresholds for these scores were used to create rule triggers in the data. Once a trigger was created to calculate the number of false positives and true positives, all subsequent triggers were ignored until the end of the episode or until 24 hours elapsed. We calculated triggers in a rolling fashion throughout the episodes of care. The EWS score was updated every time a new parameter was entered into the analytical electronic health record, and the most recent value for each was used to calculate the score. SAS statistical software was used for calculation of scores and identification of outcomes.

For our analysis, events were treated as dependent variables, and triggers were independent variables. We calculated the score for each EWS to the minute level throughout our retrospective database. If the score for a specific EWS was higher than the published/recommended threshold for that EWS, an alert was considered to have been issued, and the patient was followed for 24 hours. If the patient had an event in the subsequent 24 hours, or 1 hour before (1-hour look-back), the alert was considered a true positive; if not, a false positive. Events that were not preceded by an alert were false negatives, and 24-hour intervals without either an alert or an event were considered true negatives. This simulation exercise was performed for each EWS in both subcohorts (medical and surgical). Clusters of RRS calls followed by transfers to the ICU within 3 hours were considered as a single adverse event (RRS calls, as it was the first event to occur) to avoid double counting. We have described how well this simulation methodology,⁸ correlates with results from prospective studies.¹⁹

Statistical Analysis

To calculate whether results were statistically significant for subgroups, a jackknife method of calculating variance²⁰ was used. The jackknife method calculates variance by repeating the calculations of the statistic leaving out 1 sample at a time. In our case, we repeated the calculation of sensitivity and PPV leaving out 1 patient at a time. Once the simulation method had been run and the false/true positives/negatives had been assigned, calculation of each metric (PPV and sensitivity) was repeated for n subsamples, each leaving out 1 patient. The variance was calculated and 2 Student t tests were performed for each EWS: 1 for PPV and another for sensitivity. SAS statistical software v 9.3 was used for the simulation analysis; R statistical software v 3.0.2 (The R Foundation, Vienna, Austria) was used for the calculation of the statistical significance of results. A univariable analysis was also performed to assess the sensitivity and PPVs for the published thresholds of the most common variables in each EWS: respiratory rate, systolic blood pressure, heart rate, temperature, and mental status as measured by the modified Richmond Agitation Sedation Score.²¹

RESULTS

The initial cohort included 60,020 hospitalizations, of which the following were excluded: 2751 because of a lack of appropriate research authorization; 6433 because the patients were younger than 18 years; 2129 as psychiatric admissions; 284 as rehabilitation admissions; 872 as research purposes-only admissions; and 1185 because the patient was never in a general care bed (eg, they were either admitted directly to the ICU, or they were admitted for an outpatient surgical procedure and spent time in the postanesthesia care unit).

Table 1 summarizes patient and trigger characteristics, overall and by subgroup. The final cohort included 75,240 total episodes in 46,366 hospitalizations, from 34,898 unique patients, of which 48.7% were male. There were 23,831 medical and 22,535 surgical hospitalizations. Median length of episode was 2 days both for medical and surgical patients. Median length of stay was 3 days, both for medical and for surgical patients.

There were 3332 events in total, of which 1709 were RRS calls, 185 were resuscitation calls, and 1438 were unscheduled transfers to the ICU. The rate of events was 4.67 events per 100 episodes in the aggregate adult population. There were 3.93 events per 100 episodes for surgical hospitalizations, and 5.86 events per 100 episodes for medical hospitalizations (P < .001). The number of CRAs in our cohort was 0.27 per 100 episodes, 0.128 per hospital bed per year, or 4.37 per 1000 hospital admissions, similar to other reported numbers in the literature.^{3, 22,23}

The total number of EWS triggers varied greatly between EWS rules, with the volume ranging during the study year from 1363 triggers with the GMEWS rule to 77,711 triggers with the ViEWS score.

All scores had PPVs less than 25%. As seen in Table 2 and shown graphically in the Figure, all scores performed better on medical patients (blue) than on surgical patients (yellow). The P value was < .0001 for both PPV and sensitivity. The Worthing score had the highest sensitivity (0.78 for medical and 0.68 for surgical) but a very low PPV (0.04 for medical and 0.03 for surgical), while GMEWS was the opposite: low sensitivity (0.10 and 0.07) but the highest PPV (0.22 and 0.18).

The results of the univariable analysis can be seen in Table 3. Most of the criteria performed better (higher sensitivity and PPV) as predictors in the medical hospitalizations than in the surgical hospitalizations.

 

DISCUSSION

We hypothesized that EWS may perform differently when applied to medical rather than surgical patients. Studies had not analyzed this in a time-dependent manner,^24-26 which limited the applicability of the results.⁸

All analyzed scores performed better in medical patients than in surgical patients (Figure). This could reflect a behavioral difference by the teams on surgical and medical floors in the decision to activate the RRS, or a bias of the clinicians who designed the scores (mostly nonsurgeons). The difference could also mean that physiological deteriorations are intrinsically different in patients who have undergone anesthesia and surgery. For example, in surgical patients, a bleeding episode is more likely to be the cause of their physiological deterioration, or the lingering effects of anesthesia could mask underlying deterioration. Such patients would benefit from scores where variables such as heart rate, blood pressure, or hemoglobin had more influence.

When comparing the different scores, it was much easier for a patient to meet the alerting score with the Worthing score than with GMEWS. In the Worthing score, a respiratory rate greater than 22 breaths per minute, or a systolic blood pressure less than 100 mm Hg, already meet alerting criteria. Similar vital signs result in 0 and 1 points (respectively) in GMEWS, far from its alerting score of 5. This reflects the intrinsic tradeoff of EWS: as the threshold for considering a patient “at risk” drops, not only does the number of true positives (and the sensitivity) increase, but also the number of false positives, thus lowering the PPV.

However, none of the scores analyzed were considered to perform well based on their PPV and sensitivity, particularly in the surgical subpopulation. Focusing on another metric, the area under the receiver operator curve can give misleadingly optimistic results.^24,27 However, the extremely low prevalence of acute physiological deterioration can produce low PPVs even when specificity seems acceptable, which is why it is important to evaluate PPV directly.²⁸

To use EWS effectively to activate RRS, they need to be combined with clinical judgment to avoid high levels of false alerts, particularly in surgical patients. It has been reported that RRS is activated only 30% of the time a patient meets RRS calling criteria.²⁹ While there may be cultural characteristics inhibiting the decision to call,³⁰ our study hints at another explanation: if RRS was activated every time a patient met calling criteria based on the scores analyzed, the number of RRS calls would be very high and difficult to manage. So health providers may be doing the right thing when “filtering” RRS calls and not applying the criteria strictly, but in conjunction with clinical judgment.

A limitation of any study like this is how to define “acute physiological deterioration.” We defined an event as recognized episodes of acute physiological deterioration that are signaled by escalations of care (eg, RRS, resuscitation calls, or transfers to an ICU) or unexpected death. By definition, our calculated PPV is affected by clinicians’ recognition of clinical deteriorations. This definition, common in the literature, has the limitation of potentially underestimating EWS’ performance by missing some events that are resolved by the primary care team without an escalation of care. However, we believe our interpretation is not unreasonable since the purpose of EWS is to trigger escalations of care in a timely fashion. Prospective studies could define an event in a way that is less affected by the clinicians’ judgment.

Regarding patient demographics, age was similar between the 2 groups (average, 58.2 years for medical vs 58.9 years for surgical), and there was only a small difference in gender ratios (45.1% male in the medical vs 51.4% in the surgical group). These differences are unlikely to have affected the results significantly, but unknown differences in demographics or other patient characteristics between groups may account for differences in score performance between surgical and medical patients.

Several of the EWS analyzed had overlapping trigger criteria with our own RRS activation criteria (although as single-parameter triggers and not as aggregate). To test how these potential biases could affect our results, we performed a post hoc sensitivity analysis eliminating calls to the RRS as an outcome (so using the alternative outcome of unexpected transfers to the ICU and resuscitation calls). The results are similar to those of our main analysis, with all analyzed scores having lower sensitivity and PPV in surgical hospitalizations when compared to medical hospitalizations.

Our study suggests that, to optimize detection of physiological deterioration events, EWS should try to take into account different patient types, with the most basic distinction being surgical vs medical. This tailoring will make EWS more complex, and less suited for paper-based calculation, but new electronic health records are increasingly able to incorporate decision support, and some EWS have been developed for electronic calculation only. Of particular interest in this regard is the score developed by Escobar et al,³¹ which groups patients into categories according to the reason for admission, and calculates a different subscore based on that category. While the score by Escobar et al. does not split patients based on medical or surgical status, a more general interpretation of our results suggests that a score may be more accurate if it classifies patients into subgroups with different subscores. This seems to be confirmed by the fact that the score by Escobar et al performs better than MEWS.²⁸ Unfortunately, the paper describing it does not provide enough detail to use it in our database.

A recent systematic review showed increasing evidence that RRS may be effective in reducing CRAs occurring in a non-ICU setting and, more important, overall inhospital mortality.³² While differing implementation strategies (eg, different length of the educational effort, changes in the frequency of vital signs monitoring) can impact the success of such an initiative, it has been speculated that the afferent limb (which often includes an EWS) might be the most critical part of the system.³³ Our results show that the most widely used EWS perform significantly worse on surgical patients, and suggest that a way to improve the accuracy of EWS would be to tailor the risk calculation to different patient subgroups (eg, medical and surgical patients). Plausible next steps would be to demonstrate that tailoring risk calculation to medical and surgical patients separately can improve risk predictions and accuracy of EWS.

 

Disclosure

The authors report no financial conflicts of interest.

From the Mayo Clinic Center for Innovation (Dr. Romero-Brufau) Department of Medicine (Drs. Manning, Borrud, Keller, Kashiwagi, Huddleston, and Croghan) Department of Health Sciences Research (Mr. Cha), Mayo Clinic, Rochester, MN.

 

Abstract

• Objective: To determine whether a score of 8 or greater on the I-MOVE, a bedside instrument that evaluates the need for assistance in turning, sitting, standing, transferring from bed to a chair, and ambulating, predicts lower risk for 30-day readmission or mortality.
• Design: Retrospective cohort study of patients discharged from 2003 to 2011 from a referral hospital in Southeastern Minnesota. We used a convenience sample of 426 inpatients who had at least one documented calculation of the I-MOVE score performed as part of the clinical process during the study.
• Results: Overall 30-day mortality rate, readmission rate, and rate of the combined death/readmission outcome were 6.1% (26 patients), 15% (64 patients) and 19.7% (84 patients), respectively. After controlling for confounding variables, an I-MOVE score ≥ 8 was a significant predictive factor for 30-day mortality (OR = 0.136, P < 0.01) but not 30-day readmission (OR = 1.143, P = 0.62) or the combined outcome death/readmission (OR = 0.682, P = 0.13).
• Conclusion: The clinical information provided by a patient's I-MOVE score before discharge does not provide information about readmission risk but may offer incremental information about 30-day mortality risk.

Risk factors for hospital 30-day readmission have been studied by Hasan et al [1], van Walraven et al [2], Allaudeen et al [3], and more recently, Donze et al [4]. Risk factors found to be associated with readmission include race, length of stay, and number of hospitalizations in the last 12 months. Additionally, patients identified “feeling unprepared for discharge” and “difficulty performing activities of daily living” as top issues contributing to readmission. The Affordable Care Act established the Value-Based Purchasing (VBP) model for defined hospital illnesses such as acute myocardial infarction, heart failure, and community acquired pneumonia. This has focused more attention on post-discharge 30-day mortality and readmissions as publicly reported metrics that in part determine the Centers for Medicare and Medicaid Services care reimbursement rates [5].

The Independent Mobility Validation Evaluation (I-MOVE) [6] is a bedside discharge tool for assessing the need for assistance in turning, sitting, standing, transferring from bed to a chair, and ambulating. It was designed as a discharge planning tool for clinicians, particularly in the situation of an elder hoping to return home alone. I-MOVE was shown to have face and content validity, and very good inter-observer reliability (r = 0.91) [6], and it requires only a few minutes to administer without any special apparatus. The Figure shows the tasks and the grading system, which awards points for higher function, from a score of 1 (requires assistance to turn in bed) to 12 (walks in the hallway independently). Recently, a modified (self-starting and self-reported) form of I-MOVE was employed in a cloud-based iPad application MyCare for 150 recovering heart surgery patients in whom higher (first day in general care unit after transfer from intensive care) I-MOVE scores were associated with shorter length of stay [7].

In our hospital, over 400 inpatients have been evaluated since 2004 using the I-MOVE scoring system in the course of their usual care. I-MOVE was most commonly employed by geriatricians in the division of hospital internal medicine, who collectively endorsed the tool in their practice meetings, especially for elderly patients returning to home alone whose mobility independence was uncertain.

Although it was initially designed to help clinicians understand the mobility independence of a patient before discharge, it may provide incremental value discerning risk of 30-day readmission and/or death. We therefore hypothesized that an I-MOVE score of less than 8 (not being able to transfer from a bed to a chair without assistance) would be a significant predictor of 30-day readmission and/or death.

Methods

Study Design

We performed a retrospective cohort study using a convenience sample including the patients in which the I-MOVE score had been calculated as part of the clinical process of care.

Setting and Participants

Participants were any inpatients discharged from the general medicine unit at Mayo Clinic Rochester from January 2003 to May 2011 who had at least one documented calculation of the I-MOVE score performed as part of the clinical process. Patients in the general medicine unit are adults not requiring subspecialty cardiovascular or neurology, coronary care unit, surgical, psychiatry, or rehabilitation. Patients were excluded if there was missing key outcome information or if they died during the hospitalization. For patients with more than one I-MOVE assessment, only the one closest to discharge was used. Data were abstracted from the electronic medical records between July and August 2011.

Variables

Outcome variables were 30-day readmission, 30-day mortality, and the combined outcome of mortality or readmission. We used the last I-MOVE score as a dichotomous variable with a cut-off of 8, which corresponds to the ability to transfer from bed to a chair unaided, for predicting the 2 outcomes. Only readmissions to the study hospital were captured. Deaths were identified from the electronic medical record. Mayo Clinic patient records are updated monthly with external reports of confirmed, actuarial records of deaths reported from public databases.

To control for possible confounding variables, we included the following covariates: age, gender, race/ethnicity, dates of admission and discharge, insurance (Medicare, Medicaid, self-pay, or private), marital status (currently married/not currently married), length of hospital stay, emergent admission, number of hospital admissions in the last 12 months, number of visits to the emergency department in the last 6 months and Charlson Index. All variables were abstracted from the electronic medical record.

Sample

A search was performed in the electronic medical record to find clinical documents (admission notes, progress notes, and hospital summaries) that mentioned the term “I-MOVE.” Manual review of the records was performed to confirm inclusion criteria.

Statistical Analysis

Separate analyses were performed for the 2 outcomes considered. First, a univariate analysis was performed with all covariates for variable selection. Variables that were significantly predictive with P < 0.1 were included in the multivariate model. Variables included in the first run of the multivariate model were excluded from the final multivariate model if they were not independently significant with P < 0.05. The I-MOVE variable was then added to that model to check its predictive power beyond that of the included covariates.

 

Results

Patient Characteristics

The initial search returned 452 records, of which a total of 26 (5.7%) were excluded for either being duplicates or not meeting the inclusion/exclusion criteria. Patient characteristics are shown in Table 1.

For the final dataset of 426 patients, 30-day mortality rate, readmission rate, and rate of the combined death/readmission outcome were 6.1% (26 patients), 15% (64 patients) and 19.7% (84 patients), respectively. A total of 6 patients were readmitted and died within 30 days after the initial discharge. The number of patients that had an I-MOVE score greater than or equal to 8 was 232 (54.4%). Table 2 presents the mean, standard deviation, and median I-MOVE score by patient discharge destination. Patients discharged home had an average I-MOVE score of 11.98, versus 7.24 for patients discharge to a skilled nursing facility (P = 0.2).

 

Analysis

Table 3 presents the odds ratios and coefficient estimate of the models. In the univariate analysis, an I-MOVE score greater than or equal to 8 was significantly correlated with 30-day mortality (P < 0.001), and the combined outcome (P = 0.044) but not with 30-day readmission (P = 0.76). After controlling for confounding variables, I-MOVE greater than or equal to 8 was a significant predictor of 30-day mortality (P < 0.01) but not 30-day readmission (P = 0.75) 

or the combined outcome death/readmission (P = 0.17).

Discussion

An I-MOVE score of less than 8 (inability to transfer from bed to a chair unassisted) is a statistically significant predictor of 30-day post-discharge mortality but not readmission or the combined outcome of death/readmission.

A recent review that evaluated published models that attempted to predict readmissions concluded that most current readmission risk prediction models designed for either comparative or clinical purposes perform poorly and that efforts are needed to improve their performance as use becomes more widespread [8]. Health care providers’ ability to predict which patients would be readmitted within 30 days was also shown by a recent study to be very poor, with C-statistics around 0.60 [9]. This inability of both experts and statistical methods to accurately predict readmissions may reflect some inherent randomness or unpredictability of readmissions, or the fact that a paradigm shift is still needed in the identification of the most important risk factors for readmissions. Along the same line, a recent evaluation of interventions aimed at reducing readmissions found that none of those identified in the literature managed to consistently reduce readmission rates long-term [10]. In addition, hospitals with greater adherence to recommended care processes did not achieve meaningfully better 30-day hospital readmission rates compared to those with lower levels of performance.

Conceptually, readmissions are an example of what is called “complexity science,” where many agents or factors (including the patient’s underlying illness, quality of care delivered, continuity and coordination of care, and resources available in the patient’s environment) and their interactions all play a role in the outcome [11,12]. Since I-MOVE primarily evaluates the physical capacity of the patient, and not any of the other variables that strongly affect readmission, it is perhaps not surprising that it did not predict readmission. It can be argued, on the other hand, that short term (30-day) mortality is more dependent on the patient’s physical and functional status [13] and so more likely to correlate with a measure such as I-MOVE. Inouye et al [13] found that pre-hospital, self-reported need for assistance in 7 basic “activities of daily living” (among which are transfers and ambulation) correlated with 90-day, and 2-year, post-hospital mortality.

The study has the advantage of a relatively large sample size, and the fact that the I-MOVE score was assessed before discharge eliminates the possibility of assessor bias. However, it has some limitations. We used a convenience sample, which may have introduced selection bias. Although we have no data on how providers selected patients for I-MOVE assessment, it would be reasonable to assume that patients were selected from among those whose activity level was, in terms of independence, doubtful or uncertain. That is, those who were not clearly vigorous (up and walking easily), nor clearly debilitated (in need of great assistance) may have been more likely to be assessed using I-MOVE. A more systematic selection of subjects might increase or decrease the predictive performance of the I-MOVE assessment. In addition, although we attempted to control for potential confounders, it is possible that additional confounders were left out of our analysis.

In summary, although the predictive performance of I-MOVE still needs to be confirmed by prospective studies with a comprehensive selection of subjects, the I-MOVE score at discharge appears to be associated with 30-day post-discharge mortality.

 

Acknowledgments: We thank the Department of Medicine’s clinical research office for their help in study design, data acquisition, and statistical analysis. 

Corresponding author: Santiago Romero-Brufau, MD, Mayo Clinic Center for Innovation,  200 First St. SW, Rochester, MN 55905,  romerobrufau.santiago@mayo.edu.

Funding/support: This publication was supported by grant number UL1 TR000135 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.  

 

From the Mayo Clinic Center for Innovation (Dr. Romero-Brufau) Department of Medicine (Drs. Manning, Borrud, Keller, Kashiwagi, Huddleston, and Croghan) Department of Health Sciences Research (Mr. Cha), Mayo Clinic, Rochester, MN.

 

Abstract

• Objective: To determine whether a score of 8 or greater on the I-MOVE, a bedside instrument that evaluates the need for assistance in turning, sitting, standing, transferring from bed to a chair, and ambulating, predicts lower risk for 30-day readmission or mortality.
• Design: Retrospective cohort study of patients discharged from 2003 to 2011 from a referral hospital in Southeastern Minnesota. We used a convenience sample of 426 inpatients who had at least one documented calculation of the I-MOVE score performed as part of the clinical process during the study.
• Results: Overall 30-day mortality rate, readmission rate, and rate of the combined death/readmission outcome were 6.1% (26 patients), 15% (64 patients) and 19.7% (84 patients), respectively. After controlling for confounding variables, an I-MOVE score ≥ 8 was a significant predictive factor for 30-day mortality (OR = 0.136, P < 0.01) but not 30-day readmission (OR = 1.143, P = 0.62) or the combined outcome death/readmission (OR = 0.682, P = 0.13).
• Conclusion: The clinical information provided by a patient's I-MOVE score before discharge does not provide information about readmission risk but may offer incremental information about 30-day mortality risk.

Risk factors for hospital 30-day readmission have been studied by Hasan et al [1], van Walraven et al [2], Allaudeen et al [3], and more recently, Donze et al [4]. Risk factors found to be associated with readmission include race, length of stay, and number of hospitalizations in the last 12 months. Additionally, patients identified “feeling unprepared for discharge” and “difficulty performing activities of daily living” as top issues contributing to readmission. The Affordable Care Act established the Value-Based Purchasing (VBP) model for defined hospital illnesses such as acute myocardial infarction, heart failure, and community acquired pneumonia. This has focused more attention on post-discharge 30-day mortality and readmissions as publicly reported metrics that in part determine the Centers for Medicare and Medicaid Services care reimbursement rates [5].

The Independent Mobility Validation Evaluation (I-MOVE) [6] is a bedside discharge tool for assessing the need for assistance in turning, sitting, standing, transferring from bed to a chair, and ambulating. It was designed as a discharge planning tool for clinicians, particularly in the situation of an elder hoping to return home alone. I-MOVE was shown to have face and content validity, and very good inter-observer reliability (r = 0.91) [6], and it requires only a few minutes to administer without any special apparatus. The Figure shows the tasks and the grading system, which awards points for higher function, from a score of 1 (requires assistance to turn in bed) to 12 (walks in the hallway independently). Recently, a modified (self-starting and self-reported) form of I-MOVE was employed in a cloud-based iPad application MyCare for 150 recovering heart surgery patients in whom higher (first day in general care unit after transfer from intensive care) I-MOVE scores were associated with shorter length of stay [7].

In our hospital, over 400 inpatients have been evaluated since 2004 using the I-MOVE scoring system in the course of their usual care. I-MOVE was most commonly employed by geriatricians in the division of hospital internal medicine, who collectively endorsed the tool in their practice meetings, especially for elderly patients returning to home alone whose mobility independence was uncertain.

Although it was initially designed to help clinicians understand the mobility independence of a patient before discharge, it may provide incremental value discerning risk of 30-day readmission and/or death. We therefore hypothesized that an I-MOVE score of less than 8 (not being able to transfer from a bed to a chair without assistance) would be a significant predictor of 30-day readmission and/or death.

Methods

Study Design

We performed a retrospective cohort study using a convenience sample including the patients in which the I-MOVE score had been calculated as part of the clinical process of care.

Setting and Participants

Participants were any inpatients discharged from the general medicine unit at Mayo Clinic Rochester from January 2003 to May 2011 who had at least one documented calculation of the I-MOVE score performed as part of the clinical process. Patients in the general medicine unit are adults not requiring subspecialty cardiovascular or neurology, coronary care unit, surgical, psychiatry, or rehabilitation. Patients were excluded if there was missing key outcome information or if they died during the hospitalization. For patients with more than one I-MOVE assessment, only the one closest to discharge was used. Data were abstracted from the electronic medical records between July and August 2011.

Variables

Outcome variables were 30-day readmission, 30-day mortality, and the combined outcome of mortality or readmission. We used the last I-MOVE score as a dichotomous variable with a cut-off of 8, which corresponds to the ability to transfer from bed to a chair unaided, for predicting the 2 outcomes. Only readmissions to the study hospital were captured. Deaths were identified from the electronic medical record. Mayo Clinic patient records are updated monthly with external reports of confirmed, actuarial records of deaths reported from public databases.

To control for possible confounding variables, we included the following covariates: age, gender, race/ethnicity, dates of admission and discharge, insurance (Medicare, Medicaid, self-pay, or private), marital status (currently married/not currently married), length of hospital stay, emergent admission, number of hospital admissions in the last 12 months, number of visits to the emergency department in the last 6 months and Charlson Index. All variables were abstracted from the electronic medical record.

Sample

A search was performed in the electronic medical record to find clinical documents (admission notes, progress notes, and hospital summaries) that mentioned the term “I-MOVE.” Manual review of the records was performed to confirm inclusion criteria.

Statistical Analysis

Separate analyses were performed for the 2 outcomes considered. First, a univariate analysis was performed with all covariates for variable selection. Variables that were significantly predictive with P < 0.1 were included in the multivariate model. Variables included in the first run of the multivariate model were excluded from the final multivariate model if they were not independently significant with P < 0.05. The I-MOVE variable was then added to that model to check its predictive power beyond that of the included covariates.

 

Results

Patient Characteristics

The initial search returned 452 records, of which a total of 26 (5.7%) were excluded for either being duplicates or not meeting the inclusion/exclusion criteria. Patient characteristics are shown in Table 1.

For the final dataset of 426 patients, 30-day mortality rate, readmission rate, and rate of the combined death/readmission outcome were 6.1% (26 patients), 15% (64 patients) and 19.7% (84 patients), respectively. A total of 6 patients were readmitted and died within 30 days after the initial discharge. The number of patients that had an I-MOVE score greater than or equal to 8 was 232 (54.4%). Table 2 presents the mean, standard deviation, and median I-MOVE score by patient discharge destination. Patients discharged home had an average I-MOVE score of 11.98, versus 7.24 for patients discharge to a skilled nursing facility (P = 0.2).

 

Analysis

Table 3 presents the odds ratios and coefficient estimate of the models. In the univariate analysis, an I-MOVE score greater than or equal to 8 was significantly correlated with 30-day mortality (P < 0.001), and the combined outcome (P = 0.044) but not with 30-day readmission (P = 0.76). After controlling for confounding variables, I-MOVE greater than or equal to 8 was a significant predictor of 30-day mortality (P < 0.01) but not 30-day readmission (P = 0.75) 

or the combined outcome death/readmission (P = 0.17).

Discussion

An I-MOVE score of less than 8 (inability to transfer from bed to a chair unassisted) is a statistically significant predictor of 30-day post-discharge mortality but not readmission or the combined outcome of death/readmission.

A recent review that evaluated published models that attempted to predict readmissions concluded that most current readmission risk prediction models designed for either comparative or clinical purposes perform poorly and that efforts are needed to improve their performance as use becomes more widespread [8]. Health care providers’ ability to predict which patients would be readmitted within 30 days was also shown by a recent study to be very poor, with C-statistics around 0.60 [9]. This inability of both experts and statistical methods to accurately predict readmissions may reflect some inherent randomness or unpredictability of readmissions, or the fact that a paradigm shift is still needed in the identification of the most important risk factors for readmissions. Along the same line, a recent evaluation of interventions aimed at reducing readmissions found that none of those identified in the literature managed to consistently reduce readmission rates long-term [10]. In addition, hospitals with greater adherence to recommended care processes did not achieve meaningfully better 30-day hospital readmission rates compared to those with lower levels of performance.

Conceptually, readmissions are an example of what is called “complexity science,” where many agents or factors (including the patient’s underlying illness, quality of care delivered, continuity and coordination of care, and resources available in the patient’s environment) and their interactions all play a role in the outcome [11,12]. Since I-MOVE primarily evaluates the physical capacity of the patient, and not any of the other variables that strongly affect readmission, it is perhaps not surprising that it did not predict readmission. It can be argued, on the other hand, that short term (30-day) mortality is more dependent on the patient’s physical and functional status [13] and so more likely to correlate with a measure such as I-MOVE. Inouye et al [13] found that pre-hospital, self-reported need for assistance in 7 basic “activities of daily living” (among which are transfers and ambulation) correlated with 90-day, and 2-year, post-hospital mortality.

The study has the advantage of a relatively large sample size, and the fact that the I-MOVE score was assessed before discharge eliminates the possibility of assessor bias. However, it has some limitations. We used a convenience sample, which may have introduced selection bias. Although we have no data on how providers selected patients for I-MOVE assessment, it would be reasonable to assume that patients were selected from among those whose activity level was, in terms of independence, doubtful or uncertain. That is, those who were not clearly vigorous (up and walking easily), nor clearly debilitated (in need of great assistance) may have been more likely to be assessed using I-MOVE. A more systematic selection of subjects might increase or decrease the predictive performance of the I-MOVE assessment. In addition, although we attempted to control for potential confounders, it is possible that additional confounders were left out of our analysis.

In summary, although the predictive performance of I-MOVE still needs to be confirmed by prospective studies with a comprehensive selection of subjects, the I-MOVE score at discharge appears to be associated with 30-day post-discharge mortality.

 

Acknowledgments: We thank the Department of Medicine’s clinical research office for their help in study design, data acquisition, and statistical analysis. 

Corresponding author: Santiago Romero-Brufau, MD, Mayo Clinic Center for Innovation,  200 First St. SW, Rochester, MN 55905,  romerobrufau.santiago@mayo.edu.

Funding/support: This publication was supported by grant number UL1 TR000135 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.  

 

From the Mayo Clinic Center for Innovation (Dr. Romero-Brufau) Department of Medicine (Drs. Manning, Borrud, Keller, Kashiwagi, Huddleston, and Croghan) Department of Health Sciences Research (Mr. Cha), Mayo Clinic, Rochester, MN.

 

Abstract

• Objective: To determine whether a score of 8 or greater on the I-MOVE, a bedside instrument that evaluates the need for assistance in turning, sitting, standing, transferring from bed to a chair, and ambulating, predicts lower risk for 30-day readmission or mortality.
• Design: Retrospective cohort study of patients discharged from 2003 to 2011 from a referral hospital in Southeastern Minnesota. We used a convenience sample of 426 inpatients who had at least one documented calculation of the I-MOVE score performed as part of the clinical process during the study.
• Results: Overall 30-day mortality rate, readmission rate, and rate of the combined death/readmission outcome were 6.1% (26 patients), 15% (64 patients) and 19.7% (84 patients), respectively. After controlling for confounding variables, an I-MOVE score ≥ 8 was a significant predictive factor for 30-day mortality (OR = 0.136, P < 0.01) but not 30-day readmission (OR = 1.143, P = 0.62) or the combined outcome death/readmission (OR = 0.682, P = 0.13).
• Conclusion: The clinical information provided by a patient's I-MOVE score before discharge does not provide information about readmission risk but may offer incremental information about 30-day mortality risk.

Risk factors for hospital 30-day readmission have been studied by Hasan et al [1], van Walraven et al [2], Allaudeen et al [3], and more recently, Donze et al [4]. Risk factors found to be associated with readmission include race, length of stay, and number of hospitalizations in the last 12 months. Additionally, patients identified “feeling unprepared for discharge” and “difficulty performing activities of daily living” as top issues contributing to readmission. The Affordable Care Act established the Value-Based Purchasing (VBP) model for defined hospital illnesses such as acute myocardial infarction, heart failure, and community acquired pneumonia. This has focused more attention on post-discharge 30-day mortality and readmissions as publicly reported metrics that in part determine the Centers for Medicare and Medicaid Services care reimbursement rates [5].

The Independent Mobility Validation Evaluation (I-MOVE) [6] is a bedside discharge tool for assessing the need for assistance in turning, sitting, standing, transferring from bed to a chair, and ambulating. It was designed as a discharge planning tool for clinicians, particularly in the situation of an elder hoping to return home alone. I-MOVE was shown to have face and content validity, and very good inter-observer reliability (r = 0.91) [6], and it requires only a few minutes to administer without any special apparatus. The Figure shows the tasks and the grading system, which awards points for higher function, from a score of 1 (requires assistance to turn in bed) to 12 (walks in the hallway independently). Recently, a modified (self-starting and self-reported) form of I-MOVE was employed in a cloud-based iPad application MyCare for 150 recovering heart surgery patients in whom higher (first day in general care unit after transfer from intensive care) I-MOVE scores were associated with shorter length of stay [7].

In our hospital, over 400 inpatients have been evaluated since 2004 using the I-MOVE scoring system in the course of their usual care. I-MOVE was most commonly employed by geriatricians in the division of hospital internal medicine, who collectively endorsed the tool in their practice meetings, especially for elderly patients returning to home alone whose mobility independence was uncertain.

Although it was initially designed to help clinicians understand the mobility independence of a patient before discharge, it may provide incremental value discerning risk of 30-day readmission and/or death. We therefore hypothesized that an I-MOVE score of less than 8 (not being able to transfer from a bed to a chair without assistance) would be a significant predictor of 30-day readmission and/or death.

Methods

Study Design

We performed a retrospective cohort study using a convenience sample including the patients in which the I-MOVE score had been calculated as part of the clinical process of care.

Setting and Participants

Participants were any inpatients discharged from the general medicine unit at Mayo Clinic Rochester from January 2003 to May 2011 who had at least one documented calculation of the I-MOVE score performed as part of the clinical process. Patients in the general medicine unit are adults not requiring subspecialty cardiovascular or neurology, coronary care unit, surgical, psychiatry, or rehabilitation. Patients were excluded if there was missing key outcome information or if they died during the hospitalization. For patients with more than one I-MOVE assessment, only the one closest to discharge was used. Data were abstracted from the electronic medical records between July and August 2011.

Variables

Outcome variables were 30-day readmission, 30-day mortality, and the combined outcome of mortality or readmission. We used the last I-MOVE score as a dichotomous variable with a cut-off of 8, which corresponds to the ability to transfer from bed to a chair unaided, for predicting the 2 outcomes. Only readmissions to the study hospital were captured. Deaths were identified from the electronic medical record. Mayo Clinic patient records are updated monthly with external reports of confirmed, actuarial records of deaths reported from public databases.

To control for possible confounding variables, we included the following covariates: age, gender, race/ethnicity, dates of admission and discharge, insurance (Medicare, Medicaid, self-pay, or private), marital status (currently married/not currently married), length of hospital stay, emergent admission, number of hospital admissions in the last 12 months, number of visits to the emergency department in the last 6 months and Charlson Index. All variables were abstracted from the electronic medical record.

Sample

A search was performed in the electronic medical record to find clinical documents (admission notes, progress notes, and hospital summaries) that mentioned the term “I-MOVE.” Manual review of the records was performed to confirm inclusion criteria.

Statistical Analysis

Separate analyses were performed for the 2 outcomes considered. First, a univariate analysis was performed with all covariates for variable selection. Variables that were significantly predictive with P < 0.1 were included in the multivariate model. Variables included in the first run of the multivariate model were excluded from the final multivariate model if they were not independently significant with P < 0.05. The I-MOVE variable was then added to that model to check its predictive power beyond that of the included covariates.

 

Results

Patient Characteristics

The initial search returned 452 records, of which a total of 26 (5.7%) were excluded for either being duplicates or not meeting the inclusion/exclusion criteria. Patient characteristics are shown in Table 1.

For the final dataset of 426 patients, 30-day mortality rate, readmission rate, and rate of the combined death/readmission outcome were 6.1% (26 patients), 15% (64 patients) and 19.7% (84 patients), respectively. A total of 6 patients were readmitted and died within 30 days after the initial discharge. The number of patients that had an I-MOVE score greater than or equal to 8 was 232 (54.4%). Table 2 presents the mean, standard deviation, and median I-MOVE score by patient discharge destination. Patients discharged home had an average I-MOVE score of 11.98, versus 7.24 for patients discharge to a skilled nursing facility (P = 0.2).

 

Analysis

Table 3 presents the odds ratios and coefficient estimate of the models. In the univariate analysis, an I-MOVE score greater than or equal to 8 was significantly correlated with 30-day mortality (P < 0.001), and the combined outcome (P = 0.044) but not with 30-day readmission (P = 0.76). After controlling for confounding variables, I-MOVE greater than or equal to 8 was a significant predictor of 30-day mortality (P < 0.01) but not 30-day readmission (P = 0.75) 

or the combined outcome death/readmission (P = 0.17).

Discussion

An I-MOVE score of less than 8 (inability to transfer from bed to a chair unassisted) is a statistically significant predictor of 30-day post-discharge mortality but not readmission or the combined outcome of death/readmission.

A recent review that evaluated published models that attempted to predict readmissions concluded that most current readmission risk prediction models designed for either comparative or clinical purposes perform poorly and that efforts are needed to improve their performance as use becomes more widespread [8]. Health care providers’ ability to predict which patients would be readmitted within 30 days was also shown by a recent study to be very poor, with C-statistics around 0.60 [9]. This inability of both experts and statistical methods to accurately predict readmissions may reflect some inherent randomness or unpredictability of readmissions, or the fact that a paradigm shift is still needed in the identification of the most important risk factors for readmissions. Along the same line, a recent evaluation of interventions aimed at reducing readmissions found that none of those identified in the literature managed to consistently reduce readmission rates long-term [10]. In addition, hospitals with greater adherence to recommended care processes did not achieve meaningfully better 30-day hospital readmission rates compared to those with lower levels of performance.

Conceptually, readmissions are an example of what is called “complexity science,” where many agents or factors (including the patient’s underlying illness, quality of care delivered, continuity and coordination of care, and resources available in the patient’s environment) and their interactions all play a role in the outcome [11,12]. Since I-MOVE primarily evaluates the physical capacity of the patient, and not any of the other variables that strongly affect readmission, it is perhaps not surprising that it did not predict readmission. It can be argued, on the other hand, that short term (30-day) mortality is more dependent on the patient’s physical and functional status [13] and so more likely to correlate with a measure such as I-MOVE. Inouye et al [13] found that pre-hospital, self-reported need for assistance in 7 basic “activities of daily living” (among which are transfers and ambulation) correlated with 90-day, and 2-year, post-hospital mortality.

The study has the advantage of a relatively large sample size, and the fact that the I-MOVE score was assessed before discharge eliminates the possibility of assessor bias. However, it has some limitations. We used a convenience sample, which may have introduced selection bias. Although we have no data on how providers selected patients for I-MOVE assessment, it would be reasonable to assume that patients were selected from among those whose activity level was, in terms of independence, doubtful or uncertain. That is, those who were not clearly vigorous (up and walking easily), nor clearly debilitated (in need of great assistance) may have been more likely to be assessed using I-MOVE. A more systematic selection of subjects might increase or decrease the predictive performance of the I-MOVE assessment. In addition, although we attempted to control for potential confounders, it is possible that additional confounders were left out of our analysis.

In summary, although the predictive performance of I-MOVE still needs to be confirmed by prospective studies with a comprehensive selection of subjects, the I-MOVE score at discharge appears to be associated with 30-day post-discharge mortality.

 

Acknowledgments: We thank the Department of Medicine’s clinical research office for their help in study design, data acquisition, and statistical analysis. 

Corresponding author: Santiago Romero-Brufau, MD, Mayo Clinic Center for Innovation,  200 First St. SW, Rochester, MN 55905,  romerobrufau.santiago@mayo.edu.

Funding/support: This publication was supported by grant number UL1 TR000135 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.