Mark L. Metersky, MD

Warfarin is 1 of the most common causes of adverse drug events, with hospitalized patients being particularly at risk compared to outpatients.[1] Despite the availability of new oral anticoagulants (NOACs), physicians commonly prescribe warfarin to hospitalized patients,[2] likely in part due to the greater difficulty in reversing NOACs compared to warfarin. Furthermore, uptake of the NOACs is likely to be slow in resource‐poor countries due to the lower cost of warfarin.[3] However, the narrow therapeutic index, frequent drug‐drug interactions, and patient variability in metabolism of warfarin makes management challenging.[4] Thus, warfarin remains a significant cause of adverse events in hospitalized patients, occurring in approximately 3% to 8% of exposed patients, depending on underlying condition.[2, 5]

An elevated international normalized ratio (INR) is a strong predictor of drug‐associated adverse events (patient harm). In a study employing 21 different electronic triggers to identify potential adverse events, an elevated INR had the highest yield for events associated with harm (96% of INRs >5.0 associated with harm).[6] Although pharmacist‐managed inpatient anticoagulation services have been shown to improve warfarin management,[7, 8] there are evidence gaps regarding the causes of warfarin‐related adverse events and practice changes that could decrease their frequency. Although overanticoagulation is a well‐known risk factor for warfarin‐related adverse events,[9, 10] there are few evidence‐based warfarin monitoring and dosing recommendations for hospitalized patients.[10] For example, the 2012 American College of Chest Physicians Antithrombotic Guidelines[11] provide a weak recommendation on initial dosing of warfarin, but no recommendations on how frequently to monitor the INR, or appropriate dosing responses to INR levels. Although many hospitals employ protocols that suggest daily INR monitoring until stable, there are no evidence‐based guidelines to support this practice.[12] Conversely, there are reports of flags to order an INR level that are not activated unless greater than 2[13] or 3 days[14] pass since the prior INR. Protocols from some major academic medical centers suggest that after a therapeutic INR is reached, INR levels can be measured intermittently, as infrequently as twice a week.[15, 16]

The 2015 Joint Commission anticoagulant‐focused National Patient Safety Goal[17] (initially issued in 2008) mandates the assessment of baseline coagulation status before starting warfarin, and warfarin dosing based on a current INR; however, current is not defined. Neither the extent to which the mandate for assessing baseline coagulation status is adhered to nor the relationship between this process of care and patient outcomes is known. The importance of adverse drug events associated with anticoagulants, included warfarin, was also recently highlighted in the 2014 federal National Action Plan for Adverse Drug Event Prevention. In this document, the prevention of adverse drug events associated with anticoagulants was 1 of the 3 areas selected for special national attention and action.[18]

The Medicare Patient Safety Monitoring System (MPSMS) is a national chart abstraction‐based system that includes 21 in‐hospital adverse event measures, including warfarin‐associated adverse drug events.[2] Because of the importance of warfarin‐associated bleeding in hospitalized patients, we analyzed MPSMS data to determine what factors related to INR monitoring practices place patients at risk for these events. We were particularly interested in determining if we could detect potentially modifiable predictors of overanticoagulation and warfarin‐associated adverse events.

METHODS

RESULTS

There were 130,828 patients included in the 2009 to 2013 MPSMS sample, of whom 19,445 (14.9%) received warfarin during their hospital stay and had at least 1 INR measurement. Among these patients, 5228 (26.9%) had no INR level above 1.5 and were excluded from further analysis, leaving 14,217 included patients. Of these patients, 1055 (7.4%) developed a warfarin‐associated adverse event. Table 1 demonstrates the baseline demographics and comorbidities of the included patients.

Warfarin was started on hospital day 1 for 6825 (48.0%) of 14,217 patients. Among these patients, 6539 (95.8%) had an INR measured within 1 calendar day. We were unable to determine how many patients who started warfarin later in their hospital stay had a baseline INR, as we did not capture INRs performed prior to the day that warfarin was initiated.

Supporting Table 1 in the online version of this article demonstrates the association between an INR 6.0 and the occurrence of warfarin‐associated adverse events. A maximum INR 6.0 occurred in 469 (3.3%) of the patients included in the study, and among those patients, 133 (28.4%) experienced a warfarin‐associated adverse event compared to 922 (6.7%) adverse events in the 13,748 patients who did not develop an INR 6.0 (P < 0.001).

Among 8529 patients who received warfarin for at least 3 days, beginning on the third day of warfarin, 1549 patients (18.2%) did not have INR measured at least once each day that they received warfarin. Table 2 demonstrates that patients who had 2 or more days on which the INR was not measured had higher rates of INR 6.0 than patients for whom the INR was measured daily. A similar association was seen for warfarin‐associated adverse events (Table 2).

Figure 1A demonstrates the association between the number of days without an INR measurement and the subsequent development of an INR 6.0 or a warfarin‐associated adverse event, adjusted for baseline patient characteristics, receipt of heparin and LMWH, and number of days on warfarin. Patients with 1 or more days without an INR measurement had higher risk‐adjusted ORs of a subsequent INR 6.0, although the difference was not statistically significant for surgical patients. The analysis results based on inverse propensity scoring are seen in Figure 1B. Cardiac and surgical patients with 2 or more days without an INR measurement were at higher risk of having a warfarin‐associated adverse event, whereas cardiac and pneumonia patients with 1 or more days without an INR measurement were at higher risk of developing an INR 6.0.

Figure 1

Supporting Table 2 in the online version of this article demonstrates the relationship between patient characteristics and the occurrence of an INR 6.0 or a warfarin‐related adverse event. The only characteristic that was associated with either of these outcomes for all 3 patient conditions was renal disease, which was positively associated with a warfarin‐associated adverse event. Warfarin use prior to arrival was associated with lower risks of both an INR 6.0 and a warfarin‐associated adverse event, except for among surgical patients. Supporting Table 3 in the online version of this article demonstrates the differences in patient characteristics between patients who had daily INR measurement and those who had at least 1 day without an INR measurement.

Figure 2 illustrates the relationship of the maximum INR to the prior 1‐day change in INR in 4963 patients whose INR on the day prior to the maximum INR was 2.0 to 3.5. When the increase in INR was <0.9, the risk of the next day's INR being 6.0 was 0.7%, and if the increase was 0.9, the risk was 5.2%. The risk of developing an INR 5.0 was 1.9% if the preceding day's INR increase was <0.9 and 15.3% if the prior day's INR rise was 0.9. Overall, 51% of INRs 6.0 and 55% of INRs 5.0 were immediately preceded by an INR increase of 0.9. The positive likelihood ratio (LR) for a 0.9 rise in INR predicting an INR of 6.0 was 4.2, and the positive LR was 4.9 for predicting an INR 5.0.

Figure 2

There was no decline in the frequency of warfarin use among the patients in the MPSMS sample during the study period (16.7% in 2009 and 17.3% in 2013).

DISCUSSION

We studied warfarin‐associated adverse events in a nationally representative study of patients who received warfarin while in an acute care hospital for a primary diagnosis of cardiac disease, pneumonia, or major surgery. Several findings resulted from our analysis. First, warfarin is still commonly prescribed to hospitalized patients and remains a frequent cause of adverse events; 7.4% of the 2009 to 2013 MPSMS population who received warfarin and had at least 1 INR >1.5 developed a warfarin‐associated adverse event.

Over 95% of patients who received warfarin on the day of hospital admission had an INR performed within 1 day. This is similar to the results from a 2006 single center study in which 95% of patients had an INR measured prior to their first dose of warfarin.[10] Since 2008, The Joint Commission's National Patient Safety Goal has required the assessment of coagulation status before starting warfarin.[17] The high level of adherence to this standard suggests that further attention to this process of care is unlikely to significantly improve patient safety.

We also found that the lack of daily INR measurements was associated with an increased risk of an INR 6.0 and warfarin‐associated adverse events in some patient populations. There is limited evidence addressing the appropriate frequency of INR measurement in hospitalized patients receiving warfarin. The Joint Commission National Patient Safety Goal requires use of a current INR to adjust this therapy, but provides no specifics.[17] Although some experts believe that INRs should be monitored daily in hospitalized patients, this does not appear to be uniformly accepted. In some reports, 2[13] or 3[14] consecutive days without the performance of an INR was required to activate a reminder. Protocols from some major teaching hospitals specify intermittent monitoring once the INR is therapeutic.[15, 16] Because our results suggest that lapses in INR measurement lead to overanticoagulation and warfarin‐related adverse events, it may be appropriate to measure INRs daily in most hospitalized patients receiving warfarin. This would be consistent with the many known causes of INR instability in patients admitted to the hospital, including drug‐drug interactions, hepatic dysfunction, and changes in volume of distribution, such that truly stable hospitalized patients are likely rare. Indeed, hospital admission is a well‐known predictor of instability of warfarin effect. [9] Although our results suggest that daily INR measurement is associated with a lower rate of overanticoagulation, future studies might better define lower risk patients for whom daily INR measurement would not be necessary.

A prior INR increase 0.9 in 1 day was associated with an increased risk of subsequent overanticoagulation. Although a rapidly rising INR is known to predict overanticoagulation[10, 14] we could find no evidence as to what specific rate of rise confers this risk. Our results suggest that use of a warfarin dosing protocol that considers both the absolute value of the INR and the rate of rise could reduce warfarin‐related adverse events.

There are important limitations of our study. We did not abstract warfarin dosages, which precluded study of the appropriateness of both initial warfarin dosing and adjustment of the warfarin dose based on INR results. MPSMS does not reliably capture antiplatelet agents or other agents that result in drug‐drug interactions with warfarin, such as antibiotics, so this factor could theoretically have confounded our results. Antibiotic use seems unlikely to be a major confounder, because patients with acute cardiovascular disease demonstrated a similar relationship between INR measurement and an INR 6.0 to that seen with pneumonia and surgical patients, despite the latter patients likely having greater antibiotics exposure. Furthermore, MPSMS does not capture indices of severity of illness, so other unmeasured confounders could have influenced our results. Although we have data for patients admitted to the hospital for only 4 conditions, these are conditions that represent approximately 22% of hospital admissions in the United States.[2] Strengths of our study include the nationally representative and randomly selected cases and use of data that were obtained from chart abstraction as opposed to administrative data. Through the use of centralized data abstraction, we avoided the potential bias introduced when hospitals self‐report adverse events.

In summary, in a national sample of patients admitted to the hospital for 4 common conditions, warfarin‐associated adverse events were detected in 7.4% of patients who received warfarin. Lack of daily INR measurement was associated with an increased risk of overanticoagulation and warfarin‐associated adverse events in certain patient populations. A 1‐day increase in the INR of 0.9 predicted subsequent overanticoagulation. These results provide actionable opportunities to improve safety in some hospitalized patients receiving warfarin.

Warfarin is 1 of the most common causes of adverse drug events, with hospitalized patients being particularly at risk compared to outpatients.[1] Despite the availability of new oral anticoagulants (NOACs), physicians commonly prescribe warfarin to hospitalized patients,[2] likely in part due to the greater difficulty in reversing NOACs compared to warfarin. Furthermore, uptake of the NOACs is likely to be slow in resource‐poor countries due to the lower cost of warfarin.[3] However, the narrow therapeutic index, frequent drug‐drug interactions, and patient variability in metabolism of warfarin makes management challenging.[4] Thus, warfarin remains a significant cause of adverse events in hospitalized patients, occurring in approximately 3% to 8% of exposed patients, depending on underlying condition.[2, 5]

An elevated international normalized ratio (INR) is a strong predictor of drug‐associated adverse events (patient harm). In a study employing 21 different electronic triggers to identify potential adverse events, an elevated INR had the highest yield for events associated with harm (96% of INRs >5.0 associated with harm).[6] Although pharmacist‐managed inpatient anticoagulation services have been shown to improve warfarin management,[7, 8] there are evidence gaps regarding the causes of warfarin‐related adverse events and practice changes that could decrease their frequency. Although overanticoagulation is a well‐known risk factor for warfarin‐related adverse events,[9, 10] there are few evidence‐based warfarin monitoring and dosing recommendations for hospitalized patients.[10] For example, the 2012 American College of Chest Physicians Antithrombotic Guidelines[11] provide a weak recommendation on initial dosing of warfarin, but no recommendations on how frequently to monitor the INR, or appropriate dosing responses to INR levels. Although many hospitals employ protocols that suggest daily INR monitoring until stable, there are no evidence‐based guidelines to support this practice.[12] Conversely, there are reports of flags to order an INR level that are not activated unless greater than 2[13] or 3 days[14] pass since the prior INR. Protocols from some major academic medical centers suggest that after a therapeutic INR is reached, INR levels can be measured intermittently, as infrequently as twice a week.[15, 16]

The 2015 Joint Commission anticoagulant‐focused National Patient Safety Goal[17] (initially issued in 2008) mandates the assessment of baseline coagulation status before starting warfarin, and warfarin dosing based on a current INR; however, current is not defined. Neither the extent to which the mandate for assessing baseline coagulation status is adhered to nor the relationship between this process of care and patient outcomes is known. The importance of adverse drug events associated with anticoagulants, included warfarin, was also recently highlighted in the 2014 federal National Action Plan for Adverse Drug Event Prevention. In this document, the prevention of adverse drug events associated with anticoagulants was 1 of the 3 areas selected for special national attention and action.[18]

The Medicare Patient Safety Monitoring System (MPSMS) is a national chart abstraction‐based system that includes 21 in‐hospital adverse event measures, including warfarin‐associated adverse drug events.[2] Because of the importance of warfarin‐associated bleeding in hospitalized patients, we analyzed MPSMS data to determine what factors related to INR monitoring practices place patients at risk for these events. We were particularly interested in determining if we could detect potentially modifiable predictors of overanticoagulation and warfarin‐associated adverse events.

METHODS

RESULTS

There were 130,828 patients included in the 2009 to 2013 MPSMS sample, of whom 19,445 (14.9%) received warfarin during their hospital stay and had at least 1 INR measurement. Among these patients, 5228 (26.9%) had no INR level above 1.5 and were excluded from further analysis, leaving 14,217 included patients. Of these patients, 1055 (7.4%) developed a warfarin‐associated adverse event. Table 1 demonstrates the baseline demographics and comorbidities of the included patients.

Warfarin was started on hospital day 1 for 6825 (48.0%) of 14,217 patients. Among these patients, 6539 (95.8%) had an INR measured within 1 calendar day. We were unable to determine how many patients who started warfarin later in their hospital stay had a baseline INR, as we did not capture INRs performed prior to the day that warfarin was initiated.

Supporting Table 1 in the online version of this article demonstrates the association between an INR 6.0 and the occurrence of warfarin‐associated adverse events. A maximum INR 6.0 occurred in 469 (3.3%) of the patients included in the study, and among those patients, 133 (28.4%) experienced a warfarin‐associated adverse event compared to 922 (6.7%) adverse events in the 13,748 patients who did not develop an INR 6.0 (P < 0.001).

Among 8529 patients who received warfarin for at least 3 days, beginning on the third day of warfarin, 1549 patients (18.2%) did not have INR measured at least once each day that they received warfarin. Table 2 demonstrates that patients who had 2 or more days on which the INR was not measured had higher rates of INR 6.0 than patients for whom the INR was measured daily. A similar association was seen for warfarin‐associated adverse events (Table 2).

Figure 1A demonstrates the association between the number of days without an INR measurement and the subsequent development of an INR 6.0 or a warfarin‐associated adverse event, adjusted for baseline patient characteristics, receipt of heparin and LMWH, and number of days on warfarin. Patients with 1 or more days without an INR measurement had higher risk‐adjusted ORs of a subsequent INR 6.0, although the difference was not statistically significant for surgical patients. The analysis results based on inverse propensity scoring are seen in Figure 1B. Cardiac and surgical patients with 2 or more days without an INR measurement were at higher risk of having a warfarin‐associated adverse event, whereas cardiac and pneumonia patients with 1 or more days without an INR measurement were at higher risk of developing an INR 6.0.

Figure 1

Supporting Table 2 in the online version of this article demonstrates the relationship between patient characteristics and the occurrence of an INR 6.0 or a warfarin‐related adverse event. The only characteristic that was associated with either of these outcomes for all 3 patient conditions was renal disease, which was positively associated with a warfarin‐associated adverse event. Warfarin use prior to arrival was associated with lower risks of both an INR 6.0 and a warfarin‐associated adverse event, except for among surgical patients. Supporting Table 3 in the online version of this article demonstrates the differences in patient characteristics between patients who had daily INR measurement and those who had at least 1 day without an INR measurement.

Figure 2 illustrates the relationship of the maximum INR to the prior 1‐day change in INR in 4963 patients whose INR on the day prior to the maximum INR was 2.0 to 3.5. When the increase in INR was <0.9, the risk of the next day's INR being 6.0 was 0.7%, and if the increase was 0.9, the risk was 5.2%. The risk of developing an INR 5.0 was 1.9% if the preceding day's INR increase was <0.9 and 15.3% if the prior day's INR rise was 0.9. Overall, 51% of INRs 6.0 and 55% of INRs 5.0 were immediately preceded by an INR increase of 0.9. The positive likelihood ratio (LR) for a 0.9 rise in INR predicting an INR of 6.0 was 4.2, and the positive LR was 4.9 for predicting an INR 5.0.

Figure 2

There was no decline in the frequency of warfarin use among the patients in the MPSMS sample during the study period (16.7% in 2009 and 17.3% in 2013).

DISCUSSION

We studied warfarin‐associated adverse events in a nationally representative study of patients who received warfarin while in an acute care hospital for a primary diagnosis of cardiac disease, pneumonia, or major surgery. Several findings resulted from our analysis. First, warfarin is still commonly prescribed to hospitalized patients and remains a frequent cause of adverse events; 7.4% of the 2009 to 2013 MPSMS population who received warfarin and had at least 1 INR >1.5 developed a warfarin‐associated adverse event.

Over 95% of patients who received warfarin on the day of hospital admission had an INR performed within 1 day. This is similar to the results from a 2006 single center study in which 95% of patients had an INR measured prior to their first dose of warfarin.[10] Since 2008, The Joint Commission's National Patient Safety Goal has required the assessment of coagulation status before starting warfarin.[17] The high level of adherence to this standard suggests that further attention to this process of care is unlikely to significantly improve patient safety.

We also found that the lack of daily INR measurements was associated with an increased risk of an INR 6.0 and warfarin‐associated adverse events in some patient populations. There is limited evidence addressing the appropriate frequency of INR measurement in hospitalized patients receiving warfarin. The Joint Commission National Patient Safety Goal requires use of a current INR to adjust this therapy, but provides no specifics.[17] Although some experts believe that INRs should be monitored daily in hospitalized patients, this does not appear to be uniformly accepted. In some reports, 2[13] or 3[14] consecutive days without the performance of an INR was required to activate a reminder. Protocols from some major teaching hospitals specify intermittent monitoring once the INR is therapeutic.[15, 16] Because our results suggest that lapses in INR measurement lead to overanticoagulation and warfarin‐related adverse events, it may be appropriate to measure INRs daily in most hospitalized patients receiving warfarin. This would be consistent with the many known causes of INR instability in patients admitted to the hospital, including drug‐drug interactions, hepatic dysfunction, and changes in volume of distribution, such that truly stable hospitalized patients are likely rare. Indeed, hospital admission is a well‐known predictor of instability of warfarin effect. [9] Although our results suggest that daily INR measurement is associated with a lower rate of overanticoagulation, future studies might better define lower risk patients for whom daily INR measurement would not be necessary.

A prior INR increase 0.9 in 1 day was associated with an increased risk of subsequent overanticoagulation. Although a rapidly rising INR is known to predict overanticoagulation[10, 14] we could find no evidence as to what specific rate of rise confers this risk. Our results suggest that use of a warfarin dosing protocol that considers both the absolute value of the INR and the rate of rise could reduce warfarin‐related adverse events.

There are important limitations of our study. We did not abstract warfarin dosages, which precluded study of the appropriateness of both initial warfarin dosing and adjustment of the warfarin dose based on INR results. MPSMS does not reliably capture antiplatelet agents or other agents that result in drug‐drug interactions with warfarin, such as antibiotics, so this factor could theoretically have confounded our results. Antibiotic use seems unlikely to be a major confounder, because patients with acute cardiovascular disease demonstrated a similar relationship between INR measurement and an INR 6.0 to that seen with pneumonia and surgical patients, despite the latter patients likely having greater antibiotics exposure. Furthermore, MPSMS does not capture indices of severity of illness, so other unmeasured confounders could have influenced our results. Although we have data for patients admitted to the hospital for only 4 conditions, these are conditions that represent approximately 22% of hospital admissions in the United States.[2] Strengths of our study include the nationally representative and randomly selected cases and use of data that were obtained from chart abstraction as opposed to administrative data. Through the use of centralized data abstraction, we avoided the potential bias introduced when hospitals self‐report adverse events.

In summary, in a national sample of patients admitted to the hospital for 4 common conditions, warfarin‐associated adverse events were detected in 7.4% of patients who received warfarin. Lack of daily INR measurement was associated with an increased risk of overanticoagulation and warfarin‐associated adverse events in certain patient populations. A 1‐day increase in the INR of 0.9 predicted subsequent overanticoagulation. These results provide actionable opportunities to improve safety in some hospitalized patients receiving warfarin.

Warfarin is 1 of the most common causes of adverse drug events, with hospitalized patients being particularly at risk compared to outpatients.[1] Despite the availability of new oral anticoagulants (NOACs), physicians commonly prescribe warfarin to hospitalized patients,[2] likely in part due to the greater difficulty in reversing NOACs compared to warfarin. Furthermore, uptake of the NOACs is likely to be slow in resource‐poor countries due to the lower cost of warfarin.[3] However, the narrow therapeutic index, frequent drug‐drug interactions, and patient variability in metabolism of warfarin makes management challenging.[4] Thus, warfarin remains a significant cause of adverse events in hospitalized patients, occurring in approximately 3% to 8% of exposed patients, depending on underlying condition.[2, 5]

An elevated international normalized ratio (INR) is a strong predictor of drug‐associated adverse events (patient harm). In a study employing 21 different electronic triggers to identify potential adverse events, an elevated INR had the highest yield for events associated with harm (96% of INRs >5.0 associated with harm).[6] Although pharmacist‐managed inpatient anticoagulation services have been shown to improve warfarin management,[7, 8] there are evidence gaps regarding the causes of warfarin‐related adverse events and practice changes that could decrease their frequency. Although overanticoagulation is a well‐known risk factor for warfarin‐related adverse events,[9, 10] there are few evidence‐based warfarin monitoring and dosing recommendations for hospitalized patients.[10] For example, the 2012 American College of Chest Physicians Antithrombotic Guidelines[11] provide a weak recommendation on initial dosing of warfarin, but no recommendations on how frequently to monitor the INR, or appropriate dosing responses to INR levels. Although many hospitals employ protocols that suggest daily INR monitoring until stable, there are no evidence‐based guidelines to support this practice.[12] Conversely, there are reports of flags to order an INR level that are not activated unless greater than 2[13] or 3 days[14] pass since the prior INR. Protocols from some major academic medical centers suggest that after a therapeutic INR is reached, INR levels can be measured intermittently, as infrequently as twice a week.[15, 16]

The 2015 Joint Commission anticoagulant‐focused National Patient Safety Goal[17] (initially issued in 2008) mandates the assessment of baseline coagulation status before starting warfarin, and warfarin dosing based on a current INR; however, current is not defined. Neither the extent to which the mandate for assessing baseline coagulation status is adhered to nor the relationship between this process of care and patient outcomes is known. The importance of adverse drug events associated with anticoagulants, included warfarin, was also recently highlighted in the 2014 federal National Action Plan for Adverse Drug Event Prevention. In this document, the prevention of adverse drug events associated with anticoagulants was 1 of the 3 areas selected for special national attention and action.[18]

The Medicare Patient Safety Monitoring System (MPSMS) is a national chart abstraction‐based system that includes 21 in‐hospital adverse event measures, including warfarin‐associated adverse drug events.[2] Because of the importance of warfarin‐associated bleeding in hospitalized patients, we analyzed MPSMS data to determine what factors related to INR monitoring practices place patients at risk for these events. We were particularly interested in determining if we could detect potentially modifiable predictors of overanticoagulation and warfarin‐associated adverse events.

METHODS

RESULTS

There were 130,828 patients included in the 2009 to 2013 MPSMS sample, of whom 19,445 (14.9%) received warfarin during their hospital stay and had at least 1 INR measurement. Among these patients, 5228 (26.9%) had no INR level above 1.5 and were excluded from further analysis, leaving 14,217 included patients. Of these patients, 1055 (7.4%) developed a warfarin‐associated adverse event. Table 1 demonstrates the baseline demographics and comorbidities of the included patients.

Warfarin was started on hospital day 1 for 6825 (48.0%) of 14,217 patients. Among these patients, 6539 (95.8%) had an INR measured within 1 calendar day. We were unable to determine how many patients who started warfarin later in their hospital stay had a baseline INR, as we did not capture INRs performed prior to the day that warfarin was initiated.

Supporting Table 1 in the online version of this article demonstrates the association between an INR 6.0 and the occurrence of warfarin‐associated adverse events. A maximum INR 6.0 occurred in 469 (3.3%) of the patients included in the study, and among those patients, 133 (28.4%) experienced a warfarin‐associated adverse event compared to 922 (6.7%) adverse events in the 13,748 patients who did not develop an INR 6.0 (P < 0.001).

Among 8529 patients who received warfarin for at least 3 days, beginning on the third day of warfarin, 1549 patients (18.2%) did not have INR measured at least once each day that they received warfarin. Table 2 demonstrates that patients who had 2 or more days on which the INR was not measured had higher rates of INR 6.0 than patients for whom the INR was measured daily. A similar association was seen for warfarin‐associated adverse events (Table 2).

Figure 1A demonstrates the association between the number of days without an INR measurement and the subsequent development of an INR 6.0 or a warfarin‐associated adverse event, adjusted for baseline patient characteristics, receipt of heparin and LMWH, and number of days on warfarin. Patients with 1 or more days without an INR measurement had higher risk‐adjusted ORs of a subsequent INR 6.0, although the difference was not statistically significant for surgical patients. The analysis results based on inverse propensity scoring are seen in Figure 1B. Cardiac and surgical patients with 2 or more days without an INR measurement were at higher risk of having a warfarin‐associated adverse event, whereas cardiac and pneumonia patients with 1 or more days without an INR measurement were at higher risk of developing an INR 6.0.

Figure 1

Supporting Table 2 in the online version of this article demonstrates the relationship between patient characteristics and the occurrence of an INR 6.0 or a warfarin‐related adverse event. The only characteristic that was associated with either of these outcomes for all 3 patient conditions was renal disease, which was positively associated with a warfarin‐associated adverse event. Warfarin use prior to arrival was associated with lower risks of both an INR 6.0 and a warfarin‐associated adverse event, except for among surgical patients. Supporting Table 3 in the online version of this article demonstrates the differences in patient characteristics between patients who had daily INR measurement and those who had at least 1 day without an INR measurement.

Figure 2 illustrates the relationship of the maximum INR to the prior 1‐day change in INR in 4963 patients whose INR on the day prior to the maximum INR was 2.0 to 3.5. When the increase in INR was <0.9, the risk of the next day's INR being 6.0 was 0.7%, and if the increase was 0.9, the risk was 5.2%. The risk of developing an INR 5.0 was 1.9% if the preceding day's INR increase was <0.9 and 15.3% if the prior day's INR rise was 0.9. Overall, 51% of INRs 6.0 and 55% of INRs 5.0 were immediately preceded by an INR increase of 0.9. The positive likelihood ratio (LR) for a 0.9 rise in INR predicting an INR of 6.0 was 4.2, and the positive LR was 4.9 for predicting an INR 5.0.

Figure 2

There was no decline in the frequency of warfarin use among the patients in the MPSMS sample during the study period (16.7% in 2009 and 17.3% in 2013).

DISCUSSION

We studied warfarin‐associated adverse events in a nationally representative study of patients who received warfarin while in an acute care hospital for a primary diagnosis of cardiac disease, pneumonia, or major surgery. Several findings resulted from our analysis. First, warfarin is still commonly prescribed to hospitalized patients and remains a frequent cause of adverse events; 7.4% of the 2009 to 2013 MPSMS population who received warfarin and had at least 1 INR >1.5 developed a warfarin‐associated adverse event.

Over 95% of patients who received warfarin on the day of hospital admission had an INR performed within 1 day. This is similar to the results from a 2006 single center study in which 95% of patients had an INR measured prior to their first dose of warfarin.[10] Since 2008, The Joint Commission's National Patient Safety Goal has required the assessment of coagulation status before starting warfarin.[17] The high level of adherence to this standard suggests that further attention to this process of care is unlikely to significantly improve patient safety.

We also found that the lack of daily INR measurements was associated with an increased risk of an INR 6.0 and warfarin‐associated adverse events in some patient populations. There is limited evidence addressing the appropriate frequency of INR measurement in hospitalized patients receiving warfarin. The Joint Commission National Patient Safety Goal requires use of a current INR to adjust this therapy, but provides no specifics.[17] Although some experts believe that INRs should be monitored daily in hospitalized patients, this does not appear to be uniformly accepted. In some reports, 2[13] or 3[14] consecutive days without the performance of an INR was required to activate a reminder. Protocols from some major teaching hospitals specify intermittent monitoring once the INR is therapeutic.[15, 16] Because our results suggest that lapses in INR measurement lead to overanticoagulation and warfarin‐related adverse events, it may be appropriate to measure INRs daily in most hospitalized patients receiving warfarin. This would be consistent with the many known causes of INR instability in patients admitted to the hospital, including drug‐drug interactions, hepatic dysfunction, and changes in volume of distribution, such that truly stable hospitalized patients are likely rare. Indeed, hospital admission is a well‐known predictor of instability of warfarin effect. [9] Although our results suggest that daily INR measurement is associated with a lower rate of overanticoagulation, future studies might better define lower risk patients for whom daily INR measurement would not be necessary.

A prior INR increase 0.9 in 1 day was associated with an increased risk of subsequent overanticoagulation. Although a rapidly rising INR is known to predict overanticoagulation[10, 14] we could find no evidence as to what specific rate of rise confers this risk. Our results suggest that use of a warfarin dosing protocol that considers both the absolute value of the INR and the rate of rise could reduce warfarin‐related adverse events.

There are important limitations of our study. We did not abstract warfarin dosages, which precluded study of the appropriateness of both initial warfarin dosing and adjustment of the warfarin dose based on INR results. MPSMS does not reliably capture antiplatelet agents or other agents that result in drug‐drug interactions with warfarin, such as antibiotics, so this factor could theoretically have confounded our results. Antibiotic use seems unlikely to be a major confounder, because patients with acute cardiovascular disease demonstrated a similar relationship between INR measurement and an INR 6.0 to that seen with pneumonia and surgical patients, despite the latter patients likely having greater antibiotics exposure. Furthermore, MPSMS does not capture indices of severity of illness, so other unmeasured confounders could have influenced our results. Although we have data for patients admitted to the hospital for only 4 conditions, these are conditions that represent approximately 22% of hospital admissions in the United States.[2] Strengths of our study include the nationally representative and randomly selected cases and use of data that were obtained from chart abstraction as opposed to administrative data. Through the use of centralized data abstraction, we avoided the potential bias introduced when hospitals self‐report adverse events.

In summary, in a national sample of patients admitted to the hospital for 4 common conditions, warfarin‐associated adverse events were detected in 7.4% of patients who received warfarin. Lack of daily INR measurement was associated with an increased risk of overanticoagulation and warfarin‐associated adverse events in certain patient populations. A 1‐day increase in the INR of 0.9 predicted subsequent overanticoagulation. These results provide actionable opportunities to improve safety in some hospitalized patients receiving warfarin.

If you wish to receive credit for this activity, which beginson the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to:

• Identify the approximate 30‐day readmission rate of Medicare patient hospitalized initially for pneumonia.

• Distinguish which variables were accounted and unaccounted for in the development of a pneumonia readmission model.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

If you wish to receive credit for this activity, which beginson the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to:

• Identify the approximate 30‐day readmission rate of Medicare patient hospitalized initially for pneumonia.

• Distinguish which variables were accounted and unaccounted for in the development of a pneumonia readmission model.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

If you wish to receive credit for this activity, which beginson the next page, please refer to the website: www.blackwellpublishing.com/cme.

Accreditation and Designation Statement

Blackwell Futura Media Services designates this educational activity for a 1 AMA PRA Category 1 Credit. Physicians should only claim credit commensurate with the extent of their participation in the activity.

Blackwell Futura Media Services is accredited by the Accreditation Council for Continuing Medical Education to provide continuing medical education for physicians.

Educational Objectives

Upon completion of this educational activity, participants will be better able to:

• Identify the approximate 30‐day readmission rate of Medicare patient hospitalized initially for pneumonia.

• Distinguish which variables were accounted and unaccounted for in the development of a pneumonia readmission model.

Continuous participation in the Journal of Hospital Medicine CME program will enable learners to be better able to:

• Interpret clinical guidelines and their applications for higher quality and more efficient care for all hospitalized patients.

• Describe the standard of care for common illnesses and conditions treated in the hospital; such as pneumonia, COPD exacerbation, acute coronary syndrome, HF exacerbation, glycemic control, venous thromboembolic disease, stroke, etc.

• Discuss evidence‐based recommendations involving transitions of care, including the hospital discharge process.

• Gain insights into the roles of hospitalists as medical educators, researchers, medical ethicists, palliative care providers, and hospital‐based geriatricians.

• Incorporate best practices for hospitalist administration, including quality improvement, patient safety, practice management, leadership, and demonstrating hospitalist value.

• Identify evidence‐based best practices and trends for both adult and pediatric hospital medicine.

Instructions on Receiving Credit

For information on applicability and acceptance of continuing medical education credit for this activity, please consult your professional licensing board.

This activity is designed to be completed within the time designated on the title page; physicians should claim only those credits that reflect the time actually spent in the activity. To successfully earn credit, participants must complete the activity during the valid credit period that is noted on the title page.

Follow these steps to earn credit:

• Log on to www.blackwellpublishing.com/cme.

• Read the target audience, learning objectives, and author disclosures.

• Read the article in print or online format.

• Reflect on the article.

• Access the CME Exam, and choose the best answer to each question.

• Complete the required evaluation component of the activity.

Hospital readmissions are emblematic of the numerous challenges facing the US health care system. Despite high levels of spending, nearly 20% of Medicare beneficiaries are readmitted within 30 days of hospital discharge, many readmissions are considered preventable, and rates vary widely by hospital and region.1 Further, while readmissions have been estimated to cost taxpayers as much as $17 billion annually, the current fee‐for‐service method of paying for the acute care needs of seniors rewards hospitals financially for readmission, not their prevention.2

Pneumonia is the second most common reason for hospitalization among Medicare beneficiaries, accounting for approximately 650,000 admissions annually,3 and has been a focus of national quality‐improvement efforts for more than a decade.4, 5 Despite improvements in key processes of care, rates of readmission within 30 days of discharge following a hospitalization for pneumonia have been reported to vary from 10% to 24%.68 Among several factors, readmissions are believed to be influenced by the quality of both inpatient and outpatient care, and by care‐coordination activities occurring in the transition from inpatient to outpatient status.912

Public reporting of hospital performance is considered a key strategy for improving quality, reducing costs, and increasing the value of hospital care, both in the US and worldwide.13 In 2009, the Centers for Medicare & Medicaid Services (CMS) expanded its reporting initiatives by adding risk‐adjusted hospital readmission rates for acute myocardial infarction, heart failure, and pneumonia to the Hospital Compare website.14, 15 Readmission rates are an attractive focus for public reporting for several reasons. First, in contrast to most process‐based measures of quality (eg, whether a patient with pneumonia received a particular antibiotic), a readmission is an adverse outcome that matters to patients and families.16 Second, unlike process measures whose assessment requires detailed review of medical records, readmissions can be easily determined from standard hospital claims. Finally, readmissions are costly, and their prevention could yield substantial savings to society.

A necessary prerequisite for public reporting of readmission is a validated, risk‐adjusted measure that can be used to track performance over time and can facilitate comparisons across institutions. Toward this end, we describe the development, validation, and results of a National Quality Forum‐approved and CMS‐adopted model to estimate hospital‐specific, risk‐standardized, 30‐day readmission rates for Medicare patients hospitalized with pneumonia.17

METHODS

RESULTS

Model Derivation and Performance

After exclusions were applied, the 2006 sample included 453,251 pneumonia hospitalizations (Figure 1). The development sample consisted of 226,545 hospitalizations at 4675 hospitals, with an overall unadjusted 30‐day readmission rate of 17.4%. In 11,694 index cases (5.2%), the patient died within 30 days without being readmitted. Median readmission rate was 16.3%, 25th and 75th percentile rates were 11.1% and 21.3%, and at the 10th and 90th percentile, hospital readmission rates ranged from 4.6% to 26.7% (Figure 2).

Figure 1

Figure 2

The claims model included 39 variables (age, sex, and 37 clinical variables) (Table 1). The mean age of the cohort was 80.0 years, with 55.5% women and 11.1% nonwhite patients. Mean observed readmission rate in the development sample ranged from 9% in the lowest decile of predicted pneumonia readmission rates to 32% in the highest predicted decile, a range of 23%. The AUC was 0.63. For comparison, a model with only age and sex had an AUC of 0.51, and a model with all candidate variables had an AUC equal to 0.63 (Table 2).

Table 2.Readmission Model Performance of Administrative Claims Models

		Calibration (0, 1)*	Discrimination		Residuals Lack of Fit (Pearson Residual Fall %)				Model 2 (No. of Covariates)
			Predictive Ability (Lowest Decile, Highest Decile)	AUC	(<2)	(2, 0)	(0, 2)	(2+)
NOTE: Over‐fitting indices (0, 1) provide evidence of over‐fitting and require several steps to calculate. Let b denote the estimated vector of regression coefficients. Predicted Probabilities (p) = 1/(1+exp{Xb}), and Z = Xb (eg, the linear predictor that is a scalar value for everyone). A new logistic regression model that includes only an intercept and a slope by regressing the logits on Z is fitted in the validation sample; eg, Logit(P(Y = 1|Z)) = 0 + 1Z. Estimated values of 0 far from 0 and estimated values of 1 far from 1 provide evidence of over‐fitting. Abbreviations: AUC, area under the receiver operating curve. * Max‐rescaled R‐square. Observed rates. Wald chi‐square.
Development sample
2006	(1st half) N = 226,545	(0, 1)	(0.09, 0.32)	0.63	0	82.62	7.39	9.99	6,843 (40)
Validation sample
2006	(2nd half) N = 226,706	(0.002, 0.997)	(0.09, 0.31)	0.63	0	82.55	7.45	9.99	6,870 (40)
2005	N = 536,015	(0.035, 1.008)	(0.08, 0.31)	0.63	0	82.67	7.31	10.03	16,241 (40)

Hospital Risk‐Standardized Readmission Rates

Risk‐standardized readmission rates varied across hospitals (Figure 3). Median risk‐standardized readmission rate was 17.3%, and the 25th and 75th percentiles were 16.9% and 17.9%, respectively. The 5th percentile was 16.0% and the 95th percentile was 19.1%. Odds of readmission for a hospital one standard deviation above average was 1.4 times that of a hospital one standard deviation below average.

Figure 3

Medical Record Validation

After exclusions, the medical record sample taken from the National Pneumonia Project included 47,429 cases, with an unadjusted 30‐day readmission rate of 17.0%. The final medical record risk‐adjustment model included a total of 35 variables, whose prevalence and association with readmission risk varied modestly (Table 3). Performance of the medical record and administrative models was similar (areas under the ROC curve 0.59 and 0.63, respectively) (Table 4). Additionally, in the administrative model, predicted readmission rates ranged from 8% in the lowest predicted decile to 30% in the highest predicted decile, while in the medical record model, the corresponding rates varied from 10% to 26%.

Table 3.Regression Model Results from Medical Record Sample

Variable	Percent	Estimate	Standard Error	Odds Ratio	95% CI
NOTE: Between‐state variance = 0.024; standard error = 0.00. Abbreviations: BP, blood pressure; BUN, blood urea nitrogen; CI, confidence interval; SD, standard deviation; WBC, white blood cell count.
Age 65, mean (SD)	15.24 (7.87)	0.003	0.002	0.997	0.993	1.000
Male	46.18	0.122	0.025	1.130	1.075	1.188
Nursing home resident	17.71	0.035	0.037	1.036	0.963	1.114
Neoplastic disease	6.80	0.130	0.049	1.139	1.034	1.254
Liver disease	1.04	0.089	0.123	0.915	0.719	1.164
History of heart failure	28.98	0.234	0.029	1.264	1.194	1.339
History of renal disease	8.51	0.188	0.047	1.206	1.100	1.323
Altered mental status	17.95	0.009	0.034	1.009	0.944	1.080
Pleural effusion	21.20	0.165	0.030	1.179	1.111	1.251
BUN 30 mg/dl	23.28	0.160	0.033	1.174	1.100	1.252
BUN missing	14.56	0.101	0.185	0.904	0.630	1.298
Systolic BP <90 mmHg	2.95	0.068	0.070	1.070	0.932	1.228
Systolic BP missing	11.21	0.149	0.425	1.160	0.504	2.669
Pulse 125/min	7.73	0.036	0.047	1.036	0.945	1.137
Pulse missing	11.22	0.210	0.405	1.234	0.558	2.729
Respiratory rate 30/min	16.38	0.079	0.034	1.082	1.012	1.157
Respiratory rate missing	11.39	0.204	0.240	1.226	0.765	1.964
Sodium <130 mmol/L	4.82	0.136	0.057	1.145	1.025	1.280
Sodium missing	14.39	0.049	0.143	1.050	0.793	1.391
Glucose 250 mg/dl	5.19	0.005	0.057	0.995	0.889	1.114
Glucose missing	15.44	0.156	0.105	0.855	0.696	1.051
Hematocrit <30%	7.77	0.270	0.044	1.310	1.202	1.428
Hematocrit missing	13.62	0.071	0.135	0.932	0.715	1.215
Creatinine 2.5 mg/dL	4.68	0.109	0.062	1.115	0.989	1.258
Creatinine missing	14.63	0.200	0.167	1.221	0.880	1.695
WBC 6‐12 b/L	38.04	0.021	0.049	0.979	0.889	1.079
WBC >12 b/L	41.45	0.068	0.049	0.934	0.848	1.029
WBC missing	12.85	0.167	0.162	1.181	0.860	1.623
Immunosuppressive therapy	15.01	0.347	0.035	1.415	1.321	1.516
Chronic lung disease	42.16	0.137	0.028	1.147	1.086	1.211
Coronary artery disease	39.57	0.150	0.028	1.162	1.100	1.227
Diabetes mellitus	20.90	0.137	0.033	1.147	1.076	1.223
Alcohol/drug abuse	3.40	0.099	0.071	0.906	0.788	1.041
Dementia/Alzheimer's disease	16.38	0.125	0.038	1.133	1.052	1.222
Splenectomy	0.44	0.016	0.186	1.016	0.706	1.463

Table 4.Model Performance of Medical Record Model

Model	Calibration (0, 1)*	Discrimination		Residuals Lack of Fit (Pearson Residual Fall %)				Model 2 (No. of Covariates)
		Predictive Ability (Lowest Decile, Highest Decile)	AUC	(<2)	(2, 0)	(0, 2)	(2+)
Abbreviations: AUC, area under the receiver operating curve. * Max‐rescaled R‐square. Observed rates. Wald chi‐square.
Medical Record Model Development Sample (NP)
N = 47,429 No. of 30‐day readmissions = 8,042	(1, 0)	(0.10, 0.26)	0.59	0	83.04	5.28	11.68	710 (35)
Linked Administrative Model Validation Sample
N = 47,429 No. of 30‐day readmissions = 8,042	(1, 0)	(0.08, 0.30)	0.63	0	83.04	6.94	10.01	1,414 (40)

The correlation coefficient of the estimated state‐specific standardized readmission rates from the administrative and medical record models was 0.96, and the proportion of the variance explained by the model was 0.92 (Figure 4).

Figure 4

DISCUSSION

We have described the development, validation, and results of a hospital, 30‐day, risk‐standardized readmission model for pneumonia that was created to support current federal transparency initiatives. The model uses administrative claims data from Medicare fee‐for‐service patients and produces results that are comparable to a model based on information obtained through manual abstraction of medical records. We observed an overall 30‐day readmission rate of 17%, and our analyses revealed substantial variation across US hospitals, suggesting that improvement by lower performing institutions is an achievable goal.

Because more than one in six pneumonia patients are rehospitalized shortly after discharge, and because pneumonia hospitalizations represent an enormous expense to the Medicare program, prevention of readmissions is now widely recognized to offer a substantial opportunity to improve patient outcomes while simultaneously lowering health care costs. Accordingly, promotion of strategies to reduce readmission rates has become a key priority for payers and quality‐improvement organizations. These range from policy‐level attempts to stimulate change, such as publicly reporting hospital readmission rates on government websites, to establishing accreditation standardssuch as the Joint Commission's requirement to accurately reconcile medications, to the creation of quality improvement collaboratives focused on sharing best practices across institutions. Regardless of the approach taken, a valid, risk‐adjusted measure of performance is required to evaluate and track performance over time. The measure we have described meets the National Quality Forum's measure evaluation criteria in that it addresses an important clinical topic for which there appears to be significant opportunities for improvement, the measure is precisely defined and has been subjected to validity and reliability testing, it is risk‐adjusted based on patient clinical factors present at the start of care, is feasible to produce, and is understandable by a broad range of potential users.21 Because hospitalists are the physicians primarily responsible for the care of patients with pneumonia at US hospitals, and because they frequently serve as the physician champions for quality improvement activities related to pneumonia, it is especially important that they maintain a thorough understanding of the measures and methodologies underlying current efforts to measure hospital performance.

Several features of our approach warrant additional comment. First, we deliberately chose to measure all readmission events rather than attempt to discriminate between potentially preventable and nonpreventable readmissions. From the patient perspective, readmission for any reason is a concern, and limiting the measure to pneumonia‐related readmissions could make it susceptible to gaming by hospitals. Moreover, determining whether a readmission is related to a potential quality problem is not straightforward. For example, a patient with pneumonia whose discharge medications were prescribed incorrectly may be readmitted with a hip fracture following an episode of syncope. It would be inappropriate to treat this readmission as unrelated to the care the patient received for pneumonia. Additionally, while our approach does not presume that every readmission is preventable, the goal is to reduce the risk of readmissions generally (not just in narrowly defined subpopulations), and successful interventions to reduce rehospitalization have typically demonstrated reductions in all‐cause readmission.9, 22 Second, deaths that occurred within 30 days of discharge, yet that were not accompanied by a hospital readmission, were not counted as a readmission outcome. While it may seem inappropriate to treat a postdischarge death as a nonevent (rather than censoring or excluding such cases), alternative analytic approaches, such as using a hierarchical survival model, are not currently computationally feasible with large national data sets. Fortunately, only a relatively small proportion of discharges fell into this category (5.2% of index cases in the 2006 development sample died within 30 days of discharge without being readmitted). An alternative approach to handling the competing outcome of death would have been to use a composite outcome of readmission or death. However, we believe that it is important to report the outcomes separately because factors that predict readmission and mortality may differ, and when making comparisons across hospitals it would not be possible to determine whether differences in rate were due to readmission or mortality. Third, while the patient‐level readmission model showed only modest discrimination, we intentionally excluded covariates such as race and socioeconomic status, as well as in‐hospital events and potential complications of care, and whether patients were discharged home or to a skilled nursing facility. While these variables could have improved predictive ability, they may be directly or indirectly related to quality or supply factors that should not be included in a model that seeks to control for patient clinical characteristics. For example, if hospitals with a large share of poor patients have higher readmission rates, then including income in the model will obscure differences that are important to identify. While we believe that the decision to exclude such factors in the model is in the best interest of patients, and supports efforts to reduce health inequality in society more generally, we also recognize that hospitals that care for a disproportionate share of poor patients are likely to require additional resources to overcome these social factors. Fourth, we limited the analysis to patients with a principal diagnosis of pneumonia, and chose not to also include those with a principal diagnosis of sepsis or respiratory failure coupled with a secondary diagnosis of pneumonia. While the broader definition is used by CMS in the National Pneumonia Project, that initiative relied on chart abstraction to differentiate pneumonia present at the time of admission from cases developing as a complication of hospitalization. Additionally, we did not attempt to differentiate between community‐acquired and healthcare‐associated pneumonia, however our approach is consistent with the National Pneumonia Project and Pneumonia Patient Outcomes Research Team.18 Fifth, while our model estimates readmission rates at the hospital level, we recognize that readmissions are influenced by a complex and extensive range of factors. In this context, greater cooperation between hospitals and other care providers will almost certainly be required in order to achieve dramatic improvement in readmission rates, which in turn will depend upon changes to the way serious illness is paid for. Some options that have recently been described include imposing financial penalties for early readmission, extending the boundaries of case‐based payment beyond hospital discharge, and bundling payments between hospitals and physicians.2325

Our measure has several limitations. First, our models were developed and validated using Medicare data, and the results may not apply to pneumonia patients less than 65 years of age. However, most patients hospitalized with pneumonia in the US are 65 or older. In addition, we were unable to test the model with a Medicare managed care population, because data are not currently available on such patients. Finally, the medical record‐based validation was conducted by state‐level analysis because the sample size was insufficient to carry this out at the hospital level.

In conclusion, more than 17% of Medicare beneficiaries are readmitted within 30 days following discharge after a hospitalization for pneumonia, and rates vary substantially across institutions. The development of a valid measure of hospital performance and public reporting are important first steps towards focusing attention on this problem. Actual improvement will now depend on whether hospitals and partner organizations are successful at identifying and implementing effective methods to prevent readmission.

Hospital readmissions are emblematic of the numerous challenges facing the US health care system. Despite high levels of spending, nearly 20% of Medicare beneficiaries are readmitted within 30 days of hospital discharge, many readmissions are considered preventable, and rates vary widely by hospital and region.1 Further, while readmissions have been estimated to cost taxpayers as much as $17 billion annually, the current fee‐for‐service method of paying for the acute care needs of seniors rewards hospitals financially for readmission, not their prevention.2

Pneumonia is the second most common reason for hospitalization among Medicare beneficiaries, accounting for approximately 650,000 admissions annually,3 and has been a focus of national quality‐improvement efforts for more than a decade.4, 5 Despite improvements in key processes of care, rates of readmission within 30 days of discharge following a hospitalization for pneumonia have been reported to vary from 10% to 24%.68 Among several factors, readmissions are believed to be influenced by the quality of both inpatient and outpatient care, and by care‐coordination activities occurring in the transition from inpatient to outpatient status.912

Public reporting of hospital performance is considered a key strategy for improving quality, reducing costs, and increasing the value of hospital care, both in the US and worldwide.13 In 2009, the Centers for Medicare & Medicaid Services (CMS) expanded its reporting initiatives by adding risk‐adjusted hospital readmission rates for acute myocardial infarction, heart failure, and pneumonia to the Hospital Compare website.14, 15 Readmission rates are an attractive focus for public reporting for several reasons. First, in contrast to most process‐based measures of quality (eg, whether a patient with pneumonia received a particular antibiotic), a readmission is an adverse outcome that matters to patients and families.16 Second, unlike process measures whose assessment requires detailed review of medical records, readmissions can be easily determined from standard hospital claims. Finally, readmissions are costly, and their prevention could yield substantial savings to society.

A necessary prerequisite for public reporting of readmission is a validated, risk‐adjusted measure that can be used to track performance over time and can facilitate comparisons across institutions. Toward this end, we describe the development, validation, and results of a National Quality Forum‐approved and CMS‐adopted model to estimate hospital‐specific, risk‐standardized, 30‐day readmission rates for Medicare patients hospitalized with pneumonia.17

METHODS

RESULTS

Model Derivation and Performance

After exclusions were applied, the 2006 sample included 453,251 pneumonia hospitalizations (Figure 1). The development sample consisted of 226,545 hospitalizations at 4675 hospitals, with an overall unadjusted 30‐day readmission rate of 17.4%. In 11,694 index cases (5.2%), the patient died within 30 days without being readmitted. Median readmission rate was 16.3%, 25th and 75th percentile rates were 11.1% and 21.3%, and at the 10th and 90th percentile, hospital readmission rates ranged from 4.6% to 26.7% (Figure 2).

Figure 1

Figure 2

The claims model included 39 variables (age, sex, and 37 clinical variables) (Table 1). The mean age of the cohort was 80.0 years, with 55.5% women and 11.1% nonwhite patients. Mean observed readmission rate in the development sample ranged from 9% in the lowest decile of predicted pneumonia readmission rates to 32% in the highest predicted decile, a range of 23%. The AUC was 0.63. For comparison, a model with only age and sex had an AUC of 0.51, and a model with all candidate variables had an AUC equal to 0.63 (Table 2).

Table 2.Readmission Model Performance of Administrative Claims Models

		Calibration (0, 1)*	Discrimination		Residuals Lack of Fit (Pearson Residual Fall %)				Model 2 (No. of Covariates)
			Predictive Ability (Lowest Decile, Highest Decile)	AUC	(<2)	(2, 0)	(0, 2)	(2+)
NOTE: Over‐fitting indices (0, 1) provide evidence of over‐fitting and require several steps to calculate. Let b denote the estimated vector of regression coefficients. Predicted Probabilities (p) = 1/(1+exp{Xb}), and Z = Xb (eg, the linear predictor that is a scalar value for everyone). A new logistic regression model that includes only an intercept and a slope by regressing the logits on Z is fitted in the validation sample; eg, Logit(P(Y = 1|Z)) = 0 + 1Z. Estimated values of 0 far from 0 and estimated values of 1 far from 1 provide evidence of over‐fitting. Abbreviations: AUC, area under the receiver operating curve. * Max‐rescaled R‐square. Observed rates. Wald chi‐square.
Development sample
2006	(1st half) N = 226,545	(0, 1)	(0.09, 0.32)	0.63	0	82.62	7.39	9.99	6,843 (40)
Validation sample
2006	(2nd half) N = 226,706	(0.002, 0.997)	(0.09, 0.31)	0.63	0	82.55	7.45	9.99	6,870 (40)
2005	N = 536,015	(0.035, 1.008)	(0.08, 0.31)	0.63	0	82.67	7.31	10.03	16,241 (40)

Hospital Risk‐Standardized Readmission Rates

Risk‐standardized readmission rates varied across hospitals (Figure 3). Median risk‐standardized readmission rate was 17.3%, and the 25th and 75th percentiles were 16.9% and 17.9%, respectively. The 5th percentile was 16.0% and the 95th percentile was 19.1%. Odds of readmission for a hospital one standard deviation above average was 1.4 times that of a hospital one standard deviation below average.

Figure 3

Medical Record Validation

After exclusions, the medical record sample taken from the National Pneumonia Project included 47,429 cases, with an unadjusted 30‐day readmission rate of 17.0%. The final medical record risk‐adjustment model included a total of 35 variables, whose prevalence and association with readmission risk varied modestly (Table 3). Performance of the medical record and administrative models was similar (areas under the ROC curve 0.59 and 0.63, respectively) (Table 4). Additionally, in the administrative model, predicted readmission rates ranged from 8% in the lowest predicted decile to 30% in the highest predicted decile, while in the medical record model, the corresponding rates varied from 10% to 26%.

Table 3.Regression Model Results from Medical Record Sample

Variable	Percent	Estimate	Standard Error	Odds Ratio	95% CI
NOTE: Between‐state variance = 0.024; standard error = 0.00. Abbreviations: BP, blood pressure; BUN, blood urea nitrogen; CI, confidence interval; SD, standard deviation; WBC, white blood cell count.
Age 65, mean (SD)	15.24 (7.87)	0.003	0.002	0.997	0.993	1.000
Male	46.18	0.122	0.025	1.130	1.075	1.188
Nursing home resident	17.71	0.035	0.037	1.036	0.963	1.114
Neoplastic disease	6.80	0.130	0.049	1.139	1.034	1.254
Liver disease	1.04	0.089	0.123	0.915	0.719	1.164
History of heart failure	28.98	0.234	0.029	1.264	1.194	1.339
History of renal disease	8.51	0.188	0.047	1.206	1.100	1.323
Altered mental status	17.95	0.009	0.034	1.009	0.944	1.080
Pleural effusion	21.20	0.165	0.030	1.179	1.111	1.251
BUN 30 mg/dl	23.28	0.160	0.033	1.174	1.100	1.252
BUN missing	14.56	0.101	0.185	0.904	0.630	1.298
Systolic BP <90 mmHg	2.95	0.068	0.070	1.070	0.932	1.228
Systolic BP missing	11.21	0.149	0.425	1.160	0.504	2.669
Pulse 125/min	7.73	0.036	0.047	1.036	0.945	1.137
Pulse missing	11.22	0.210	0.405	1.234	0.558	2.729
Respiratory rate 30/min	16.38	0.079	0.034	1.082	1.012	1.157
Respiratory rate missing	11.39	0.204	0.240	1.226	0.765	1.964
Sodium <130 mmol/L	4.82	0.136	0.057	1.145	1.025	1.280
Sodium missing	14.39	0.049	0.143	1.050	0.793	1.391
Glucose 250 mg/dl	5.19	0.005	0.057	0.995	0.889	1.114
Glucose missing	15.44	0.156	0.105	0.855	0.696	1.051
Hematocrit <30%	7.77	0.270	0.044	1.310	1.202	1.428
Hematocrit missing	13.62	0.071	0.135	0.932	0.715	1.215
Creatinine 2.5 mg/dL	4.68	0.109	0.062	1.115	0.989	1.258
Creatinine missing	14.63	0.200	0.167	1.221	0.880	1.695
WBC 6‐12 b/L	38.04	0.021	0.049	0.979	0.889	1.079
WBC >12 b/L	41.45	0.068	0.049	0.934	0.848	1.029
WBC missing	12.85	0.167	0.162	1.181	0.860	1.623
Immunosuppressive therapy	15.01	0.347	0.035	1.415	1.321	1.516
Chronic lung disease	42.16	0.137	0.028	1.147	1.086	1.211
Coronary artery disease	39.57	0.150	0.028	1.162	1.100	1.227
Diabetes mellitus	20.90	0.137	0.033	1.147	1.076	1.223
Alcohol/drug abuse	3.40	0.099	0.071	0.906	0.788	1.041
Dementia/Alzheimer's disease	16.38	0.125	0.038	1.133	1.052	1.222
Splenectomy	0.44	0.016	0.186	1.016	0.706	1.463

Table 4.Model Performance of Medical Record Model

Model	Calibration (0, 1)*	Discrimination		Residuals Lack of Fit (Pearson Residual Fall %)				Model 2 (No. of Covariates)
		Predictive Ability (Lowest Decile, Highest Decile)	AUC	(<2)	(2, 0)	(0, 2)	(2+)
Abbreviations: AUC, area under the receiver operating curve. * Max‐rescaled R‐square. Observed rates. Wald chi‐square.
Medical Record Model Development Sample (NP)
N = 47,429 No. of 30‐day readmissions = 8,042	(1, 0)	(0.10, 0.26)	0.59	0	83.04	5.28	11.68	710 (35)
Linked Administrative Model Validation Sample
N = 47,429 No. of 30‐day readmissions = 8,042	(1, 0)	(0.08, 0.30)	0.63	0	83.04	6.94	10.01	1,414 (40)

The correlation coefficient of the estimated state‐specific standardized readmission rates from the administrative and medical record models was 0.96, and the proportion of the variance explained by the model was 0.92 (Figure 4).

Figure 4

DISCUSSION

We have described the development, validation, and results of a hospital, 30‐day, risk‐standardized readmission model for pneumonia that was created to support current federal transparency initiatives. The model uses administrative claims data from Medicare fee‐for‐service patients and produces results that are comparable to a model based on information obtained through manual abstraction of medical records. We observed an overall 30‐day readmission rate of 17%, and our analyses revealed substantial variation across US hospitals, suggesting that improvement by lower performing institutions is an achievable goal.

Because more than one in six pneumonia patients are rehospitalized shortly after discharge, and because pneumonia hospitalizations represent an enormous expense to the Medicare program, prevention of readmissions is now widely recognized to offer a substantial opportunity to improve patient outcomes while simultaneously lowering health care costs. Accordingly, promotion of strategies to reduce readmission rates has become a key priority for payers and quality‐improvement organizations. These range from policy‐level attempts to stimulate change, such as publicly reporting hospital readmission rates on government websites, to establishing accreditation standardssuch as the Joint Commission's requirement to accurately reconcile medications, to the creation of quality improvement collaboratives focused on sharing best practices across institutions. Regardless of the approach taken, a valid, risk‐adjusted measure of performance is required to evaluate and track performance over time. The measure we have described meets the National Quality Forum's measure evaluation criteria in that it addresses an important clinical topic for which there appears to be significant opportunities for improvement, the measure is precisely defined and has been subjected to validity and reliability testing, it is risk‐adjusted based on patient clinical factors present at the start of care, is feasible to produce, and is understandable by a broad range of potential users.21 Because hospitalists are the physicians primarily responsible for the care of patients with pneumonia at US hospitals, and because they frequently serve as the physician champions for quality improvement activities related to pneumonia, it is especially important that they maintain a thorough understanding of the measures and methodologies underlying current efforts to measure hospital performance.

Several features of our approach warrant additional comment. First, we deliberately chose to measure all readmission events rather than attempt to discriminate between potentially preventable and nonpreventable readmissions. From the patient perspective, readmission for any reason is a concern, and limiting the measure to pneumonia‐related readmissions could make it susceptible to gaming by hospitals. Moreover, determining whether a readmission is related to a potential quality problem is not straightforward. For example, a patient with pneumonia whose discharge medications were prescribed incorrectly may be readmitted with a hip fracture following an episode of syncope. It would be inappropriate to treat this readmission as unrelated to the care the patient received for pneumonia. Additionally, while our approach does not presume that every readmission is preventable, the goal is to reduce the risk of readmissions generally (not just in narrowly defined subpopulations), and successful interventions to reduce rehospitalization have typically demonstrated reductions in all‐cause readmission.9, 22 Second, deaths that occurred within 30 days of discharge, yet that were not accompanied by a hospital readmission, were not counted as a readmission outcome. While it may seem inappropriate to treat a postdischarge death as a nonevent (rather than censoring or excluding such cases), alternative analytic approaches, such as using a hierarchical survival model, are not currently computationally feasible with large national data sets. Fortunately, only a relatively small proportion of discharges fell into this category (5.2% of index cases in the 2006 development sample died within 30 days of discharge without being readmitted). An alternative approach to handling the competing outcome of death would have been to use a composite outcome of readmission or death. However, we believe that it is important to report the outcomes separately because factors that predict readmission and mortality may differ, and when making comparisons across hospitals it would not be possible to determine whether differences in rate were due to readmission or mortality. Third, while the patient‐level readmission model showed only modest discrimination, we intentionally excluded covariates such as race and socioeconomic status, as well as in‐hospital events and potential complications of care, and whether patients were discharged home or to a skilled nursing facility. While these variables could have improved predictive ability, they may be directly or indirectly related to quality or supply factors that should not be included in a model that seeks to control for patient clinical characteristics. For example, if hospitals with a large share of poor patients have higher readmission rates, then including income in the model will obscure differences that are important to identify. While we believe that the decision to exclude such factors in the model is in the best interest of patients, and supports efforts to reduce health inequality in society more generally, we also recognize that hospitals that care for a disproportionate share of poor patients are likely to require additional resources to overcome these social factors. Fourth, we limited the analysis to patients with a principal diagnosis of pneumonia, and chose not to also include those with a principal diagnosis of sepsis or respiratory failure coupled with a secondary diagnosis of pneumonia. While the broader definition is used by CMS in the National Pneumonia Project, that initiative relied on chart abstraction to differentiate pneumonia present at the time of admission from cases developing as a complication of hospitalization. Additionally, we did not attempt to differentiate between community‐acquired and healthcare‐associated pneumonia, however our approach is consistent with the National Pneumonia Project and Pneumonia Patient Outcomes Research Team.18 Fifth, while our model estimates readmission rates at the hospital level, we recognize that readmissions are influenced by a complex and extensive range of factors. In this context, greater cooperation between hospitals and other care providers will almost certainly be required in order to achieve dramatic improvement in readmission rates, which in turn will depend upon changes to the way serious illness is paid for. Some options that have recently been described include imposing financial penalties for early readmission, extending the boundaries of case‐based payment beyond hospital discharge, and bundling payments between hospitals and physicians.2325

Our measure has several limitations. First, our models were developed and validated using Medicare data, and the results may not apply to pneumonia patients less than 65 years of age. However, most patients hospitalized with pneumonia in the US are 65 or older. In addition, we were unable to test the model with a Medicare managed care population, because data are not currently available on such patients. Finally, the medical record‐based validation was conducted by state‐level analysis because the sample size was insufficient to carry this out at the hospital level.

In conclusion, more than 17% of Medicare beneficiaries are readmitted within 30 days following discharge after a hospitalization for pneumonia, and rates vary substantially across institutions. The development of a valid measure of hospital performance and public reporting are important first steps towards focusing attention on this problem. Actual improvement will now depend on whether hospitals and partner organizations are successful at identifying and implementing effective methods to prevent readmission.

Hospital readmissions are emblematic of the numerous challenges facing the US health care system. Despite high levels of spending, nearly 20% of Medicare beneficiaries are readmitted within 30 days of hospital discharge, many readmissions are considered preventable, and rates vary widely by hospital and region.1 Further, while readmissions have been estimated to cost taxpayers as much as $17 billion annually, the current fee‐for‐service method of paying for the acute care needs of seniors rewards hospitals financially for readmission, not their prevention.2

Pneumonia is the second most common reason for hospitalization among Medicare beneficiaries, accounting for approximately 650,000 admissions annually,3 and has been a focus of national quality‐improvement efforts for more than a decade.4, 5 Despite improvements in key processes of care, rates of readmission within 30 days of discharge following a hospitalization for pneumonia have been reported to vary from 10% to 24%.68 Among several factors, readmissions are believed to be influenced by the quality of both inpatient and outpatient care, and by care‐coordination activities occurring in the transition from inpatient to outpatient status.912

Public reporting of hospital performance is considered a key strategy for improving quality, reducing costs, and increasing the value of hospital care, both in the US and worldwide.13 In 2009, the Centers for Medicare & Medicaid Services (CMS) expanded its reporting initiatives by adding risk‐adjusted hospital readmission rates for acute myocardial infarction, heart failure, and pneumonia to the Hospital Compare website.14, 15 Readmission rates are an attractive focus for public reporting for several reasons. First, in contrast to most process‐based measures of quality (eg, whether a patient with pneumonia received a particular antibiotic), a readmission is an adverse outcome that matters to patients and families.16 Second, unlike process measures whose assessment requires detailed review of medical records, readmissions can be easily determined from standard hospital claims. Finally, readmissions are costly, and their prevention could yield substantial savings to society.

A necessary prerequisite for public reporting of readmission is a validated, risk‐adjusted measure that can be used to track performance over time and can facilitate comparisons across institutions. Toward this end, we describe the development, validation, and results of a National Quality Forum‐approved and CMS‐adopted model to estimate hospital‐specific, risk‐standardized, 30‐day readmission rates for Medicare patients hospitalized with pneumonia.17

METHODS

RESULTS

Model Derivation and Performance

After exclusions were applied, the 2006 sample included 453,251 pneumonia hospitalizations (Figure 1). The development sample consisted of 226,545 hospitalizations at 4675 hospitals, with an overall unadjusted 30‐day readmission rate of 17.4%. In 11,694 index cases (5.2%), the patient died within 30 days without being readmitted. Median readmission rate was 16.3%, 25th and 75th percentile rates were 11.1% and 21.3%, and at the 10th and 90th percentile, hospital readmission rates ranged from 4.6% to 26.7% (Figure 2).

Figure 1

Figure 2

The claims model included 39 variables (age, sex, and 37 clinical variables) (Table 1). The mean age of the cohort was 80.0 years, with 55.5% women and 11.1% nonwhite patients. Mean observed readmission rate in the development sample ranged from 9% in the lowest decile of predicted pneumonia readmission rates to 32% in the highest predicted decile, a range of 23%. The AUC was 0.63. For comparison, a model with only age and sex had an AUC of 0.51, and a model with all candidate variables had an AUC equal to 0.63 (Table 2).

Table 2.Readmission Model Performance of Administrative Claims Models

		Calibration (0, 1)*	Discrimination		Residuals Lack of Fit (Pearson Residual Fall %)				Model 2 (No. of Covariates)
			Predictive Ability (Lowest Decile, Highest Decile)	AUC	(<2)	(2, 0)	(0, 2)	(2+)
NOTE: Over‐fitting indices (0, 1) provide evidence of over‐fitting and require several steps to calculate. Let b denote the estimated vector of regression coefficients. Predicted Probabilities (p) = 1/(1+exp{Xb}), and Z = Xb (eg, the linear predictor that is a scalar value for everyone). A new logistic regression model that includes only an intercept and a slope by regressing the logits on Z is fitted in the validation sample; eg, Logit(P(Y = 1|Z)) = 0 + 1Z. Estimated values of 0 far from 0 and estimated values of 1 far from 1 provide evidence of over‐fitting. Abbreviations: AUC, area under the receiver operating curve. * Max‐rescaled R‐square. Observed rates. Wald chi‐square.
Development sample
2006	(1st half) N = 226,545	(0, 1)	(0.09, 0.32)	0.63	0	82.62	7.39	9.99	6,843 (40)
Validation sample
2006	(2nd half) N = 226,706	(0.002, 0.997)	(0.09, 0.31)	0.63	0	82.55	7.45	9.99	6,870 (40)
2005	N = 536,015	(0.035, 1.008)	(0.08, 0.31)	0.63	0	82.67	7.31	10.03	16,241 (40)

Hospital Risk‐Standardized Readmission Rates

Risk‐standardized readmission rates varied across hospitals (Figure 3). Median risk‐standardized readmission rate was 17.3%, and the 25th and 75th percentiles were 16.9% and 17.9%, respectively. The 5th percentile was 16.0% and the 95th percentile was 19.1%. Odds of readmission for a hospital one standard deviation above average was 1.4 times that of a hospital one standard deviation below average.

Figure 3

Medical Record Validation

After exclusions, the medical record sample taken from the National Pneumonia Project included 47,429 cases, with an unadjusted 30‐day readmission rate of 17.0%. The final medical record risk‐adjustment model included a total of 35 variables, whose prevalence and association with readmission risk varied modestly (Table 3). Performance of the medical record and administrative models was similar (areas under the ROC curve 0.59 and 0.63, respectively) (Table 4). Additionally, in the administrative model, predicted readmission rates ranged from 8% in the lowest predicted decile to 30% in the highest predicted decile, while in the medical record model, the corresponding rates varied from 10% to 26%.

Table 3.Regression Model Results from Medical Record Sample

Variable	Percent	Estimate	Standard Error	Odds Ratio	95% CI
NOTE: Between‐state variance = 0.024; standard error = 0.00. Abbreviations: BP, blood pressure; BUN, blood urea nitrogen; CI, confidence interval; SD, standard deviation; WBC, white blood cell count.
Age 65, mean (SD)	15.24 (7.87)	0.003	0.002	0.997	0.993	1.000
Male	46.18	0.122	0.025	1.130	1.075	1.188
Nursing home resident	17.71	0.035	0.037	1.036	0.963	1.114
Neoplastic disease	6.80	0.130	0.049	1.139	1.034	1.254
Liver disease	1.04	0.089	0.123	0.915	0.719	1.164
History of heart failure	28.98	0.234	0.029	1.264	1.194	1.339
History of renal disease	8.51	0.188	0.047	1.206	1.100	1.323
Altered mental status	17.95	0.009	0.034	1.009	0.944	1.080
Pleural effusion	21.20	0.165	0.030	1.179	1.111	1.251
BUN 30 mg/dl	23.28	0.160	0.033	1.174	1.100	1.252
BUN missing	14.56	0.101	0.185	0.904	0.630	1.298
Systolic BP <90 mmHg	2.95	0.068	0.070	1.070	0.932	1.228
Systolic BP missing	11.21	0.149	0.425	1.160	0.504	2.669
Pulse 125/min	7.73	0.036	0.047	1.036	0.945	1.137
Pulse missing	11.22	0.210	0.405	1.234	0.558	2.729
Respiratory rate 30/min	16.38	0.079	0.034	1.082	1.012	1.157
Respiratory rate missing	11.39	0.204	0.240	1.226	0.765	1.964
Sodium <130 mmol/L	4.82	0.136	0.057	1.145	1.025	1.280
Sodium missing	14.39	0.049	0.143	1.050	0.793	1.391
Glucose 250 mg/dl	5.19	0.005	0.057	0.995	0.889	1.114
Glucose missing	15.44	0.156	0.105	0.855	0.696	1.051
Hematocrit <30%	7.77	0.270	0.044	1.310	1.202	1.428
Hematocrit missing	13.62	0.071	0.135	0.932	0.715	1.215
Creatinine 2.5 mg/dL	4.68	0.109	0.062	1.115	0.989	1.258
Creatinine missing	14.63	0.200	0.167	1.221	0.880	1.695
WBC 6‐12 b/L	38.04	0.021	0.049	0.979	0.889	1.079
WBC >12 b/L	41.45	0.068	0.049	0.934	0.848	1.029
WBC missing	12.85	0.167	0.162	1.181	0.860	1.623
Immunosuppressive therapy	15.01	0.347	0.035	1.415	1.321	1.516
Chronic lung disease	42.16	0.137	0.028	1.147	1.086	1.211
Coronary artery disease	39.57	0.150	0.028	1.162	1.100	1.227
Diabetes mellitus	20.90	0.137	0.033	1.147	1.076	1.223
Alcohol/drug abuse	3.40	0.099	0.071	0.906	0.788	1.041
Dementia/Alzheimer's disease	16.38	0.125	0.038	1.133	1.052	1.222
Splenectomy	0.44	0.016	0.186	1.016	0.706	1.463

Table 4.Model Performance of Medical Record Model

Model	Calibration (0, 1)*	Discrimination		Residuals Lack of Fit (Pearson Residual Fall %)				Model 2 (No. of Covariates)
		Predictive Ability (Lowest Decile, Highest Decile)	AUC	(<2)	(2, 0)	(0, 2)	(2+)
Abbreviations: AUC, area under the receiver operating curve. * Max‐rescaled R‐square. Observed rates. Wald chi‐square.
Medical Record Model Development Sample (NP)
N = 47,429 No. of 30‐day readmissions = 8,042	(1, 0)	(0.10, 0.26)	0.59	0	83.04	5.28	11.68	710 (35)
Linked Administrative Model Validation Sample
N = 47,429 No. of 30‐day readmissions = 8,042	(1, 0)	(0.08, 0.30)	0.63	0	83.04	6.94	10.01	1,414 (40)

The correlation coefficient of the estimated state‐specific standardized readmission rates from the administrative and medical record models was 0.96, and the proportion of the variance explained by the model was 0.92 (Figure 4).

Figure 4

DISCUSSION

We have described the development, validation, and results of a hospital, 30‐day, risk‐standardized readmission model for pneumonia that was created to support current federal transparency initiatives. The model uses administrative claims data from Medicare fee‐for‐service patients and produces results that are comparable to a model based on information obtained through manual abstraction of medical records. We observed an overall 30‐day readmission rate of 17%, and our analyses revealed substantial variation across US hospitals, suggesting that improvement by lower performing institutions is an achievable goal.

Because more than one in six pneumonia patients are rehospitalized shortly after discharge, and because pneumonia hospitalizations represent an enormous expense to the Medicare program, prevention of readmissions is now widely recognized to offer a substantial opportunity to improve patient outcomes while simultaneously lowering health care costs. Accordingly, promotion of strategies to reduce readmission rates has become a key priority for payers and quality‐improvement organizations. These range from policy‐level attempts to stimulate change, such as publicly reporting hospital readmission rates on government websites, to establishing accreditation standardssuch as the Joint Commission's requirement to accurately reconcile medications, to the creation of quality improvement collaboratives focused on sharing best practices across institutions. Regardless of the approach taken, a valid, risk‐adjusted measure of performance is required to evaluate and track performance over time. The measure we have described meets the National Quality Forum's measure evaluation criteria in that it addresses an important clinical topic for which there appears to be significant opportunities for improvement, the measure is precisely defined and has been subjected to validity and reliability testing, it is risk‐adjusted based on patient clinical factors present at the start of care, is feasible to produce, and is understandable by a broad range of potential users.21 Because hospitalists are the physicians primarily responsible for the care of patients with pneumonia at US hospitals, and because they frequently serve as the physician champions for quality improvement activities related to pneumonia, it is especially important that they maintain a thorough understanding of the measures and methodologies underlying current efforts to measure hospital performance.

Several features of our approach warrant additional comment. First, we deliberately chose to measure all readmission events rather than attempt to discriminate between potentially preventable and nonpreventable readmissions. From the patient perspective, readmission for any reason is a concern, and limiting the measure to pneumonia‐related readmissions could make it susceptible to gaming by hospitals. Moreover, determining whether a readmission is related to a potential quality problem is not straightforward. For example, a patient with pneumonia whose discharge medications were prescribed incorrectly may be readmitted with a hip fracture following an episode of syncope. It would be inappropriate to treat this readmission as unrelated to the care the patient received for pneumonia. Additionally, while our approach does not presume that every readmission is preventable, the goal is to reduce the risk of readmissions generally (not just in narrowly defined subpopulations), and successful interventions to reduce rehospitalization have typically demonstrated reductions in all‐cause readmission.9, 22 Second, deaths that occurred within 30 days of discharge, yet that were not accompanied by a hospital readmission, were not counted as a readmission outcome. While it may seem inappropriate to treat a postdischarge death as a nonevent (rather than censoring or excluding such cases), alternative analytic approaches, such as using a hierarchical survival model, are not currently computationally feasible with large national data sets. Fortunately, only a relatively small proportion of discharges fell into this category (5.2% of index cases in the 2006 development sample died within 30 days of discharge without being readmitted). An alternative approach to handling the competing outcome of death would have been to use a composite outcome of readmission or death. However, we believe that it is important to report the outcomes separately because factors that predict readmission and mortality may differ, and when making comparisons across hospitals it would not be possible to determine whether differences in rate were due to readmission or mortality. Third, while the patient‐level readmission model showed only modest discrimination, we intentionally excluded covariates such as race and socioeconomic status, as well as in‐hospital events and potential complications of care, and whether patients were discharged home or to a skilled nursing facility. While these variables could have improved predictive ability, they may be directly or indirectly related to quality or supply factors that should not be included in a model that seeks to control for patient clinical characteristics. For example, if hospitals with a large share of poor patients have higher readmission rates, then including income in the model will obscure differences that are important to identify. While we believe that the decision to exclude such factors in the model is in the best interest of patients, and supports efforts to reduce health inequality in society more generally, we also recognize that hospitals that care for a disproportionate share of poor patients are likely to require additional resources to overcome these social factors. Fourth, we limited the analysis to patients with a principal diagnosis of pneumonia, and chose not to also include those with a principal diagnosis of sepsis or respiratory failure coupled with a secondary diagnosis of pneumonia. While the broader definition is used by CMS in the National Pneumonia Project, that initiative relied on chart abstraction to differentiate pneumonia present at the time of admission from cases developing as a complication of hospitalization. Additionally, we did not attempt to differentiate between community‐acquired and healthcare‐associated pneumonia, however our approach is consistent with the National Pneumonia Project and Pneumonia Patient Outcomes Research Team.18 Fifth, while our model estimates readmission rates at the hospital level, we recognize that readmissions are influenced by a complex and extensive range of factors. In this context, greater cooperation between hospitals and other care providers will almost certainly be required in order to achieve dramatic improvement in readmission rates, which in turn will depend upon changes to the way serious illness is paid for. Some options that have recently been described include imposing financial penalties for early readmission, extending the boundaries of case‐based payment beyond hospital discharge, and bundling payments between hospitals and physicians.2325

Our measure has several limitations. First, our models were developed and validated using Medicare data, and the results may not apply to pneumonia patients less than 65 years of age. However, most patients hospitalized with pneumonia in the US are 65 or older. In addition, we were unable to test the model with a Medicare managed care population, because data are not currently available on such patients. Finally, the medical record‐based validation was conducted by state‐level analysis because the sample size was insufficient to carry this out at the hospital level.

In conclusion, more than 17% of Medicare beneficiaries are readmitted within 30 days following discharge after a hospitalization for pneumonia, and rates vary substantially across institutions. The development of a valid measure of hospital performance and public reporting are important first steps towards focusing attention on this problem. Actual improvement will now depend on whether hospitals and partner organizations are successful at identifying and implementing effective methods to prevent readmission.