User login
Developing a Real-Time Prediction Model for Medicine Service 30-Day Readmissions
From Tufts Medical Center, Boston, MA (Dr. Trautwein and Dr. Schwartz contributed equally to this article).
Abstract
- Objective: To examine whether an institution- and service-specific readmission prediction instrument has improved performance compared to universal tools.
- Design: Retrospective cohort study.
- Setting: Academic medical center located in Boston, MA.
- Participants: Adult inpatients admitted to a medicine service.
- Measurements: Patient attributes, inpatient service assignment, and 30-day readmission rate.
- Results: Of 7972 index admissions, 12.6% were readmitted within 30 days. Thirty-day readmissions were associated with number of medications on admission (adjusted odds ratio [OR], 1.34; 95% confidence interval [CI], 1.11-1.61) for ≥ 11 compared with ≤ 5 medications; prior admission or overnight observation (OR, 1.89; 95% CI, 1.61-2.23); and discharge (OR, 2.45; 95% CI, 1.97-3.06) to an acute care facility compared to home without services. The subspecialty services with the highest risk of readmission were bone marrow transplant/hematology (OR, 2.46; 95% CI, 1.78-3.40) and oncology (OR, 2.26; 95% CI, 1.67-3.05), as compared with general medicine/geriatrics. The C statistic for the derivation cohort was 0.67, as compared with a C statistic of 0.63 for the LACE index.
- Conclusion: A hospital service–specific 30-day readmission prediction tool showed incrementally improved performance over the widely used LACE index.
Keywords: rehospitalization; quality of care; predictive model; hospital medicine.
Readmissions are a costly problem in the United States. The readmission rate among Medicare beneficiaries aged 65 years and older was 17.1 per 100 live discharges in 2016.1 Although both the relationship between readmissions and quality of care and the use of readmissions as a quality measure are contested,2 efforts to reduce potentially avoidable rehospitalizations have received widespread attention and readmissions are a focus of reimbursement reform.3-5
The LACE index is a widely used model for readmission risk prediction.6 Derived from a study of hospitalized patients in Ontario, Canada, LACE uses information about length of stay, acuity, comorbidities, and emergency department utilization to estimate readmission risk. Models such as LACE+, LACE-rt, and HOSPITAL have tried to improve on LACE’s performance, but models with strong discriminative ability are lacking.7-9 Institution-specific models exist,10 as do well-organized multicenter studies,7 but their generalizability is limited due to population differences or inclusion of data difficult to extract from patients in real-time, such as data gathered through a socioeconomic questionnaire.
Given hospital-specific differences in operational performance and patient population, we sought to develop a statistical model for 30-day readmission prediction and demonstrate the process that could be utilized by other institutions to identify high-risk patients for intensive case management and discharge planning.
Methods
Study Design
We conducted a retrospective cohort study of all admissions to the medicine service at a 415-bed teaching hospital in Boston, MA, from September 1, 2013, through August 31, 2016. Patients are admitted through the emergency department or directly admitted from hospital-based practices or a statewide network of private practices.
Data Collection
Data were abstracted from electronic medical and billing records from the first (index) admission for each patient during the study period. Thirty-day readmission was defined as an unplanned admission in the 30 days following the index discharge date. We excluded patients readmitted after leaving against medical advice and planned readmissions based on information in discharge summaries.
The study team identified candidate risk factors by referencing related published research and with input from a multidisciplinary task force charged with developing strategies to reduce 30-day readmissions. Task force members included attending and resident physicians, pharmacists, nurses, case managers, and administrators. The task force considered factors that could be extracted from the electronic medical record, including demographics, location of care, and clinical measures such as diagnostic codes, as well as data available in nursing, social work, and case management notes. Decisions regarding potential risk factors were reached within the group based on institutional experience, availability, and quality of data within the electronic record for specific variables, as well as published research on the subject, with the goal of selecting variables that could be easily identified before discharge and used to generate a predictive score for use in discharge planning.
Variables
Variables initially considered for inclusion in univariate analyses included demographic characteristics of age, gender, and a combined race/ethnicity variable, delineated as either non-Hispanic white, non-Hispanic black, Asian/Asian Indian, or Hispanic. Those with race listed as other or missing were set to missing. Primary language was categorized as English versus non-English. We included a number of variables related to the severity of the patient’s medical condition during the index admission, including any stay in an intensive care unit (ICU) and number of medications on admission, divided into 3 groups, 0-5, 6-10, and 11 or more. We also included separate indicators for admissions on warfarin or chronic opioids. Charlson comorbidity score as well as heart failure, diabetes, and chronic obstructive pulmonary disease were included as separate variables, since these specific diagnoses have high comorbidity and risk of readmission.
Because the hospital’s medicine service is divided into subspecialty services, we included the admitting service and discharge unit to assess whether certain teams or units were associated with readmission. Discharge disposition was categorized as home with services (ie, physical therapy and visiting nurse), home without services, skilled nursing facility, acute care facility, or other. We included a variable to assess patient frailty and mobility based on the presence of a physical therapy consult. We incorporated social determinants of health, including insurance coverage (private insurance, Medicare, Medicaid, subsidized, or uninsured); per capita income from the patient’s zip code as a proxy for economic status (divided into quartiles for analysis); and substance abuse and alcohol abuse (based on International Classification of Diseases, 10th revision codes). We considered whether the discharge was on a weekday or weekend, and considered distance to the hospital in relation to Boston, either within route 128 (roughly within 15-20 miles of the medical center), within interstate 495 (roughly within 30-40 miles of the medical center), or beyond this. We considered but were unable to incorporate candidate variables that had inconsistent availability in the electronic medical record, such as the Braden score, level of independence with activities of daily living, nursing-determined fall risk, presence of a social work or nutrition consultation, CAGE questionnaire for alcohol abuse, delirium assessment score, the number of adults living in the home, the number of dependents, and marital status.
Analysis
We created a derivation cohort using admission data from September 1, 2013, through November 30, 2015. We used a backward selection process to include variables in the derivation model. Any variable associated with 30-day readmissions with a P value < 0.10 in univariate analyses was considered as a candidate variable. To be retained in the multivariable model, each variable was required to have a significant association with 30-day readmission at the P < 0.05 level. We used beta coefficients to create a numerical score reflective of probability of readmission.
We then created a validation cohort using admissions data between December 1, 2015, and August 31, 2016. We applied the scoring algorithm from the derivation cohort to the validation cohort and compared the discriminative ability of the 2 models using the area under the receiver operating characteristic (ROC) curve. We also compared the area under the ROC curve of our predictive model to the LACE index using the nonparametric approach of DeLong and colleagues.8,11
Results
The derivation cohort consisted of 7972 index admissions, of which 12.6% were readmitted within 30 days. The patient population was 45% female, 70% white, and 85% English-speaking, with an average age of 61.4 years (standard deviation, 18.1, Table 1). Most patients had either private insurance (43%) or Medicare (41%).
Many patients were medically complex: 21% required ICU care, 29% were taking 11 or more medications on admission, and 52% had a Charlson score of 2 or more. In the previous 6 months, 14% had an emergency department visit and 16% had an admission or overnight observation. The rate of drug or alcohol abuse was 13%. The majority of patients were discharged home without services (54%), while 23% were discharged home with services, 13% were discharged to a skilled nursing facility, and 9% were discharged to an acute care facility.
Factors Associated With 30-Day Readmission
Table 2 displays the univariate and multivariate associations with 30-day readmissions in the derivation cohort. Variables significantly associated with readmission in univariate analysis were Charlson comorbidity score, history of diabetes, ICU utilization, previous emergency department visit, inpatient or observation stay within 6 months, number of medications on admission, use of opioids on admission, a diagnosis of diabetes, type of insurance, physical therapy consultation, admitting service, and discharge disposition. Variables not significant in univariate analysis included warfarin use, alcohol/drug abuse, distance from the hospital, and demographic variables (age, sex, race, language, income by zip code).
Variables associated with readmission in the final multivariate analysis model included a Charlson score of 2 or higher (compared to a score of 0; odds ratio [OR], 1.36; 95% confidence interval [CI], 1.11-1.66); any ICU stay (OR, 1.29; 95% CI, 1.08-1.53); number of medications on admission (OR, 1.34; 95% CI, 1.11-1.61) for 11 or more compared with 5 or fewer medications; prior admission or overnight observation (OR, 1.89; 95% CI, 1.61-2.23); and disposition on discharge to an acute care facility (OR, 2.45; 95% CI, 1.96-3.06), skilled nursing facility (OR, 1.53; 95% CI, 1.23-1.89), or home with services (OR, 1.64; 95% CI, 1.38-1.94) compared with home discharge without services. The hospital service from which the patient was discharged was significantly associated with readmission; the subspecialty services with the highest odds ratios were bone marrow transplant/hematology (OR, 2.46; 95% CI, 1.77-3.40) and oncology (OR, 2.26; 95% CI, 1.67-3.05), as compared with general medicine/geriatrics.
Model Derivation and Validation
We utilized the beta coefficients from the multivariate analysis to create a scoring tool to predict the likelihood of 30-day readmission. We rounded each beta coefficient and calculated a readmission score by adding together the rounded beta coefficients of each of the significant variables. Table 3 presents the cumulative percentage of discharges at each score level, as well as the calculated cumulative percentage of potential readmissions. For example, in our population, a score of 6 or greater accounted for 18% of all discharges, but 36% of all 30-day readmissions.
The ROC curves for the derivation model and LACE index are shown in the Figure. The C statistic for the derivation cohort was 0.67, as compared with a C statistic of 0.63 for the calculated LACE index (P < 0.0001). The validation cohort had a C statistic similar to that of the derivation cohort (0.66).
Discussion
We developed a predictive model that can be used during admission to stratify patients for intensive case management and discharge planning. The model included Charlson score, ICU utilization, admission to inpatient services or observation, visits to the ED in the past 6 months, number of medications on admission, hospital service, and discharge disposition. The C statistic of 0.67 is better than that of the LACE predictive model for our population, although both reflect only modest predictive value.
While our model, which was developed and validated at a single institution, may not be generalizable to other institutions, the method of developing a readmission risk prediction model specific to an institution is readily replicable. While standardized tools for predicting readmission risk exist, they do not necessarily account for unique patient populations and medical complexity at individual institutions. We examined patients discharged over 3 years from the medicine services, creating a service-specific model. Our approach could lead to the widespread development of service- and institution-specific models that take into account the risks and resources appropriate to each patient population and setting.
Many of the factors included in our model were indicative of the patients’ medical complexity, with Charlson comorbidity score and the number of discharge medications strongly associated with readmission. In the derivation of the LACE model, many patients were middle-aged and independent in their activities of daily living, and more than 75% had a Charlson comorbidity score of 0.6.6 In our population, by contrast, the majority of patients had a Charlson score of 2 or greater. Health care utilization was a strong positive predictor in both our model and in LACE. The finding that the Charlson comorbidity score was a better predictor than any single chronic illness suggests that medical complexity and comorbidities increase the likelihood of admission more than any 1 chronic condition. Our model incorporates discharge disposition in readmission prediction, another factor associated with medical complexity and frailty.
A surprising finding in our study was the lack of association between social determinants of health, such as alcohol and drug abuse, and readmission risk. We posit several reasons that may account for this finding. First, the population served by the medical center may be too small or homogeneous with respect to social determinants of health to detect a difference in readmission risk. Second, markers available in the electronic medical record to determine social needs may be too crude to distinguish degrees of vulnerability that increase the risk of readmission. We do not discount the importance of social determinants of health as predictors of readmission risk, but we do acknowledge the limitations of the data incorporated in our model.
Predictive models are useful only if they can be incorporated into workflow to identify high-risk patients. Prior to developing and using our model, we used LACE inconsistently because it required length of stay as 1 of the variables. Because the variables in our model are collected and recorded routinely at admission in our electronic medical record, the readmission risk score is calculated and displayed in a daily high-risk patient report. This automated process has afforded a more consistent and reliable approach to readmission risk assessment than previous efforts to assess the LACE index. Case managers use the high-risk patient report to identify patients who require enhanced care coordination and discharge planning. Since the introduction of this predictive model, we have noted a 10% reduction in the hospital’s 30-day readmission rate.
This project was subject to several limitations. Because data on admissions to other facilities were unavailable, we may have underestimated the risk of readmission to other facilities. Our results may not be generalizable to other organizations, although we believe that the methods are readily replicable. The performance of the model and its replication with a validation cohort are strengths of the approach.
Conclusion
We created a hospital service–specific 30-day readmission prediction tool whose performance improved incrementally over the widely used LACE index. This research suggests that readmission prediction is highly context-specific and that organizations would do well to examine the readmission risk factors most pertinent to the populations they serve. We believe that “customized” readmission risk prediction models for particular services in specific hospitals may offer a superior method to identify high-risk patients who may benefit from individualized care planning. Future research is needed to understand how best to capture information about the attributes of vulnerable populations, so that this information can be incorporated into future risk models.
Corresponding author: Karen Freund, MD, Tufts Medical Center, 800 Washington St., Boston, MA 02111; kfreund@tuftsmedicalcenter.org.
Financial disclosures: None.
Funding for this work was provided by the Commonwealth of Massachusetts, Executive Office of Health and Human Services.
1. Bailey MK, Weiss AJ, Barrett ML, Jiang HJ. Characteristics of 30-day all-cause hospital readmissions, 2010-2016. HCUP Statistical Brief #248. February 2019. Agency for Healthcare Research and Quality. Rockville, MD. www.hcup-us.ahrq.gov/reports/statbriefs/sb248-Hospital-Readmissions-2010-2016.pdf. Accessed December 12, 2019.
2. Esposito ML, Selker HP, Salem DN. Quantity over quality: how the rise in quality measures is not producing quality results. J Gen Intern Med. 2015;30:1204-1207.
3. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the medicare fee-for-service program. N Engl J Med. 2009;360:1418-1428.
4. Centers for Medicare and Medicaid. Readmissions-reduction-program. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed December 8, 2019.
5. Institute for Healthcare Improvement. Readmissions. Reduce avoidable readmissions. www.ihi.org/Topics/Readmissions/Pages/default.aspx. Accessed December 8, 2019.
6. Van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182:551-557.
7. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688-1698.
8. Robinson R, Hudali T. The HOSPITAL score and LACE index as predictors of 30 day readmission in a retrospective study at a university-affiliated community hospital. Peer J. 2017;5:e3137.
9. El Morr C, Ginsburg L, Nam S, Woollard S. Assessing the performance of a modified LACE index (LACE-rt) to predict unplanned readmission after discharge in a community teaching hospital. Interactive J Med Res. 2017;6:e2.
10. Yu S, Farooq F, Van Esbroeck A, et al. Predicting readmission risk with institution-specific prediction models. Artif Intell Med. 2015;65:89-96.
11. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;4:837-845.
From Tufts Medical Center, Boston, MA (Dr. Trautwein and Dr. Schwartz contributed equally to this article).
Abstract
- Objective: To examine whether an institution- and service-specific readmission prediction instrument has improved performance compared to universal tools.
- Design: Retrospective cohort study.
- Setting: Academic medical center located in Boston, MA.
- Participants: Adult inpatients admitted to a medicine service.
- Measurements: Patient attributes, inpatient service assignment, and 30-day readmission rate.
- Results: Of 7972 index admissions, 12.6% were readmitted within 30 days. Thirty-day readmissions were associated with number of medications on admission (adjusted odds ratio [OR], 1.34; 95% confidence interval [CI], 1.11-1.61) for ≥ 11 compared with ≤ 5 medications; prior admission or overnight observation (OR, 1.89; 95% CI, 1.61-2.23); and discharge (OR, 2.45; 95% CI, 1.97-3.06) to an acute care facility compared to home without services. The subspecialty services with the highest risk of readmission were bone marrow transplant/hematology (OR, 2.46; 95% CI, 1.78-3.40) and oncology (OR, 2.26; 95% CI, 1.67-3.05), as compared with general medicine/geriatrics. The C statistic for the derivation cohort was 0.67, as compared with a C statistic of 0.63 for the LACE index.
- Conclusion: A hospital service–specific 30-day readmission prediction tool showed incrementally improved performance over the widely used LACE index.
Keywords: rehospitalization; quality of care; predictive model; hospital medicine.
Readmissions are a costly problem in the United States. The readmission rate among Medicare beneficiaries aged 65 years and older was 17.1 per 100 live discharges in 2016.1 Although both the relationship between readmissions and quality of care and the use of readmissions as a quality measure are contested,2 efforts to reduce potentially avoidable rehospitalizations have received widespread attention and readmissions are a focus of reimbursement reform.3-5
The LACE index is a widely used model for readmission risk prediction.6 Derived from a study of hospitalized patients in Ontario, Canada, LACE uses information about length of stay, acuity, comorbidities, and emergency department utilization to estimate readmission risk. Models such as LACE+, LACE-rt, and HOSPITAL have tried to improve on LACE’s performance, but models with strong discriminative ability are lacking.7-9 Institution-specific models exist,10 as do well-organized multicenter studies,7 but their generalizability is limited due to population differences or inclusion of data difficult to extract from patients in real-time, such as data gathered through a socioeconomic questionnaire.
Given hospital-specific differences in operational performance and patient population, we sought to develop a statistical model for 30-day readmission prediction and demonstrate the process that could be utilized by other institutions to identify high-risk patients for intensive case management and discharge planning.
Methods
Study Design
We conducted a retrospective cohort study of all admissions to the medicine service at a 415-bed teaching hospital in Boston, MA, from September 1, 2013, through August 31, 2016. Patients are admitted through the emergency department or directly admitted from hospital-based practices or a statewide network of private practices.
Data Collection
Data were abstracted from electronic medical and billing records from the first (index) admission for each patient during the study period. Thirty-day readmission was defined as an unplanned admission in the 30 days following the index discharge date. We excluded patients readmitted after leaving against medical advice and planned readmissions based on information in discharge summaries.
The study team identified candidate risk factors by referencing related published research and with input from a multidisciplinary task force charged with developing strategies to reduce 30-day readmissions. Task force members included attending and resident physicians, pharmacists, nurses, case managers, and administrators. The task force considered factors that could be extracted from the electronic medical record, including demographics, location of care, and clinical measures such as diagnostic codes, as well as data available in nursing, social work, and case management notes. Decisions regarding potential risk factors were reached within the group based on institutional experience, availability, and quality of data within the electronic record for specific variables, as well as published research on the subject, with the goal of selecting variables that could be easily identified before discharge and used to generate a predictive score for use in discharge planning.
Variables
Variables initially considered for inclusion in univariate analyses included demographic characteristics of age, gender, and a combined race/ethnicity variable, delineated as either non-Hispanic white, non-Hispanic black, Asian/Asian Indian, or Hispanic. Those with race listed as other or missing were set to missing. Primary language was categorized as English versus non-English. We included a number of variables related to the severity of the patient’s medical condition during the index admission, including any stay in an intensive care unit (ICU) and number of medications on admission, divided into 3 groups, 0-5, 6-10, and 11 or more. We also included separate indicators for admissions on warfarin or chronic opioids. Charlson comorbidity score as well as heart failure, diabetes, and chronic obstructive pulmonary disease were included as separate variables, since these specific diagnoses have high comorbidity and risk of readmission.
Because the hospital’s medicine service is divided into subspecialty services, we included the admitting service and discharge unit to assess whether certain teams or units were associated with readmission. Discharge disposition was categorized as home with services (ie, physical therapy and visiting nurse), home without services, skilled nursing facility, acute care facility, or other. We included a variable to assess patient frailty and mobility based on the presence of a physical therapy consult. We incorporated social determinants of health, including insurance coverage (private insurance, Medicare, Medicaid, subsidized, or uninsured); per capita income from the patient’s zip code as a proxy for economic status (divided into quartiles for analysis); and substance abuse and alcohol abuse (based on International Classification of Diseases, 10th revision codes). We considered whether the discharge was on a weekday or weekend, and considered distance to the hospital in relation to Boston, either within route 128 (roughly within 15-20 miles of the medical center), within interstate 495 (roughly within 30-40 miles of the medical center), or beyond this. We considered but were unable to incorporate candidate variables that had inconsistent availability in the electronic medical record, such as the Braden score, level of independence with activities of daily living, nursing-determined fall risk, presence of a social work or nutrition consultation, CAGE questionnaire for alcohol abuse, delirium assessment score, the number of adults living in the home, the number of dependents, and marital status.
Analysis
We created a derivation cohort using admission data from September 1, 2013, through November 30, 2015. We used a backward selection process to include variables in the derivation model. Any variable associated with 30-day readmissions with a P value < 0.10 in univariate analyses was considered as a candidate variable. To be retained in the multivariable model, each variable was required to have a significant association with 30-day readmission at the P < 0.05 level. We used beta coefficients to create a numerical score reflective of probability of readmission.
We then created a validation cohort using admissions data between December 1, 2015, and August 31, 2016. We applied the scoring algorithm from the derivation cohort to the validation cohort and compared the discriminative ability of the 2 models using the area under the receiver operating characteristic (ROC) curve. We also compared the area under the ROC curve of our predictive model to the LACE index using the nonparametric approach of DeLong and colleagues.8,11
Results
The derivation cohort consisted of 7972 index admissions, of which 12.6% were readmitted within 30 days. The patient population was 45% female, 70% white, and 85% English-speaking, with an average age of 61.4 years (standard deviation, 18.1, Table 1). Most patients had either private insurance (43%) or Medicare (41%).
Many patients were medically complex: 21% required ICU care, 29% were taking 11 or more medications on admission, and 52% had a Charlson score of 2 or more. In the previous 6 months, 14% had an emergency department visit and 16% had an admission or overnight observation. The rate of drug or alcohol abuse was 13%. The majority of patients were discharged home without services (54%), while 23% were discharged home with services, 13% were discharged to a skilled nursing facility, and 9% were discharged to an acute care facility.
Factors Associated With 30-Day Readmission
Table 2 displays the univariate and multivariate associations with 30-day readmissions in the derivation cohort. Variables significantly associated with readmission in univariate analysis were Charlson comorbidity score, history of diabetes, ICU utilization, previous emergency department visit, inpatient or observation stay within 6 months, number of medications on admission, use of opioids on admission, a diagnosis of diabetes, type of insurance, physical therapy consultation, admitting service, and discharge disposition. Variables not significant in univariate analysis included warfarin use, alcohol/drug abuse, distance from the hospital, and demographic variables (age, sex, race, language, income by zip code).
Variables associated with readmission in the final multivariate analysis model included a Charlson score of 2 or higher (compared to a score of 0; odds ratio [OR], 1.36; 95% confidence interval [CI], 1.11-1.66); any ICU stay (OR, 1.29; 95% CI, 1.08-1.53); number of medications on admission (OR, 1.34; 95% CI, 1.11-1.61) for 11 or more compared with 5 or fewer medications; prior admission or overnight observation (OR, 1.89; 95% CI, 1.61-2.23); and disposition on discharge to an acute care facility (OR, 2.45; 95% CI, 1.96-3.06), skilled nursing facility (OR, 1.53; 95% CI, 1.23-1.89), or home with services (OR, 1.64; 95% CI, 1.38-1.94) compared with home discharge without services. The hospital service from which the patient was discharged was significantly associated with readmission; the subspecialty services with the highest odds ratios were bone marrow transplant/hematology (OR, 2.46; 95% CI, 1.77-3.40) and oncology (OR, 2.26; 95% CI, 1.67-3.05), as compared with general medicine/geriatrics.
Model Derivation and Validation
We utilized the beta coefficients from the multivariate analysis to create a scoring tool to predict the likelihood of 30-day readmission. We rounded each beta coefficient and calculated a readmission score by adding together the rounded beta coefficients of each of the significant variables. Table 3 presents the cumulative percentage of discharges at each score level, as well as the calculated cumulative percentage of potential readmissions. For example, in our population, a score of 6 or greater accounted for 18% of all discharges, but 36% of all 30-day readmissions.
The ROC curves for the derivation model and LACE index are shown in the Figure. The C statistic for the derivation cohort was 0.67, as compared with a C statistic of 0.63 for the calculated LACE index (P < 0.0001). The validation cohort had a C statistic similar to that of the derivation cohort (0.66).
Discussion
We developed a predictive model that can be used during admission to stratify patients for intensive case management and discharge planning. The model included Charlson score, ICU utilization, admission to inpatient services or observation, visits to the ED in the past 6 months, number of medications on admission, hospital service, and discharge disposition. The C statistic of 0.67 is better than that of the LACE predictive model for our population, although both reflect only modest predictive value.
While our model, which was developed and validated at a single institution, may not be generalizable to other institutions, the method of developing a readmission risk prediction model specific to an institution is readily replicable. While standardized tools for predicting readmission risk exist, they do not necessarily account for unique patient populations and medical complexity at individual institutions. We examined patients discharged over 3 years from the medicine services, creating a service-specific model. Our approach could lead to the widespread development of service- and institution-specific models that take into account the risks and resources appropriate to each patient population and setting.
Many of the factors included in our model were indicative of the patients’ medical complexity, with Charlson comorbidity score and the number of discharge medications strongly associated with readmission. In the derivation of the LACE model, many patients were middle-aged and independent in their activities of daily living, and more than 75% had a Charlson comorbidity score of 0.6.6 In our population, by contrast, the majority of patients had a Charlson score of 2 or greater. Health care utilization was a strong positive predictor in both our model and in LACE. The finding that the Charlson comorbidity score was a better predictor than any single chronic illness suggests that medical complexity and comorbidities increase the likelihood of admission more than any 1 chronic condition. Our model incorporates discharge disposition in readmission prediction, another factor associated with medical complexity and frailty.
A surprising finding in our study was the lack of association between social determinants of health, such as alcohol and drug abuse, and readmission risk. We posit several reasons that may account for this finding. First, the population served by the medical center may be too small or homogeneous with respect to social determinants of health to detect a difference in readmission risk. Second, markers available in the electronic medical record to determine social needs may be too crude to distinguish degrees of vulnerability that increase the risk of readmission. We do not discount the importance of social determinants of health as predictors of readmission risk, but we do acknowledge the limitations of the data incorporated in our model.
Predictive models are useful only if they can be incorporated into workflow to identify high-risk patients. Prior to developing and using our model, we used LACE inconsistently because it required length of stay as 1 of the variables. Because the variables in our model are collected and recorded routinely at admission in our electronic medical record, the readmission risk score is calculated and displayed in a daily high-risk patient report. This automated process has afforded a more consistent and reliable approach to readmission risk assessment than previous efforts to assess the LACE index. Case managers use the high-risk patient report to identify patients who require enhanced care coordination and discharge planning. Since the introduction of this predictive model, we have noted a 10% reduction in the hospital’s 30-day readmission rate.
This project was subject to several limitations. Because data on admissions to other facilities were unavailable, we may have underestimated the risk of readmission to other facilities. Our results may not be generalizable to other organizations, although we believe that the methods are readily replicable. The performance of the model and its replication with a validation cohort are strengths of the approach.
Conclusion
We created a hospital service–specific 30-day readmission prediction tool whose performance improved incrementally over the widely used LACE index. This research suggests that readmission prediction is highly context-specific and that organizations would do well to examine the readmission risk factors most pertinent to the populations they serve. We believe that “customized” readmission risk prediction models for particular services in specific hospitals may offer a superior method to identify high-risk patients who may benefit from individualized care planning. Future research is needed to understand how best to capture information about the attributes of vulnerable populations, so that this information can be incorporated into future risk models.
Corresponding author: Karen Freund, MD, Tufts Medical Center, 800 Washington St., Boston, MA 02111; kfreund@tuftsmedicalcenter.org.
Financial disclosures: None.
Funding for this work was provided by the Commonwealth of Massachusetts, Executive Office of Health and Human Services.
From Tufts Medical Center, Boston, MA (Dr. Trautwein and Dr. Schwartz contributed equally to this article).
Abstract
- Objective: To examine whether an institution- and service-specific readmission prediction instrument has improved performance compared to universal tools.
- Design: Retrospective cohort study.
- Setting: Academic medical center located in Boston, MA.
- Participants: Adult inpatients admitted to a medicine service.
- Measurements: Patient attributes, inpatient service assignment, and 30-day readmission rate.
- Results: Of 7972 index admissions, 12.6% were readmitted within 30 days. Thirty-day readmissions were associated with number of medications on admission (adjusted odds ratio [OR], 1.34; 95% confidence interval [CI], 1.11-1.61) for ≥ 11 compared with ≤ 5 medications; prior admission or overnight observation (OR, 1.89; 95% CI, 1.61-2.23); and discharge (OR, 2.45; 95% CI, 1.97-3.06) to an acute care facility compared to home without services. The subspecialty services with the highest risk of readmission were bone marrow transplant/hematology (OR, 2.46; 95% CI, 1.78-3.40) and oncology (OR, 2.26; 95% CI, 1.67-3.05), as compared with general medicine/geriatrics. The C statistic for the derivation cohort was 0.67, as compared with a C statistic of 0.63 for the LACE index.
- Conclusion: A hospital service–specific 30-day readmission prediction tool showed incrementally improved performance over the widely used LACE index.
Keywords: rehospitalization; quality of care; predictive model; hospital medicine.
Readmissions are a costly problem in the United States. The readmission rate among Medicare beneficiaries aged 65 years and older was 17.1 per 100 live discharges in 2016.1 Although both the relationship between readmissions and quality of care and the use of readmissions as a quality measure are contested,2 efforts to reduce potentially avoidable rehospitalizations have received widespread attention and readmissions are a focus of reimbursement reform.3-5
The LACE index is a widely used model for readmission risk prediction.6 Derived from a study of hospitalized patients in Ontario, Canada, LACE uses information about length of stay, acuity, comorbidities, and emergency department utilization to estimate readmission risk. Models such as LACE+, LACE-rt, and HOSPITAL have tried to improve on LACE’s performance, but models with strong discriminative ability are lacking.7-9 Institution-specific models exist,10 as do well-organized multicenter studies,7 but their generalizability is limited due to population differences or inclusion of data difficult to extract from patients in real-time, such as data gathered through a socioeconomic questionnaire.
Given hospital-specific differences in operational performance and patient population, we sought to develop a statistical model for 30-day readmission prediction and demonstrate the process that could be utilized by other institutions to identify high-risk patients for intensive case management and discharge planning.
Methods
Study Design
We conducted a retrospective cohort study of all admissions to the medicine service at a 415-bed teaching hospital in Boston, MA, from September 1, 2013, through August 31, 2016. Patients are admitted through the emergency department or directly admitted from hospital-based practices or a statewide network of private practices.
Data Collection
Data were abstracted from electronic medical and billing records from the first (index) admission for each patient during the study period. Thirty-day readmission was defined as an unplanned admission in the 30 days following the index discharge date. We excluded patients readmitted after leaving against medical advice and planned readmissions based on information in discharge summaries.
The study team identified candidate risk factors by referencing related published research and with input from a multidisciplinary task force charged with developing strategies to reduce 30-day readmissions. Task force members included attending and resident physicians, pharmacists, nurses, case managers, and administrators. The task force considered factors that could be extracted from the electronic medical record, including demographics, location of care, and clinical measures such as diagnostic codes, as well as data available in nursing, social work, and case management notes. Decisions regarding potential risk factors were reached within the group based on institutional experience, availability, and quality of data within the electronic record for specific variables, as well as published research on the subject, with the goal of selecting variables that could be easily identified before discharge and used to generate a predictive score for use in discharge planning.
Variables
Variables initially considered for inclusion in univariate analyses included demographic characteristics of age, gender, and a combined race/ethnicity variable, delineated as either non-Hispanic white, non-Hispanic black, Asian/Asian Indian, or Hispanic. Those with race listed as other or missing were set to missing. Primary language was categorized as English versus non-English. We included a number of variables related to the severity of the patient’s medical condition during the index admission, including any stay in an intensive care unit (ICU) and number of medications on admission, divided into 3 groups, 0-5, 6-10, and 11 or more. We also included separate indicators for admissions on warfarin or chronic opioids. Charlson comorbidity score as well as heart failure, diabetes, and chronic obstructive pulmonary disease were included as separate variables, since these specific diagnoses have high comorbidity and risk of readmission.
Because the hospital’s medicine service is divided into subspecialty services, we included the admitting service and discharge unit to assess whether certain teams or units were associated with readmission. Discharge disposition was categorized as home with services (ie, physical therapy and visiting nurse), home without services, skilled nursing facility, acute care facility, or other. We included a variable to assess patient frailty and mobility based on the presence of a physical therapy consult. We incorporated social determinants of health, including insurance coverage (private insurance, Medicare, Medicaid, subsidized, or uninsured); per capita income from the patient’s zip code as a proxy for economic status (divided into quartiles for analysis); and substance abuse and alcohol abuse (based on International Classification of Diseases, 10th revision codes). We considered whether the discharge was on a weekday or weekend, and considered distance to the hospital in relation to Boston, either within route 128 (roughly within 15-20 miles of the medical center), within interstate 495 (roughly within 30-40 miles of the medical center), or beyond this. We considered but were unable to incorporate candidate variables that had inconsistent availability in the electronic medical record, such as the Braden score, level of independence with activities of daily living, nursing-determined fall risk, presence of a social work or nutrition consultation, CAGE questionnaire for alcohol abuse, delirium assessment score, the number of adults living in the home, the number of dependents, and marital status.
Analysis
We created a derivation cohort using admission data from September 1, 2013, through November 30, 2015. We used a backward selection process to include variables in the derivation model. Any variable associated with 30-day readmissions with a P value < 0.10 in univariate analyses was considered as a candidate variable. To be retained in the multivariable model, each variable was required to have a significant association with 30-day readmission at the P < 0.05 level. We used beta coefficients to create a numerical score reflective of probability of readmission.
We then created a validation cohort using admissions data between December 1, 2015, and August 31, 2016. We applied the scoring algorithm from the derivation cohort to the validation cohort and compared the discriminative ability of the 2 models using the area under the receiver operating characteristic (ROC) curve. We also compared the area under the ROC curve of our predictive model to the LACE index using the nonparametric approach of DeLong and colleagues.8,11
Results
The derivation cohort consisted of 7972 index admissions, of which 12.6% were readmitted within 30 days. The patient population was 45% female, 70% white, and 85% English-speaking, with an average age of 61.4 years (standard deviation, 18.1, Table 1). Most patients had either private insurance (43%) or Medicare (41%).
Many patients were medically complex: 21% required ICU care, 29% were taking 11 or more medications on admission, and 52% had a Charlson score of 2 or more. In the previous 6 months, 14% had an emergency department visit and 16% had an admission or overnight observation. The rate of drug or alcohol abuse was 13%. The majority of patients were discharged home without services (54%), while 23% were discharged home with services, 13% were discharged to a skilled nursing facility, and 9% were discharged to an acute care facility.
Factors Associated With 30-Day Readmission
Table 2 displays the univariate and multivariate associations with 30-day readmissions in the derivation cohort. Variables significantly associated with readmission in univariate analysis were Charlson comorbidity score, history of diabetes, ICU utilization, previous emergency department visit, inpatient or observation stay within 6 months, number of medications on admission, use of opioids on admission, a diagnosis of diabetes, type of insurance, physical therapy consultation, admitting service, and discharge disposition. Variables not significant in univariate analysis included warfarin use, alcohol/drug abuse, distance from the hospital, and demographic variables (age, sex, race, language, income by zip code).
Variables associated with readmission in the final multivariate analysis model included a Charlson score of 2 or higher (compared to a score of 0; odds ratio [OR], 1.36; 95% confidence interval [CI], 1.11-1.66); any ICU stay (OR, 1.29; 95% CI, 1.08-1.53); number of medications on admission (OR, 1.34; 95% CI, 1.11-1.61) for 11 or more compared with 5 or fewer medications; prior admission or overnight observation (OR, 1.89; 95% CI, 1.61-2.23); and disposition on discharge to an acute care facility (OR, 2.45; 95% CI, 1.96-3.06), skilled nursing facility (OR, 1.53; 95% CI, 1.23-1.89), or home with services (OR, 1.64; 95% CI, 1.38-1.94) compared with home discharge without services. The hospital service from which the patient was discharged was significantly associated with readmission; the subspecialty services with the highest odds ratios were bone marrow transplant/hematology (OR, 2.46; 95% CI, 1.77-3.40) and oncology (OR, 2.26; 95% CI, 1.67-3.05), as compared with general medicine/geriatrics.
Model Derivation and Validation
We utilized the beta coefficients from the multivariate analysis to create a scoring tool to predict the likelihood of 30-day readmission. We rounded each beta coefficient and calculated a readmission score by adding together the rounded beta coefficients of each of the significant variables. Table 3 presents the cumulative percentage of discharges at each score level, as well as the calculated cumulative percentage of potential readmissions. For example, in our population, a score of 6 or greater accounted for 18% of all discharges, but 36% of all 30-day readmissions.
The ROC curves for the derivation model and LACE index are shown in the Figure. The C statistic for the derivation cohort was 0.67, as compared with a C statistic of 0.63 for the calculated LACE index (P < 0.0001). The validation cohort had a C statistic similar to that of the derivation cohort (0.66).
Discussion
We developed a predictive model that can be used during admission to stratify patients for intensive case management and discharge planning. The model included Charlson score, ICU utilization, admission to inpatient services or observation, visits to the ED in the past 6 months, number of medications on admission, hospital service, and discharge disposition. The C statistic of 0.67 is better than that of the LACE predictive model for our population, although both reflect only modest predictive value.
While our model, which was developed and validated at a single institution, may not be generalizable to other institutions, the method of developing a readmission risk prediction model specific to an institution is readily replicable. While standardized tools for predicting readmission risk exist, they do not necessarily account for unique patient populations and medical complexity at individual institutions. We examined patients discharged over 3 years from the medicine services, creating a service-specific model. Our approach could lead to the widespread development of service- and institution-specific models that take into account the risks and resources appropriate to each patient population and setting.
Many of the factors included in our model were indicative of the patients’ medical complexity, with Charlson comorbidity score and the number of discharge medications strongly associated with readmission. In the derivation of the LACE model, many patients were middle-aged and independent in their activities of daily living, and more than 75% had a Charlson comorbidity score of 0.6.6 In our population, by contrast, the majority of patients had a Charlson score of 2 or greater. Health care utilization was a strong positive predictor in both our model and in LACE. The finding that the Charlson comorbidity score was a better predictor than any single chronic illness suggests that medical complexity and comorbidities increase the likelihood of admission more than any 1 chronic condition. Our model incorporates discharge disposition in readmission prediction, another factor associated with medical complexity and frailty.
A surprising finding in our study was the lack of association between social determinants of health, such as alcohol and drug abuse, and readmission risk. We posit several reasons that may account for this finding. First, the population served by the medical center may be too small or homogeneous with respect to social determinants of health to detect a difference in readmission risk. Second, markers available in the electronic medical record to determine social needs may be too crude to distinguish degrees of vulnerability that increase the risk of readmission. We do not discount the importance of social determinants of health as predictors of readmission risk, but we do acknowledge the limitations of the data incorporated in our model.
Predictive models are useful only if they can be incorporated into workflow to identify high-risk patients. Prior to developing and using our model, we used LACE inconsistently because it required length of stay as 1 of the variables. Because the variables in our model are collected and recorded routinely at admission in our electronic medical record, the readmission risk score is calculated and displayed in a daily high-risk patient report. This automated process has afforded a more consistent and reliable approach to readmission risk assessment than previous efforts to assess the LACE index. Case managers use the high-risk patient report to identify patients who require enhanced care coordination and discharge planning. Since the introduction of this predictive model, we have noted a 10% reduction in the hospital’s 30-day readmission rate.
This project was subject to several limitations. Because data on admissions to other facilities were unavailable, we may have underestimated the risk of readmission to other facilities. Our results may not be generalizable to other organizations, although we believe that the methods are readily replicable. The performance of the model and its replication with a validation cohort are strengths of the approach.
Conclusion
We created a hospital service–specific 30-day readmission prediction tool whose performance improved incrementally over the widely used LACE index. This research suggests that readmission prediction is highly context-specific and that organizations would do well to examine the readmission risk factors most pertinent to the populations they serve. We believe that “customized” readmission risk prediction models for particular services in specific hospitals may offer a superior method to identify high-risk patients who may benefit from individualized care planning. Future research is needed to understand how best to capture information about the attributes of vulnerable populations, so that this information can be incorporated into future risk models.
Corresponding author: Karen Freund, MD, Tufts Medical Center, 800 Washington St., Boston, MA 02111; kfreund@tuftsmedicalcenter.org.
Financial disclosures: None.
Funding for this work was provided by the Commonwealth of Massachusetts, Executive Office of Health and Human Services.
1. Bailey MK, Weiss AJ, Barrett ML, Jiang HJ. Characteristics of 30-day all-cause hospital readmissions, 2010-2016. HCUP Statistical Brief #248. February 2019. Agency for Healthcare Research and Quality. Rockville, MD. www.hcup-us.ahrq.gov/reports/statbriefs/sb248-Hospital-Readmissions-2010-2016.pdf. Accessed December 12, 2019.
2. Esposito ML, Selker HP, Salem DN. Quantity over quality: how the rise in quality measures is not producing quality results. J Gen Intern Med. 2015;30:1204-1207.
3. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the medicare fee-for-service program. N Engl J Med. 2009;360:1418-1428.
4. Centers for Medicare and Medicaid. Readmissions-reduction-program. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed December 8, 2019.
5. Institute for Healthcare Improvement. Readmissions. Reduce avoidable readmissions. www.ihi.org/Topics/Readmissions/Pages/default.aspx. Accessed December 8, 2019.
6. Van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182:551-557.
7. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688-1698.
8. Robinson R, Hudali T. The HOSPITAL score and LACE index as predictors of 30 day readmission in a retrospective study at a university-affiliated community hospital. Peer J. 2017;5:e3137.
9. El Morr C, Ginsburg L, Nam S, Woollard S. Assessing the performance of a modified LACE index (LACE-rt) to predict unplanned readmission after discharge in a community teaching hospital. Interactive J Med Res. 2017;6:e2.
10. Yu S, Farooq F, Van Esbroeck A, et al. Predicting readmission risk with institution-specific prediction models. Artif Intell Med. 2015;65:89-96.
11. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;4:837-845.
1. Bailey MK, Weiss AJ, Barrett ML, Jiang HJ. Characteristics of 30-day all-cause hospital readmissions, 2010-2016. HCUP Statistical Brief #248. February 2019. Agency for Healthcare Research and Quality. Rockville, MD. www.hcup-us.ahrq.gov/reports/statbriefs/sb248-Hospital-Readmissions-2010-2016.pdf. Accessed December 12, 2019.
2. Esposito ML, Selker HP, Salem DN. Quantity over quality: how the rise in quality measures is not producing quality results. J Gen Intern Med. 2015;30:1204-1207.
3. Jencks SF, Williams MV, Coleman EA. Rehospitalizations among patients in the medicare fee-for-service program. N Engl J Med. 2009;360:1418-1428.
4. Centers for Medicare and Medicaid. Readmissions-reduction-program. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/Readmissions-Reduction-Program.html. Accessed December 8, 2019.
5. Institute for Healthcare Improvement. Readmissions. Reduce avoidable readmissions. www.ihi.org/Topics/Readmissions/Pages/default.aspx. Accessed December 8, 2019.
6. Van Walraven C, Dhalla IA, Bell C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182:551-557.
7. Kansagara D, Englander H, Salanitro A, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688-1698.
8. Robinson R, Hudali T. The HOSPITAL score and LACE index as predictors of 30 day readmission in a retrospective study at a university-affiliated community hospital. Peer J. 2017;5:e3137.
9. El Morr C, Ginsburg L, Nam S, Woollard S. Assessing the performance of a modified LACE index (LACE-rt) to predict unplanned readmission after discharge in a community teaching hospital. Interactive J Med Res. 2017;6:e2.
10. Yu S, Farooq F, Van Esbroeck A, et al. Predicting readmission risk with institution-specific prediction models. Artif Intell Med. 2015;65:89-96.
11. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;4:837-845.