Andrew D. Auerbach, MD

Admission to a hospital can be a stressful event,^1,2 and patients report having many concerns at the time of hospital admission.³ Over the last 20 years, the United States has widely adopted the hospitalist model of inpatient care. Although this model has clear benefits, it also has the potential to contribute to patient stress, as hospitalized patients generally lack preexisting relationships with their inpatient physicians.^4,5 In this changing hospital environment, defining and promoting effective medical communication has become an essential goal of both individual practitioners and medical centers.

Successful communication and strong therapeutic relationships with physicians support patients’ coping with illness-associated stress^6,7 as well as promote adherence to medical treatment plans.⁸ Empathy serves as an important building block of patient-centered communication and encourages a strong therapeutic alliance.⁹ Studies from primary care, oncology, and intensive care unit (ICU) settings indicate that physician empathy is associated with decreased emotional distress,^10,11 improved ratings of communication,¹² and even better medical outcomes.¹³

Prior work has shown that hospitalists, like other clinicians, underutilize empathy as a tool in their daily interactions with patients.^14-16 Our prior qualitative analysis of audio-recorded hospitalist-patient admission encounters indicated that how hospitalists respond to patient expressions of negative emotion influences relationships with patients and alignment around care plans.¹⁷ To determine whether empathic communication is associated with patient-reported outcomes in the hospitalist model, we quantitatively analyzed coded admission encounters and survey data to examine the association between hospitalists’ responses to patient expressions of negative emotion (anxiety, sadness, and anger) and patient anxiety and ratings of communication. Given the often-limited time hospitalists have to complete admission encounters, we also examined the association between response to emotion and encounter length.

METHODS

We analyzed data collected as part of an observational study of hospitalist-patient communication during hospital admission encounters¹⁴ to assess the association between the way physicians responded to patient expressions of negative emotion and patient anxiety, ratings of communication in the encounter, and encounter length. We collected data between August 2008 and March 2009 on the general medical service at 2 urban hospitals that are part of an academic medical center. Participants were attending hospitalists (not physician trainees), and patients admitted under participating hospitalists’ care who were able to communicate verbally in English and provide informed consent for the study. The institutional review board at the University of California, San Francisco approved the study; physician and patient participants provided written informed consent.

Enrollment and data collection has been described previously.¹⁷ Our cohort for this analysis included 76 patients of 27 physicians who completed encounter audio recordings and pre- and postencounter surveys. Following enrollment, patients completed a preencounter survey to collect demographic information and to measure their baseline anxiety via the State Anxiety Scale (STAI-S), which assesses transient anxious mood using 20 items answered on a 4-point scale for a final score range of 20 to 80.^10,18,19 We timed and audio-recorded admission encounters. Encounter recordings were obtained solely from patient interactions with attending hospitalists and did not take into account the time patients may have spent with other physicians, including trainees. After the encounter, patients completed postencounter surveys, which included the STAI-S and patients’ ratings of communication during the encounter. To rate communication, patients responded to 7 items on a 0- to 10-point scale that were derived from previous work (Table 1)^12,20,21; the anchors were “not at all” and “completely.” To identify patients with serious illness, which we used as a covariate in regression models, we asked physicians on a postencounter survey whether or not they “would be surprised by this patient’s death or admission to the ICU in the next year.”²²

As previously described, we professionally transcribed and coded the audio recordings.¹⁷ Following past work,^15,16,23-25 we identified patient expressions of negative emotion and categorized the initial hospitalist response to each expression. Table 2 shows examples to illustrate the coding scheme. We considered an empathic response to be one that directed further discussion toward a patient’s expressed negative emotion. A neutral response was one that directed discussion neither towards nor away from the expressed emotion, while a nonempathic physician response directed further discussion away from the patient’s emotion.¹⁵ To assess reliability, 2 coders independently coded a randomly selected 20% of encounters (n = 15); kappa statistics were 0.76 for patient expressions of emotion and 0.85 for physician responses, indicating substantial to almost perfect agreement.²⁶

We used regression models to assess the association between the number of each type of physician response (empathic, neutral, nonempathic) in an encounter and the following variables: (1) the change in the patient’s anxiety level, defined as the difference between the post- and preencounter STAI-S score (using linear regression); (2) patient ratings of the physician and encounter (using Poisson regression); and (3) encounter length (using linear regression). To assess each patient rating item, we utilized a single model that included frequencies for each type of physician response. For ratings of their encounters, most patients gave high ratings, resulting in a preponderance of 10/10 scores for several items. Thus, we focused on trying to understand “negativity,” meaning the minority of less than completely positive reactions. To do this, we analyzed reflected outcomes (defined as 10 minus the patient’s response) using zero-inflated Poisson regression models. This approach allowed us to distinguish between degrees of dissatisfaction and to determine whether additional change in ratings resulted from additional physician responses. Encounter length also demonstrated right skewness, which we addressed through log transformation; results for this are reported as percent change in the encounter length per physician response.

We considered physician as a clustering variable in the calculation of robust standard errors for all models. In addition, we included in each model covariates that were associated with the outcome at P ≤ 0.10, including patient gender, patient age, serious illness,²² preencounter anxiety, encounter length, and hospital. We considered P values < 0.05 to be statistically significant. We used Stata SE 13 (StataCorp LLC, College Station, TX) for all statistical analyses.

 

RESULTS

We analyzed data from admission encounters with 76 patients (consent rate 63%) and 27 hospitalists (consent rate 91%). Their characteristics are shown in Table 3. Median encounter length was 19 minutes (mean 21 minutes, range 3-68). Patients expressed negative emotion in 190 instances across all encounters; median number of expressions per encounter was 1 (range 0-14). Hospitalists responded empathically to 32% (n = 61) of the patient expressions, neutrally to 43% (n = 81), and nonempathically to 25% (n = 48).

The STAI-S was normally distributed. The mean preencounter STAI-S score was 39 (standard deviation [SD] 8.9). Mean postencounter STAI-S score was 38 (SD 10.7). Mean change in anxiety over the course of the encounter, calculated as the postencounter minus preencounter mean was −1.2 (SD 7.6). Table 1 shows summary statistics for the patient ratings of communication items. All items were rated highly. Across the items, between 51% and 78% of patients rated the highest score of 10.

Across the range of frequencies of emotional expressions per encounter in our data set (0-14 expressions), each additional empathic hospitalist response was associated with a 1.65-point decrease in the STAI-S (95% confidence interval [CI], 0.48-2.82). We did not find significant associations between changes in the STAI-S and the number of neutral hospitalist responses (−0.65 per response; 95% CI, −1.67-0.37) or nonempathic hospitalist responses (0.61 per response; 95% CI, −0.88-2.10).

The Figure shows the adjusted relative effects (aREs) and 95% CIs from zero-inflated multivariate Poisson regression models of the association between physician response to patient expressions of negative emotion and reflected patient ratings of the encounters, defined as 10 minus the patient’s response. Empathic hospitalist responses to patient expressions of emotion were associated with less negative patient ratings of communication in the encounter for 4 of 7 items: covering points of interest, the doctor listening, the doctor caring, and trusting the doctor. For example, for the item “I felt this doctor cared about me,” each empathic hospitalist response was associated with an approximate 77% reduction in negative patient ratings (aRE: 0.23; 95% CI, 0.06-0.85).

In addition, nonempathic responses were associated with more negative ratings of communication for 5 of the 7 items: ease of understanding information, covering points of interest, the doctor listening, the doctor caring, and trusting the doctor. For example, for the item “I felt this doctor cared about me,” each nonempathic hospitalist response was associated with a more than doubling of negative patient ratings (aRE: 2.3; 95% CI, 1.32-4.16). Neutral physician responses to patient expressions of negative emotion were associated with less negative patient ratings for 2 of the items: covering points of interest (aRE 0.68; 95% CI, 0.51-0.90) and trusting the doctor (aRE: 0.86; 95% CI, 0.75-0.99).

We did not find a statistical association between encounter length and the number of empathic hospitalist responses in the encounter (percent change in encounter length per response [PC]: 1%; 95% CI, −8%-10%) or the number of nonempathic responses (PC: 18%; 95% CI, −2%-42%). We did find a statistically significant association between the number of neutral responses and encounter length (PC: 13%; 95% CI, 3%-24%), corresponding to 2.5 minutes of additional encounter time per neutral response for the median encounter length of 19 minutes.

DISCUSSION

Our study set out to measure how hospitalists responded to expressions of negative emotion during admission encounters with patients and how those responses correlated with patient anxiety, ratings of communication, and encounter length. We found that empathic responses were associated with diminishing patient anxiety after the visit, as well as with better ratings of several domains of hospitalist communication. Moreover, nonempathic responses to negative emotion were associated with more strongly negative ratings of hospitalist communication. Finally, while clinicians may worry that encouraging patients to speak further about emotion will result in excessive visit lengths, we did not find a statistical association between empathic responses and encounter duration. To our knowledge, this is the first study to indicate an association between empathy and patient anxiety and communication ratings within the hospitalist model, which is rapidly becoming the predominant model for providing inpatient care in the United States.^4,5

As in oncologic care, anxiety is an emotion commonly confronted by clinicians meeting admitted medical patients for the first time. Studies show that not only do patient anxiety levels remain high throughout a hospital course, patients who experience higher levels of anxiety tend to stay longer in the hospital.^1,2,27-30 But unlike oncologic care or other therapy provided in an outpatient setting, the hospitalist model does not facilitate “continuity” of care, or the ability to care for the same patients over a long period of time. This reality of inpatient care makes rapid, effective rapport-building critical to establishing strong physician-patient relationships. In this setting, a simple communication tool that is potentially able to reduce inpatients’ anxiety could have a meaningful impact on hospitalist-provided care and patient outcomes.

In terms of the magnitude of the effect of empathic responses, the clinical significance of a 1.65-point decrease in the STAI-S anxiety score is not precisely clear. A prior study that examined the effect of music therapy on anxiety levels in patients with cancer found an average anxiety reduction of approximately 9.5 units on the STAIS-S scale after sensitivity analysis, suggesting a rather large meaningful effect size.³¹ Given we found a reduction of 1.65 points for each empathic response, however, with a range of 0-14 negative emotions expressed over a median 19-minute encounter, there is opportunity for hospitalists to achieve a clinically significant decrease in patient anxiety during an admission encounter. The potential to reduce anxiety is extended further when we consider that the impact of an empathic response may apply not just to the admission encounter alone but also to numerous other patient-clinician interactions over the course of a hospitalization.

A healthy body of communication research supports the associations we found in our study between empathy and patient ratings of communication and physicians. Families in ICU conferences rate communication more positively when physicians express empathy,¹² and a number of studies indicate an association between empathy and patient satisfaction in outpatient settings.⁸ Given the associations we found with negative ratings on the items in our study, promoting empathic responses to expressions of emotion and, more importantly, stressing avoidance of nonempathic responses may be relevant efforts in working to improve patient satisfaction scores on surveys reporting “top box” percentages, such as Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). More notably, evidence indicates that empathy has positive impacts beyond satisfaction surveys, such as adherence, better diagnostic and clinical outcomes, and strengthening of patient enablement.⁸Not all hospitalist responses to emotion were associated with patient ratings across the 7 communication items we assessed. For example, we did not find an association between how physicians responded to patient expressions of negative emotion and patient perception that enough time was spent in the visit or the degree to which talking with the doctor met a patient’s overall needs. It follows logically, and other research supports, that empathy would influence patient ratings of physician caring and trust,³² whereas other communication factors we were unable to measure (eg, physician body language, tone, and use of jargon and patient health literacy and primary language) may have a more significant association with patient ratings of the other items we assessed.

In considering the clinical application of our results, it is important to note that communication skills, including responding empathically to patient expressions of negative emotion, can be imparted through training in the same way as abdominal examination or electrocardiogram interpretation skills.^33-35 However, training of hospitalists in communication skills requires time and some financial investment on the part of the physician, their hospital or group, or, ideally, both. Effective training methods, like those for other skill acquisition, involve learner-centered teaching and practicing skills with role-play and feedback.³⁶ Given the importance of a learner-centered approach, learning would likely be better received and more effective if it was tailored to the specific needs and patient scenarios commonly encountered by hospitalist physicians. As these programs are developed, it will be important to assess the impact of any training on the patient-reported outcomes we assessed in this observational study, along with clinical outcomes.

Our study has several limitations. First, we were only able to evaluate whether hospitalists verbally responded to patient emotion and were thus not able to account for nonverbal empathy such as facial expressions, body language, or voice tone. Second, given our patient consent rate of 63%, patients who agreed to participate in the study may have had different opinions than those who declined to participate. Also, hospitalists and patients may have behaved differently as a result of being audio recorded. We only included patients who spoke English, and our patient population was predominately non-Hispanic white. Patients who spoke other languages or came from other cultural backgrounds may have had different responses. Third, we did not use a single validated scale for patient ratings of communication, and multiple analyses increase our risk of finding statistically significant associations by chance. The skewing of the communication rating items toward high scores may also have led to our results being driven by outliers, although the model we chose for analysis does penalize for this. Furthermore, our sample size was small, leading to wide CIs and potential for lack of statistical associations due to insufficient power. Our findings warrant replication in larger studies. Fourth, the setting of our study in an academic center may affect generalizability. Finally, the age of our data (collected between 2008 and 2009) is also a limitation. Given a recent focus on communication and patient experience since the initiation of HCAHPS feedback, a similar analysis of empathy and communication methods now may result in different outcomes.

In conclusion, our results suggest that enhancing hospitalists’ empathic responses to patient expressions of negative emotion could decrease patient anxiety and improve patients’ perceptions of (and thus possibly their relationships with) hospitalists, without sacrificing efficiency. Future work should focus on tailoring and implementing communication skills training programs for hospitalists and evaluating the impact of training on patient outcomes.

 

Acknowledgments

The authors extend their sincere thanks to the patients and physicians who participated in this study. Dr. Anderson was funded by the National Palliative Care Research Center and the University of California, San Francisco Clinical and Translational Science Institute Career Development Program, National Institutes of Health (NIH) grant number 5 KL2 RR024130-04. Project costs were funded by a grant from the University of California, San Francisco Academic Senate.

Disclosure

 All coauthors have seen and agree with the contents of this manuscript. This submission is not under review by any other publication. Wendy Anderson received funding for this project from the National Palliative Care Research Center, University of California San Francisco Clinical and Translational Science Institute (NIH grant number 5KL2RR024130-04), and the University of San Francisco Academic Senate [From Section 2 of Author Disclosure Form]. Andy Auerbach has a Patient-Centered Outcomes Research Institute research grant in development [From Section 3 of the Author Disclosure Form].

Admission to a hospital can be a stressful event,^1,2 and patients report having many concerns at the time of hospital admission.³ Over the last 20 years, the United States has widely adopted the hospitalist model of inpatient care. Although this model has clear benefits, it also has the potential to contribute to patient stress, as hospitalized patients generally lack preexisting relationships with their inpatient physicians.^4,5 In this changing hospital environment, defining and promoting effective medical communication has become an essential goal of both individual practitioners and medical centers.

Successful communication and strong therapeutic relationships with physicians support patients’ coping with illness-associated stress^6,7 as well as promote adherence to medical treatment plans.⁸ Empathy serves as an important building block of patient-centered communication and encourages a strong therapeutic alliance.⁹ Studies from primary care, oncology, and intensive care unit (ICU) settings indicate that physician empathy is associated with decreased emotional distress,^10,11 improved ratings of communication,¹² and even better medical outcomes.¹³

Prior work has shown that hospitalists, like other clinicians, underutilize empathy as a tool in their daily interactions with patients.^14-16 Our prior qualitative analysis of audio-recorded hospitalist-patient admission encounters indicated that how hospitalists respond to patient expressions of negative emotion influences relationships with patients and alignment around care plans.¹⁷ To determine whether empathic communication is associated with patient-reported outcomes in the hospitalist model, we quantitatively analyzed coded admission encounters and survey data to examine the association between hospitalists’ responses to patient expressions of negative emotion (anxiety, sadness, and anger) and patient anxiety and ratings of communication. Given the often-limited time hospitalists have to complete admission encounters, we also examined the association between response to emotion and encounter length.

METHODS

We analyzed data collected as part of an observational study of hospitalist-patient communication during hospital admission encounters¹⁴ to assess the association between the way physicians responded to patient expressions of negative emotion and patient anxiety, ratings of communication in the encounter, and encounter length. We collected data between August 2008 and March 2009 on the general medical service at 2 urban hospitals that are part of an academic medical center. Participants were attending hospitalists (not physician trainees), and patients admitted under participating hospitalists’ care who were able to communicate verbally in English and provide informed consent for the study. The institutional review board at the University of California, San Francisco approved the study; physician and patient participants provided written informed consent.

Enrollment and data collection has been described previously.¹⁷ Our cohort for this analysis included 76 patients of 27 physicians who completed encounter audio recordings and pre- and postencounter surveys. Following enrollment, patients completed a preencounter survey to collect demographic information and to measure their baseline anxiety via the State Anxiety Scale (STAI-S), which assesses transient anxious mood using 20 items answered on a 4-point scale for a final score range of 20 to 80.^10,18,19 We timed and audio-recorded admission encounters. Encounter recordings were obtained solely from patient interactions with attending hospitalists and did not take into account the time patients may have spent with other physicians, including trainees. After the encounter, patients completed postencounter surveys, which included the STAI-S and patients’ ratings of communication during the encounter. To rate communication, patients responded to 7 items on a 0- to 10-point scale that were derived from previous work (Table 1)^12,20,21; the anchors were “not at all” and “completely.” To identify patients with serious illness, which we used as a covariate in regression models, we asked physicians on a postencounter survey whether or not they “would be surprised by this patient’s death or admission to the ICU in the next year.”²²

As previously described, we professionally transcribed and coded the audio recordings.¹⁷ Following past work,^15,16,23-25 we identified patient expressions of negative emotion and categorized the initial hospitalist response to each expression. Table 2 shows examples to illustrate the coding scheme. We considered an empathic response to be one that directed further discussion toward a patient’s expressed negative emotion. A neutral response was one that directed discussion neither towards nor away from the expressed emotion, while a nonempathic physician response directed further discussion away from the patient’s emotion.¹⁵ To assess reliability, 2 coders independently coded a randomly selected 20% of encounters (n = 15); kappa statistics were 0.76 for patient expressions of emotion and 0.85 for physician responses, indicating substantial to almost perfect agreement.²⁶

We used regression models to assess the association between the number of each type of physician response (empathic, neutral, nonempathic) in an encounter and the following variables: (1) the change in the patient’s anxiety level, defined as the difference between the post- and preencounter STAI-S score (using linear regression); (2) patient ratings of the physician and encounter (using Poisson regression); and (3) encounter length (using linear regression). To assess each patient rating item, we utilized a single model that included frequencies for each type of physician response. For ratings of their encounters, most patients gave high ratings, resulting in a preponderance of 10/10 scores for several items. Thus, we focused on trying to understand “negativity,” meaning the minority of less than completely positive reactions. To do this, we analyzed reflected outcomes (defined as 10 minus the patient’s response) using zero-inflated Poisson regression models. This approach allowed us to distinguish between degrees of dissatisfaction and to determine whether additional change in ratings resulted from additional physician responses. Encounter length also demonstrated right skewness, which we addressed through log transformation; results for this are reported as percent change in the encounter length per physician response.

We considered physician as a clustering variable in the calculation of robust standard errors for all models. In addition, we included in each model covariates that were associated with the outcome at P ≤ 0.10, including patient gender, patient age, serious illness,²² preencounter anxiety, encounter length, and hospital. We considered P values < 0.05 to be statistically significant. We used Stata SE 13 (StataCorp LLC, College Station, TX) for all statistical analyses.

 

RESULTS

We analyzed data from admission encounters with 76 patients (consent rate 63%) and 27 hospitalists (consent rate 91%). Their characteristics are shown in Table 3. Median encounter length was 19 minutes (mean 21 minutes, range 3-68). Patients expressed negative emotion in 190 instances across all encounters; median number of expressions per encounter was 1 (range 0-14). Hospitalists responded empathically to 32% (n = 61) of the patient expressions, neutrally to 43% (n = 81), and nonempathically to 25% (n = 48).

The STAI-S was normally distributed. The mean preencounter STAI-S score was 39 (standard deviation [SD] 8.9). Mean postencounter STAI-S score was 38 (SD 10.7). Mean change in anxiety over the course of the encounter, calculated as the postencounter minus preencounter mean was −1.2 (SD 7.6). Table 1 shows summary statistics for the patient ratings of communication items. All items were rated highly. Across the items, between 51% and 78% of patients rated the highest score of 10.

Across the range of frequencies of emotional expressions per encounter in our data set (0-14 expressions), each additional empathic hospitalist response was associated with a 1.65-point decrease in the STAI-S (95% confidence interval [CI], 0.48-2.82). We did not find significant associations between changes in the STAI-S and the number of neutral hospitalist responses (−0.65 per response; 95% CI, −1.67-0.37) or nonempathic hospitalist responses (0.61 per response; 95% CI, −0.88-2.10).

The Figure shows the adjusted relative effects (aREs) and 95% CIs from zero-inflated multivariate Poisson regression models of the association between physician response to patient expressions of negative emotion and reflected patient ratings of the encounters, defined as 10 minus the patient’s response. Empathic hospitalist responses to patient expressions of emotion were associated with less negative patient ratings of communication in the encounter for 4 of 7 items: covering points of interest, the doctor listening, the doctor caring, and trusting the doctor. For example, for the item “I felt this doctor cared about me,” each empathic hospitalist response was associated with an approximate 77% reduction in negative patient ratings (aRE: 0.23; 95% CI, 0.06-0.85).

In addition, nonempathic responses were associated with more negative ratings of communication for 5 of the 7 items: ease of understanding information, covering points of interest, the doctor listening, the doctor caring, and trusting the doctor. For example, for the item “I felt this doctor cared about me,” each nonempathic hospitalist response was associated with a more than doubling of negative patient ratings (aRE: 2.3; 95% CI, 1.32-4.16). Neutral physician responses to patient expressions of negative emotion were associated with less negative patient ratings for 2 of the items: covering points of interest (aRE 0.68; 95% CI, 0.51-0.90) and trusting the doctor (aRE: 0.86; 95% CI, 0.75-0.99).

We did not find a statistical association between encounter length and the number of empathic hospitalist responses in the encounter (percent change in encounter length per response [PC]: 1%; 95% CI, −8%-10%) or the number of nonempathic responses (PC: 18%; 95% CI, −2%-42%). We did find a statistically significant association between the number of neutral responses and encounter length (PC: 13%; 95% CI, 3%-24%), corresponding to 2.5 minutes of additional encounter time per neutral response for the median encounter length of 19 minutes.

DISCUSSION

Our study set out to measure how hospitalists responded to expressions of negative emotion during admission encounters with patients and how those responses correlated with patient anxiety, ratings of communication, and encounter length. We found that empathic responses were associated with diminishing patient anxiety after the visit, as well as with better ratings of several domains of hospitalist communication. Moreover, nonempathic responses to negative emotion were associated with more strongly negative ratings of hospitalist communication. Finally, while clinicians may worry that encouraging patients to speak further about emotion will result in excessive visit lengths, we did not find a statistical association between empathic responses and encounter duration. To our knowledge, this is the first study to indicate an association between empathy and patient anxiety and communication ratings within the hospitalist model, which is rapidly becoming the predominant model for providing inpatient care in the United States.^4,5

As in oncologic care, anxiety is an emotion commonly confronted by clinicians meeting admitted medical patients for the first time. Studies show that not only do patient anxiety levels remain high throughout a hospital course, patients who experience higher levels of anxiety tend to stay longer in the hospital.^1,2,27-30 But unlike oncologic care or other therapy provided in an outpatient setting, the hospitalist model does not facilitate “continuity” of care, or the ability to care for the same patients over a long period of time. This reality of inpatient care makes rapid, effective rapport-building critical to establishing strong physician-patient relationships. In this setting, a simple communication tool that is potentially able to reduce inpatients’ anxiety could have a meaningful impact on hospitalist-provided care and patient outcomes.

In terms of the magnitude of the effect of empathic responses, the clinical significance of a 1.65-point decrease in the STAI-S anxiety score is not precisely clear. A prior study that examined the effect of music therapy on anxiety levels in patients with cancer found an average anxiety reduction of approximately 9.5 units on the STAIS-S scale after sensitivity analysis, suggesting a rather large meaningful effect size.³¹ Given we found a reduction of 1.65 points for each empathic response, however, with a range of 0-14 negative emotions expressed over a median 19-minute encounter, there is opportunity for hospitalists to achieve a clinically significant decrease in patient anxiety during an admission encounter. The potential to reduce anxiety is extended further when we consider that the impact of an empathic response may apply not just to the admission encounter alone but also to numerous other patient-clinician interactions over the course of a hospitalization.

A healthy body of communication research supports the associations we found in our study between empathy and patient ratings of communication and physicians. Families in ICU conferences rate communication more positively when physicians express empathy,¹² and a number of studies indicate an association between empathy and patient satisfaction in outpatient settings.⁸ Given the associations we found with negative ratings on the items in our study, promoting empathic responses to expressions of emotion and, more importantly, stressing avoidance of nonempathic responses may be relevant efforts in working to improve patient satisfaction scores on surveys reporting “top box” percentages, such as Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). More notably, evidence indicates that empathy has positive impacts beyond satisfaction surveys, such as adherence, better diagnostic and clinical outcomes, and strengthening of patient enablement.⁸Not all hospitalist responses to emotion were associated with patient ratings across the 7 communication items we assessed. For example, we did not find an association between how physicians responded to patient expressions of negative emotion and patient perception that enough time was spent in the visit or the degree to which talking with the doctor met a patient’s overall needs. It follows logically, and other research supports, that empathy would influence patient ratings of physician caring and trust,³² whereas other communication factors we were unable to measure (eg, physician body language, tone, and use of jargon and patient health literacy and primary language) may have a more significant association with patient ratings of the other items we assessed.

In considering the clinical application of our results, it is important to note that communication skills, including responding empathically to patient expressions of negative emotion, can be imparted through training in the same way as abdominal examination or electrocardiogram interpretation skills.^33-35 However, training of hospitalists in communication skills requires time and some financial investment on the part of the physician, their hospital or group, or, ideally, both. Effective training methods, like those for other skill acquisition, involve learner-centered teaching and practicing skills with role-play and feedback.³⁶ Given the importance of a learner-centered approach, learning would likely be better received and more effective if it was tailored to the specific needs and patient scenarios commonly encountered by hospitalist physicians. As these programs are developed, it will be important to assess the impact of any training on the patient-reported outcomes we assessed in this observational study, along with clinical outcomes.

Our study has several limitations. First, we were only able to evaluate whether hospitalists verbally responded to patient emotion and were thus not able to account for nonverbal empathy such as facial expressions, body language, or voice tone. Second, given our patient consent rate of 63%, patients who agreed to participate in the study may have had different opinions than those who declined to participate. Also, hospitalists and patients may have behaved differently as a result of being audio recorded. We only included patients who spoke English, and our patient population was predominately non-Hispanic white. Patients who spoke other languages or came from other cultural backgrounds may have had different responses. Third, we did not use a single validated scale for patient ratings of communication, and multiple analyses increase our risk of finding statistically significant associations by chance. The skewing of the communication rating items toward high scores may also have led to our results being driven by outliers, although the model we chose for analysis does penalize for this. Furthermore, our sample size was small, leading to wide CIs and potential for lack of statistical associations due to insufficient power. Our findings warrant replication in larger studies. Fourth, the setting of our study in an academic center may affect generalizability. Finally, the age of our data (collected between 2008 and 2009) is also a limitation. Given a recent focus on communication and patient experience since the initiation of HCAHPS feedback, a similar analysis of empathy and communication methods now may result in different outcomes.

In conclusion, our results suggest that enhancing hospitalists’ empathic responses to patient expressions of negative emotion could decrease patient anxiety and improve patients’ perceptions of (and thus possibly their relationships with) hospitalists, without sacrificing efficiency. Future work should focus on tailoring and implementing communication skills training programs for hospitalists and evaluating the impact of training on patient outcomes.

 

Acknowledgments

The authors extend their sincere thanks to the patients and physicians who participated in this study. Dr. Anderson was funded by the National Palliative Care Research Center and the University of California, San Francisco Clinical and Translational Science Institute Career Development Program, National Institutes of Health (NIH) grant number 5 KL2 RR024130-04. Project costs were funded by a grant from the University of California, San Francisco Academic Senate.

Disclosure

 All coauthors have seen and agree with the contents of this manuscript. This submission is not under review by any other publication. Wendy Anderson received funding for this project from the National Palliative Care Research Center, University of California San Francisco Clinical and Translational Science Institute (NIH grant number 5KL2RR024130-04), and the University of San Francisco Academic Senate [From Section 2 of Author Disclosure Form]. Andy Auerbach has a Patient-Centered Outcomes Research Institute research grant in development [From Section 3 of the Author Disclosure Form].

Admission to a hospital can be a stressful event,^1,2 and patients report having many concerns at the time of hospital admission.³ Over the last 20 years, the United States has widely adopted the hospitalist model of inpatient care. Although this model has clear benefits, it also has the potential to contribute to patient stress, as hospitalized patients generally lack preexisting relationships with their inpatient physicians.^4,5 In this changing hospital environment, defining and promoting effective medical communication has become an essential goal of both individual practitioners and medical centers.

Successful communication and strong therapeutic relationships with physicians support patients’ coping with illness-associated stress^6,7 as well as promote adherence to medical treatment plans.⁸ Empathy serves as an important building block of patient-centered communication and encourages a strong therapeutic alliance.⁹ Studies from primary care, oncology, and intensive care unit (ICU) settings indicate that physician empathy is associated with decreased emotional distress,^10,11 improved ratings of communication,¹² and even better medical outcomes.¹³

Prior work has shown that hospitalists, like other clinicians, underutilize empathy as a tool in their daily interactions with patients.^14-16 Our prior qualitative analysis of audio-recorded hospitalist-patient admission encounters indicated that how hospitalists respond to patient expressions of negative emotion influences relationships with patients and alignment around care plans.¹⁷ To determine whether empathic communication is associated with patient-reported outcomes in the hospitalist model, we quantitatively analyzed coded admission encounters and survey data to examine the association between hospitalists’ responses to patient expressions of negative emotion (anxiety, sadness, and anger) and patient anxiety and ratings of communication. Given the often-limited time hospitalists have to complete admission encounters, we also examined the association between response to emotion and encounter length.

METHODS

We analyzed data collected as part of an observational study of hospitalist-patient communication during hospital admission encounters¹⁴ to assess the association between the way physicians responded to patient expressions of negative emotion and patient anxiety, ratings of communication in the encounter, and encounter length. We collected data between August 2008 and March 2009 on the general medical service at 2 urban hospitals that are part of an academic medical center. Participants were attending hospitalists (not physician trainees), and patients admitted under participating hospitalists’ care who were able to communicate verbally in English and provide informed consent for the study. The institutional review board at the University of California, San Francisco approved the study; physician and patient participants provided written informed consent.

Enrollment and data collection has been described previously.¹⁷ Our cohort for this analysis included 76 patients of 27 physicians who completed encounter audio recordings and pre- and postencounter surveys. Following enrollment, patients completed a preencounter survey to collect demographic information and to measure their baseline anxiety via the State Anxiety Scale (STAI-S), which assesses transient anxious mood using 20 items answered on a 4-point scale for a final score range of 20 to 80.^10,18,19 We timed and audio-recorded admission encounters. Encounter recordings were obtained solely from patient interactions with attending hospitalists and did not take into account the time patients may have spent with other physicians, including trainees. After the encounter, patients completed postencounter surveys, which included the STAI-S and patients’ ratings of communication during the encounter. To rate communication, patients responded to 7 items on a 0- to 10-point scale that were derived from previous work (Table 1)^12,20,21; the anchors were “not at all” and “completely.” To identify patients with serious illness, which we used as a covariate in regression models, we asked physicians on a postencounter survey whether or not they “would be surprised by this patient’s death or admission to the ICU in the next year.”²²

As previously described, we professionally transcribed and coded the audio recordings.¹⁷ Following past work,^15,16,23-25 we identified patient expressions of negative emotion and categorized the initial hospitalist response to each expression. Table 2 shows examples to illustrate the coding scheme. We considered an empathic response to be one that directed further discussion toward a patient’s expressed negative emotion. A neutral response was one that directed discussion neither towards nor away from the expressed emotion, while a nonempathic physician response directed further discussion away from the patient’s emotion.¹⁵ To assess reliability, 2 coders independently coded a randomly selected 20% of encounters (n = 15); kappa statistics were 0.76 for patient expressions of emotion and 0.85 for physician responses, indicating substantial to almost perfect agreement.²⁶

We used regression models to assess the association between the number of each type of physician response (empathic, neutral, nonempathic) in an encounter and the following variables: (1) the change in the patient’s anxiety level, defined as the difference between the post- and preencounter STAI-S score (using linear regression); (2) patient ratings of the physician and encounter (using Poisson regression); and (3) encounter length (using linear regression). To assess each patient rating item, we utilized a single model that included frequencies for each type of physician response. For ratings of their encounters, most patients gave high ratings, resulting in a preponderance of 10/10 scores for several items. Thus, we focused on trying to understand “negativity,” meaning the minority of less than completely positive reactions. To do this, we analyzed reflected outcomes (defined as 10 minus the patient’s response) using zero-inflated Poisson regression models. This approach allowed us to distinguish between degrees of dissatisfaction and to determine whether additional change in ratings resulted from additional physician responses. Encounter length also demonstrated right skewness, which we addressed through log transformation; results for this are reported as percent change in the encounter length per physician response.

We considered physician as a clustering variable in the calculation of robust standard errors for all models. In addition, we included in each model covariates that were associated with the outcome at P ≤ 0.10, including patient gender, patient age, serious illness,²² preencounter anxiety, encounter length, and hospital. We considered P values < 0.05 to be statistically significant. We used Stata SE 13 (StataCorp LLC, College Station, TX) for all statistical analyses.

 

RESULTS

We analyzed data from admission encounters with 76 patients (consent rate 63%) and 27 hospitalists (consent rate 91%). Their characteristics are shown in Table 3. Median encounter length was 19 minutes (mean 21 minutes, range 3-68). Patients expressed negative emotion in 190 instances across all encounters; median number of expressions per encounter was 1 (range 0-14). Hospitalists responded empathically to 32% (n = 61) of the patient expressions, neutrally to 43% (n = 81), and nonempathically to 25% (n = 48).

The STAI-S was normally distributed. The mean preencounter STAI-S score was 39 (standard deviation [SD] 8.9). Mean postencounter STAI-S score was 38 (SD 10.7). Mean change in anxiety over the course of the encounter, calculated as the postencounter minus preencounter mean was −1.2 (SD 7.6). Table 1 shows summary statistics for the patient ratings of communication items. All items were rated highly. Across the items, between 51% and 78% of patients rated the highest score of 10.

Across the range of frequencies of emotional expressions per encounter in our data set (0-14 expressions), each additional empathic hospitalist response was associated with a 1.65-point decrease in the STAI-S (95% confidence interval [CI], 0.48-2.82). We did not find significant associations between changes in the STAI-S and the number of neutral hospitalist responses (−0.65 per response; 95% CI, −1.67-0.37) or nonempathic hospitalist responses (0.61 per response; 95% CI, −0.88-2.10).

The Figure shows the adjusted relative effects (aREs) and 95% CIs from zero-inflated multivariate Poisson regression models of the association between physician response to patient expressions of negative emotion and reflected patient ratings of the encounters, defined as 10 minus the patient’s response. Empathic hospitalist responses to patient expressions of emotion were associated with less negative patient ratings of communication in the encounter for 4 of 7 items: covering points of interest, the doctor listening, the doctor caring, and trusting the doctor. For example, for the item “I felt this doctor cared about me,” each empathic hospitalist response was associated with an approximate 77% reduction in negative patient ratings (aRE: 0.23; 95% CI, 0.06-0.85).

In addition, nonempathic responses were associated with more negative ratings of communication for 5 of the 7 items: ease of understanding information, covering points of interest, the doctor listening, the doctor caring, and trusting the doctor. For example, for the item “I felt this doctor cared about me,” each nonempathic hospitalist response was associated with a more than doubling of negative patient ratings (aRE: 2.3; 95% CI, 1.32-4.16). Neutral physician responses to patient expressions of negative emotion were associated with less negative patient ratings for 2 of the items: covering points of interest (aRE 0.68; 95% CI, 0.51-0.90) and trusting the doctor (aRE: 0.86; 95% CI, 0.75-0.99).

We did not find a statistical association between encounter length and the number of empathic hospitalist responses in the encounter (percent change in encounter length per response [PC]: 1%; 95% CI, −8%-10%) or the number of nonempathic responses (PC: 18%; 95% CI, −2%-42%). We did find a statistically significant association between the number of neutral responses and encounter length (PC: 13%; 95% CI, 3%-24%), corresponding to 2.5 minutes of additional encounter time per neutral response for the median encounter length of 19 minutes.

DISCUSSION

Our study set out to measure how hospitalists responded to expressions of negative emotion during admission encounters with patients and how those responses correlated with patient anxiety, ratings of communication, and encounter length. We found that empathic responses were associated with diminishing patient anxiety after the visit, as well as with better ratings of several domains of hospitalist communication. Moreover, nonempathic responses to negative emotion were associated with more strongly negative ratings of hospitalist communication. Finally, while clinicians may worry that encouraging patients to speak further about emotion will result in excessive visit lengths, we did not find a statistical association between empathic responses and encounter duration. To our knowledge, this is the first study to indicate an association between empathy and patient anxiety and communication ratings within the hospitalist model, which is rapidly becoming the predominant model for providing inpatient care in the United States.^4,5

As in oncologic care, anxiety is an emotion commonly confronted by clinicians meeting admitted medical patients for the first time. Studies show that not only do patient anxiety levels remain high throughout a hospital course, patients who experience higher levels of anxiety tend to stay longer in the hospital.^1,2,27-30 But unlike oncologic care or other therapy provided in an outpatient setting, the hospitalist model does not facilitate “continuity” of care, or the ability to care for the same patients over a long period of time. This reality of inpatient care makes rapid, effective rapport-building critical to establishing strong physician-patient relationships. In this setting, a simple communication tool that is potentially able to reduce inpatients’ anxiety could have a meaningful impact on hospitalist-provided care and patient outcomes.

In terms of the magnitude of the effect of empathic responses, the clinical significance of a 1.65-point decrease in the STAI-S anxiety score is not precisely clear. A prior study that examined the effect of music therapy on anxiety levels in patients with cancer found an average anxiety reduction of approximately 9.5 units on the STAIS-S scale after sensitivity analysis, suggesting a rather large meaningful effect size.³¹ Given we found a reduction of 1.65 points for each empathic response, however, with a range of 0-14 negative emotions expressed over a median 19-minute encounter, there is opportunity for hospitalists to achieve a clinically significant decrease in patient anxiety during an admission encounter. The potential to reduce anxiety is extended further when we consider that the impact of an empathic response may apply not just to the admission encounter alone but also to numerous other patient-clinician interactions over the course of a hospitalization.

A healthy body of communication research supports the associations we found in our study between empathy and patient ratings of communication and physicians. Families in ICU conferences rate communication more positively when physicians express empathy,¹² and a number of studies indicate an association between empathy and patient satisfaction in outpatient settings.⁸ Given the associations we found with negative ratings on the items in our study, promoting empathic responses to expressions of emotion and, more importantly, stressing avoidance of nonempathic responses may be relevant efforts in working to improve patient satisfaction scores on surveys reporting “top box” percentages, such as Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). More notably, evidence indicates that empathy has positive impacts beyond satisfaction surveys, such as adherence, better diagnostic and clinical outcomes, and strengthening of patient enablement.⁸Not all hospitalist responses to emotion were associated with patient ratings across the 7 communication items we assessed. For example, we did not find an association between how physicians responded to patient expressions of negative emotion and patient perception that enough time was spent in the visit or the degree to which talking with the doctor met a patient’s overall needs. It follows logically, and other research supports, that empathy would influence patient ratings of physician caring and trust,³² whereas other communication factors we were unable to measure (eg, physician body language, tone, and use of jargon and patient health literacy and primary language) may have a more significant association with patient ratings of the other items we assessed.

In considering the clinical application of our results, it is important to note that communication skills, including responding empathically to patient expressions of negative emotion, can be imparted through training in the same way as abdominal examination or electrocardiogram interpretation skills.^33-35 However, training of hospitalists in communication skills requires time and some financial investment on the part of the physician, their hospital or group, or, ideally, both. Effective training methods, like those for other skill acquisition, involve learner-centered teaching and practicing skills with role-play and feedback.³⁶ Given the importance of a learner-centered approach, learning would likely be better received and more effective if it was tailored to the specific needs and patient scenarios commonly encountered by hospitalist physicians. As these programs are developed, it will be important to assess the impact of any training on the patient-reported outcomes we assessed in this observational study, along with clinical outcomes.

Our study has several limitations. First, we were only able to evaluate whether hospitalists verbally responded to patient emotion and were thus not able to account for nonverbal empathy such as facial expressions, body language, or voice tone. Second, given our patient consent rate of 63%, patients who agreed to participate in the study may have had different opinions than those who declined to participate. Also, hospitalists and patients may have behaved differently as a result of being audio recorded. We only included patients who spoke English, and our patient population was predominately non-Hispanic white. Patients who spoke other languages or came from other cultural backgrounds may have had different responses. Third, we did not use a single validated scale for patient ratings of communication, and multiple analyses increase our risk of finding statistically significant associations by chance. The skewing of the communication rating items toward high scores may also have led to our results being driven by outliers, although the model we chose for analysis does penalize for this. Furthermore, our sample size was small, leading to wide CIs and potential for lack of statistical associations due to insufficient power. Our findings warrant replication in larger studies. Fourth, the setting of our study in an academic center may affect generalizability. Finally, the age of our data (collected between 2008 and 2009) is also a limitation. Given a recent focus on communication and patient experience since the initiation of HCAHPS feedback, a similar analysis of empathy and communication methods now may result in different outcomes.

In conclusion, our results suggest that enhancing hospitalists’ empathic responses to patient expressions of negative emotion could decrease patient anxiety and improve patients’ perceptions of (and thus possibly their relationships with) hospitalists, without sacrificing efficiency. Future work should focus on tailoring and implementing communication skills training programs for hospitalists and evaluating the impact of training on patient outcomes.

 

Acknowledgments

The authors extend their sincere thanks to the patients and physicians who participated in this study. Dr. Anderson was funded by the National Palliative Care Research Center and the University of California, San Francisco Clinical and Translational Science Institute Career Development Program, National Institutes of Health (NIH) grant number 5 KL2 RR024130-04. Project costs were funded by a grant from the University of California, San Francisco Academic Senate.

Disclosure

 All coauthors have seen and agree with the contents of this manuscript. This submission is not under review by any other publication. Wendy Anderson received funding for this project from the National Palliative Care Research Center, University of California San Francisco Clinical and Translational Science Institute (NIH grant number 5KL2RR024130-04), and the University of San Francisco Academic Senate [From Section 2 of Author Disclosure Form]. Andy Auerbach has a Patient-Centered Outcomes Research Institute research grant in development [From Section 3 of the Author Disclosure Form].

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

METHODS

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

 

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

 

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

RESULTS

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Our entire cohort of 30,726 unique patients admitted during the study period included 260 (0.8%) who experienced hospital-acquired VTEs (Table 3). In patients receiving no pharmacologic prophylaxis, the average APPS was 4.0 (SD, 2.4) for those without VTE and 7.1 (SD, 2.3) for those with VTE. In patients who had received pharmacologic prophylaxis, those without hospital-acquired VTE had an average APPS of 4.9 (SD, 2.6) and those with hospital-acquired VTE averaged 7.7 (SD, 2.6). APPS’ ROC curves for “no pharmacologic prophylaxis” had an AUC = 0.81 (CI, 0.79 – 0.83) that was significant (P < 0.001) with a cutoff value of 5. There was similar performance in the pharmacologic prophylaxis group with an AUC = 0.79 (CI, 0.76 – 0.82) and cutoff value of 5, as shown in the Figure. Over the entire cohort, APPS had a sensitivity of 85.4%, specificity of 53.3%, positive predictive value (PPV) of 1.5%, and a negative predictive value (NPV) of 99.8% when using a cutoff of 5. The average APPS calculation time was 0.03 seconds per encounter. Additional information on individual criteria can be found in Table 3.

 

DISCUSSION

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

The APPS was found to have near-equal performance in the pharmacologic vs. no pharmacologic prophylaxis cohorts. This finding agrees with a study that found no significant difference in predicting 90-day VTE when looking at 86 risk factors vs. the most significant 4, none of which related to prescribed prophylaxis.¹⁸ The original PPS had a reported sensitivity of 94.6%, specificity 62%, PPV 7.5%, and NPV 99.7% in its derivation cohort.¹³ We matched APPS to the ratio of sensitivity to specificity, using 5 as the cutoff value. APPS performed slightly worse with sensitivity of 85.4%, specificity 53.3%, PPV 1.5%, and NPV 99.8%. This difference may have resulted from the original PPS study’s use of 90-day follow-up to determine VTE occurrence, whereas we looked only until the end of current hospitalization, an average of 9.2 days. Furthermore, the PPS had significantly poorer performance (AUC = 0.62) than that seen in the original derivation cohort in a separate study that manually calculated the score on more than 1000 patients.¹⁵

There are important limitations to our study. It was done at a single academic institution using a dataset of VTE-associated, validated research that was well-known to the researchers.²⁰ Another major limitation is the dependence of the algorithm on data available within the first 4 hours of admission and earlier; thus, previous encounters may frequently play an important role. Patients presenting to our health system for the first time would have significantly fewer data available at the time of calculation. Additionally, our data could not reliably tell us the total doses of pharmacologic prophylaxis that a patient received. While most patients will maintain a consistent VTEP regimen once initiated in the hospital, 2 patients with the same LOS may have received differing amounts of pharmacologic prophylaxis. This research study did not assess how much time automatic calculation of VTE risk might save providers, because we did not record the time for each manual abstraction; however, from discussion with the main abstracter, chart review and manual calculation for this study took from 2 to 14 minutes per patient, depending on the number of previous interactions with the health system. Finally, although we chose data elements that are likely to exist at most institutions using an EHR, many institutions’ EHRs do not have EDW capabilities nor programmers who can assist with an automated risk score.

The EHR interventions to assist providers in determining appropriate VTEP have been able to increase rates of VTEP and decrease VTE-associated mortality.^16,21 In addition to automating the calculation of guideline-adherent risk scores, there is a need for wider adoption for clinical decision support for VTE. For this reason, we chose only structured data fields from some of the most common elements within our EHR’s data warehouse to derive APPS (Appendix 1). Our study supports the idea that automated calculation of scores requiring input of more complex data such as diagnoses, recent medical events, and current clinical status remains predictive of hospital-acquired VTE risk. Because it is calculated automatically in the background while the clinician completes his or her assessment, the APPS holds the potential to significantly reduce the burden on providers while making guideline-adherent risk assessment more readily accessible. Further research is required to determine the exact amount of time automatic calculation saves, and, more important, if the relatively high predictive capacity we observed using APPS would be reproducible across institutions and could reduce incidence of hospital-acquired VTE.

 

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

METHODS

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

 

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

 

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

RESULTS

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Our entire cohort of 30,726 unique patients admitted during the study period included 260 (0.8%) who experienced hospital-acquired VTEs (Table 3). In patients receiving no pharmacologic prophylaxis, the average APPS was 4.0 (SD, 2.4) for those without VTE and 7.1 (SD, 2.3) for those with VTE. In patients who had received pharmacologic prophylaxis, those without hospital-acquired VTE had an average APPS of 4.9 (SD, 2.6) and those with hospital-acquired VTE averaged 7.7 (SD, 2.6). APPS’ ROC curves for “no pharmacologic prophylaxis” had an AUC = 0.81 (CI, 0.79 – 0.83) that was significant (P < 0.001) with a cutoff value of 5. There was similar performance in the pharmacologic prophylaxis group with an AUC = 0.79 (CI, 0.76 – 0.82) and cutoff value of 5, as shown in the Figure. Over the entire cohort, APPS had a sensitivity of 85.4%, specificity of 53.3%, positive predictive value (PPV) of 1.5%, and a negative predictive value (NPV) of 99.8% when using a cutoff of 5. The average APPS calculation time was 0.03 seconds per encounter. Additional information on individual criteria can be found in Table 3.

 

DISCUSSION

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

The APPS was found to have near-equal performance in the pharmacologic vs. no pharmacologic prophylaxis cohorts. This finding agrees with a study that found no significant difference in predicting 90-day VTE when looking at 86 risk factors vs. the most significant 4, none of which related to prescribed prophylaxis.¹⁸ The original PPS had a reported sensitivity of 94.6%, specificity 62%, PPV 7.5%, and NPV 99.7% in its derivation cohort.¹³ We matched APPS to the ratio of sensitivity to specificity, using 5 as the cutoff value. APPS performed slightly worse with sensitivity of 85.4%, specificity 53.3%, PPV 1.5%, and NPV 99.8%. This difference may have resulted from the original PPS study’s use of 90-day follow-up to determine VTE occurrence, whereas we looked only until the end of current hospitalization, an average of 9.2 days. Furthermore, the PPS had significantly poorer performance (AUC = 0.62) than that seen in the original derivation cohort in a separate study that manually calculated the score on more than 1000 patients.¹⁵

There are important limitations to our study. It was done at a single academic institution using a dataset of VTE-associated, validated research that was well-known to the researchers.²⁰ Another major limitation is the dependence of the algorithm on data available within the first 4 hours of admission and earlier; thus, previous encounters may frequently play an important role. Patients presenting to our health system for the first time would have significantly fewer data available at the time of calculation. Additionally, our data could not reliably tell us the total doses of pharmacologic prophylaxis that a patient received. While most patients will maintain a consistent VTEP regimen once initiated in the hospital, 2 patients with the same LOS may have received differing amounts of pharmacologic prophylaxis. This research study did not assess how much time automatic calculation of VTE risk might save providers, because we did not record the time for each manual abstraction; however, from discussion with the main abstracter, chart review and manual calculation for this study took from 2 to 14 minutes per patient, depending on the number of previous interactions with the health system. Finally, although we chose data elements that are likely to exist at most institutions using an EHR, many institutions’ EHRs do not have EDW capabilities nor programmers who can assist with an automated risk score.

The EHR interventions to assist providers in determining appropriate VTEP have been able to increase rates of VTEP and decrease VTE-associated mortality.^16,21 In addition to automating the calculation of guideline-adherent risk scores, there is a need for wider adoption for clinical decision support for VTE. For this reason, we chose only structured data fields from some of the most common elements within our EHR’s data warehouse to derive APPS (Appendix 1). Our study supports the idea that automated calculation of scores requiring input of more complex data such as diagnoses, recent medical events, and current clinical status remains predictive of hospital-acquired VTE risk. Because it is calculated automatically in the background while the clinician completes his or her assessment, the APPS holds the potential to significantly reduce the burden on providers while making guideline-adherent risk assessment more readily accessible. Further research is required to determine the exact amount of time automatic calculation saves, and, more important, if the relatively high predictive capacity we observed using APPS would be reproducible across institutions and could reduce incidence of hospital-acquired VTE.

 

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.

Hospital-acquired venous thromboembolism (VTE) continues to be a critical quality challenge for U.S. hospitals,¹ and high-risk patients are often not adequately prophylaxed. Use of VTE prophylaxis (VTEP) varies as widely as 26% to 85% of patients in various studies, as does patient outcomes and care expenditures.^2-6 The 9th edition of the American College of Chest Physicians (CHEST) guidelines⁷ recommend the Padua Prediction Score (PPS) to select individual patients who may be at high risk for venous thromboembolism (VTE) and could benefit from thromboprophylaxis. Use of the manually calculated PPS to select patients for thromboprophylaxis has been shown to help decrease 30-day and 90-day mortality associated with VTE events after hospitalization to medical services.⁸ However, the PPS requires time-consuming manual calculation by a provider, who may be focused on more immediate aspects of patient care and several other risk scores competing for his attention, potentially decreasing its use.

Other risk scores that use only discrete scalar data, such as vital signs and lab results to predict early recognition of sepsis, have been successfully automated and implemented within electronic health records (EHRs).^9-11 Successful automation of scores requiring input of diagnoses, recent medical events, and current clinical status such as the PPS remains difficult.¹² Data representing these characteristics are more prone to error, and harder to translate clearly into a single data field than discrete elements like heart rate, potentially impacting validity of the calculated result.¹³ To improve usage of guideline based VTE risk assessment and decrease physician burden, we developed an algorithm called Automated Padua Prediction Score (APPS) that automatically calculates the PPS using only EHR data available within prior encounters and the first 4 hours of admission, a similar timeframe to when admitting providers would be entering orders. Our goal was to assess if an automatically calculated version of the PPS, a score that depends on criteria more complex than vital signs and labs, would accurately assess risk for hospital-acquired VTE when compared to traditional manual calculation of the Padua Prediction Score by a provider.

METHODS

Site Description and Ethics

The study was conducted at University of California, San Francisco Medical Center, a 790-bed academic hospital; its Institutional Review Board approved the study and collection of data via chart review. Handling of patient information complied with the Health Insurance Portability and Accountability Act of 1996.

 

Patient Inclusion

Adult patients admitted to a medical or surgical service between July 1, 2012 and April 1, 2014 were included in the study if they were candidates for VTEP, defined as: length of stay (LOS) greater than 2 days, not on hospice care, not pregnant at admission, no present on admission VTE diagnosis, no known contraindications to prophylaxis (eg, gastrointestinal bleed), and were not receiving therapeutic doses of warfarin, low molecular weight heparins, heparin, or novel anticoagulants prior to admission.

Data Sources

Clinical variables were extracted from the EHR’s enterprise data warehouse (EDW) by SQL Server query (Microsoft, Redmond, Washington) and deposited in a secure database. Chart review was conducted by a trained researcher (Mr. Jacolbia) using the EHR and a standardized protocol. Findings were recorded using REDCap (REDCap Consortium, Vanderbilt University, Nashville, Tennessee). The specific ICD-9, procedure, and lab codes used to determine each criterion of APPS are available in the Appendix.

Creation of the Automated Padua Prediction Score (APPS)

We developed APPS from the original 11 criteria that comprise the Padua Prediction Score: active cancer, previous VTE (excluding superficial vein thrombosis), reduced mobility, known thrombophilic condition, recent (1 month or less) trauma and/or surgery, age 70 years or older, heart and/or respiratory failure, acute myocardial infarction and/or ischemic stroke, acute infection and/or rheumatologic disorder, body mass index (BMI) 30 or higher, and ongoing hormonal treatment.¹³ APPS has the same scoring methodology as PPS: criteria are weighted from 1 to 3 points and summed with a maximum score of 20, representing highest risk of VTE. To automate the score calculation from data routinely available in the EHR, APPS checks pre-selected structured data fields for specific values within laboratory results, orders, nursing flowsheets and claims. Claims data included all ICD-9 and procedure codes used for billing purposes. If any of the predetermined data elements are found, then the specific criterion is considered positive; otherwise, it is scored as negative. The creators of the PPS were consulted in the generation of these data queries to replicate the original standards for deeming a criterion positive. The automated calculation required no use of natural language processing.

Characterization of Study Population

We recorded patient demographics (age, race, gender, BMI), LOS, and rate of hospital-acquired VTE. These patients were separated into 2 cohorts determined by the VTE prophylaxis they received. The risk profile of patients who received pharmacologic prophylaxis was hypothesized to be inherently different from those who had not. To evaluate APPS within this heterogeneous cohort, patients were divided into 2 major categories: pharmacologic vs. no pharmacologic prophylaxis. If they had a completed order or medication administration record on the institution’s approved formulary for pharmacologic VTEP, they were considered to have received pharmacologic prophylaxis. If they had only a completed order for usage of mechanical prophylaxis (sequential compression devices) or no evidence of any form of VTEP, they were considered to have received no pharmacologic prophylaxis. Patients with evidence of both pharmacologic and mechanical were placed in the pharmacologic prophylaxis group. To ensure that automated designation of prophylaxis group was accurate, we reviewed 40 randomly chosen charts because prior researchers were able to achieve sensitivity and specificity greater than 90% with that sample size.¹⁴

The primary outcome of hospital-acquired VTE was defined as an ICD-9 code for VTE (specific codes are found in the Appendix) paired with a “present on admission = no” flag on that encounter’s hospital billing data, abstracted from the EDW. A previous study at this institution used the same methodology and found 212/226 (94%) of patients with a VTE ICD-9 code on claim had evidence of a hospital-acquired VTE event upon chart review.¹⁴ Chart review was also completed to ensure that the primary outcome of newly discovered hospital-acquired VTE was differentiated from chronic VTE or history of VTE. Theoretically, ICD-9 codes and other data elements treat chronic VTE, history of VTE, and hospital-acquired VTE as distinct diagnoses, but it was unclear if this was true in our dataset. For 75 randomly selected cases of presumed hospital-acquired VTE, charts were reviewed for evidence that confirmed newly found VTE during that encounter.

Validation of APPS through Comparison to Manual Calculation of the Original PPS

To compare our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on 300 random patients, a subsample of the entire study cohort. The largest study we could find had manually calculated the PPS of 1,080 hospitalized patients with a mean PPS of 4.86 (standard deviation [SD], 2.26).¹⁵ One researcher (Mr. Jacolbia) accessed the EHR with all patient information available to physicians, including admission notes, orders, labs, flowsheets, past medical history, and all prior encounters to calculate and record the PPS. To limit potential score bias, 2 authors (Drs. Elias and Davies) assessed 30 randomly selected charts from the cohort of 300. The standardized chart review protocol mimicked a physician’s approach to determine if a patient met a criterion, such as concluding if he/she had active cancer by examining medication lists for chemotherapy, procedure notes for radiation, and recent diagnoses on problem lists. After the original PPS was manually calculated, APPS was automatically calculated for the same 300 patients. We intended to characterize similarities and differences between APPS and manual calculation prior to investigating APPS’ predictive capacity for the entire study population, because it would not be feasible to manually calculate the PPS for all 30,726 patients.

 

Statistical Analysis

For the 75 randomly selected cases of presumed hospital-acquired VTE, the number of cases was chosen by powering our analysis to find a difference in proportion of 20% with 90% power, α = 0.05 (two-sided). We conducted χ² tests on the entire study cohort to determine if there were significant differences in demographics, LOS, and incidence of hospital-acquired VTE by prophylaxis received. For both the pharmacologic and the no pharmacologic prophylaxis groups, we conducted 2-sample Student t tests to determine significant differences in demographics and LOS between patients who experienced a hospital-acquired VTE and those who did not.

For the comparison of our automated calculation to standard clinical practice, we manually calculated the PPS through chart review within the first 2 days of admission on a subsample of 300 random patients. We powered our analysis to detect a difference in mean PPS from 4.86 to 4.36, enough to alter the point value, with 90% power and α = 0.05 (two-sided) and found 300 patients to be comfortably above the required sample size. We compared APPS and manual calculation in the 300-patient cohort using: 2-sample Student t tests to compare mean scores, χ² tests to compare the frequency with which criteria were positive, and receiver operating characteristic (ROC) curves to determine capacity to predict a hospital-acquired VTE event. Pearson’s correlation was also completed to assess score agreement between APPS and manual calculation on a per-patient basis. After comparing automated calculation of APPS to manual chart review on the same 300 patients, we used APPS to calculate scores for the entire study cohort (n = 30,726). We calculated the mean of APPS by prophylaxis group and whether hospital-acquired VTE had occurred. We analyzed APPS’ ROC curve statistics by prophylaxis group to determine its overall predictive capacity in our study population. Lastly, we computed the time required to calculate APPS per patient. Statistical analyses were conducted using SPSS Statistics (IBM, Armonk, New York) and Python 2.7 (Python Software Foundation, Beaverton, Oregon); 95% confidence intervals (CI) and (SD) were reported when appropriate.

RESULTS

Among the 30,726 unique patients in our entire cohort (all patients admitted during the time period who met the study criteria), we found 6574 (21.4%) on pharmacologic (with or without mechanical) prophylaxis, 13,511 (44.0%) on mechanical only, and 10,641 (34.6%) on no prophylaxis. χ² tests found no significant differences in demographics, LOS, or incidence of hospital-acquired VTE between the patients who received mechanical prophylaxis only and those who received no prophylaxis (Table 1). Similarly, there were no differences in these characteristics in patients receiving pharmacologic prophylaxis with or without the addition of mechanical prophylaxis. Designation of prophylaxis group by manual chart review vs. our automated process was found to agree in categorization for 39/40 (97.5%) sampled encounters. When comparing the cohort that received pharmacologic prophylaxis against the cohort that did not, there were significant differences in racial distribution, sex, BMI, and average LOS as shown in Table 1. Those who received pharmacologic prophylaxis were found to be significantly older than those who did not (62.7 years versus 53.2 years, P < 0.001), more likely to be male (50.6% vs, 42.4%, P < 0.001), more likely to have hospital-acquired VTE (2.2% vs. 0.5%, P < 0.001), and to have a shorter LOS (7.1 days vs. 9.8, P < 0.001).

Within the cohort group receiving pharmacologic prophylaxis (n = 6574), hospital-acquired VTE occurred in patients who were significantly younger (58.2 years vs. 62.8 years, P = 0.003) with a greater LOS (23.8 days vs. 6.7, P < 0.001) than those without. Within the group receiving no pharmacologic prophylaxis (n = 24,152), hospital-acquired VTE occurred in patients who were significantly older (57.1 years vs. 53.2 years, P = 0.014) with more than twice the LOS (20.2 days vs. 9.7 days, P < 0.001) compared to those without. Sixty-six of 75 (88%) randomly selected patients in which new VTE was identified by the automated electronic query had this diagnosis confirmed during manual chart review.

As shown in Table 2, automated calculation on a subsample of 300 randomly selected patients using APPS had a mean of 5.5 (SD, 2.9) while manual calculation of the original PPS on the same patients had a mean of 5.1 (SD, 2.6). There was no significant difference in mean between manual calculation and APPS (P = 0.073). There were, however, significant differences in how often individual criteria were considered present. The largest contributors to the difference in scores between APPS and manual calculation were “prior VTE” (positive, 16% vs. 8.3%, respectively) and “reduced mobility” (positive, 74.3% vs. 66%, respectively) as shown in Table 2. In the subsample, there were a total of 6 (2.0%) hospital-acquired VTE events. APPS’ automated calculation had an AUC = 0.79 (CI, 0.63-0.95) that was significant (P = 0.016) with a cutoff value of 5. Chart review’s manual calculation of the PPS had an AUC = 0.76 (CI 0.61-0.91) that was also significant (P = 0.029).

Distribution of Patient Characteristics in Cohort

Our entire cohort of 30,726 unique patients admitted during the study period included 260 (0.8%) who experienced hospital-acquired VTEs (Table 3). In patients receiving no pharmacologic prophylaxis, the average APPS was 4.0 (SD, 2.4) for those without VTE and 7.1 (SD, 2.3) for those with VTE. In patients who had received pharmacologic prophylaxis, those without hospital-acquired VTE had an average APPS of 4.9 (SD, 2.6) and those with hospital-acquired VTE averaged 7.7 (SD, 2.6). APPS’ ROC curves for “no pharmacologic prophylaxis” had an AUC = 0.81 (CI, 0.79 – 0.83) that was significant (P < 0.001) with a cutoff value of 5. There was similar performance in the pharmacologic prophylaxis group with an AUC = 0.79 (CI, 0.76 – 0.82) and cutoff value of 5, as shown in the Figure. Over the entire cohort, APPS had a sensitivity of 85.4%, specificity of 53.3%, positive predictive value (PPV) of 1.5%, and a negative predictive value (NPV) of 99.8% when using a cutoff of 5. The average APPS calculation time was 0.03 seconds per encounter. Additional information on individual criteria can be found in Table 3.

 

DISCUSSION

Automated calculation of APPS using EHR data from prior encounters and the first 4 hours of admission was predictive of in-hospital VTE. APPS performed as well as traditional manual score calculation of the PPS. It was able to do so with no physician input, significantly lessening the burden of calculation and potentially increasing frequency of data-driven VTE risk assessment.

While automated calculation of certain scores is becoming more common, risk calculators that require data beyond vital signs and lab results have lagged,^16-19 in part because of uncertainty about 2 issues. The first is whether EHR data accurately represent the current clinical picture. The second is if a machine-interpretable algorithm to determine a clinical status (eg, “active cancer”) would be similar to a doctor’s perception of that same concept. We attempted to better understand these 2 challenges through developing APPS. Concerning accuracy, EHR data correctly represent the clinical scenario: designations of VTEP and hospital-acquired VTE were accurate in approximately 90% of reviewed cases. Regarding the second concern, when comparing APPS to manual calculation, we found significant differences (P < 0.001) in how often 8 of the 11 criteria were positive, yet no significant difference in overall score and similar predictive capacity. Manual calculation appeared more likely to find data in the index encounter or in structured data. For example, “active cancer” may be documented only in a physician’s note, easily accounted for during a physician’s calculation but missed by APPS looking only for structured data. In contrast, automated calculation found historic criteria, such as “prior VTE” or “known thrombophilic condition,” positive more often. If the patient is being admitted for a problem unrelated to blood clots, the physician may have little time or interest to look through hundreds of EHR documents to discover a 2-year-old VTE. As patients’ records become larger and denser, more historic data can become buried and forgotten. While the 2 scores differ on individual criteria, they are similarly predictive and able to bifurcate the at-risk population to those who should and should not receive pharmacologic prophylaxis.

The APPS was found to have near-equal performance in the pharmacologic vs. no pharmacologic prophylaxis cohorts. This finding agrees with a study that found no significant difference in predicting 90-day VTE when looking at 86 risk factors vs. the most significant 4, none of which related to prescribed prophylaxis.¹⁸ The original PPS had a reported sensitivity of 94.6%, specificity 62%, PPV 7.5%, and NPV 99.7% in its derivation cohort.¹³ We matched APPS to the ratio of sensitivity to specificity, using 5 as the cutoff value. APPS performed slightly worse with sensitivity of 85.4%, specificity 53.3%, PPV 1.5%, and NPV 99.8%. This difference may have resulted from the original PPS study’s use of 90-day follow-up to determine VTE occurrence, whereas we looked only until the end of current hospitalization, an average of 9.2 days. Furthermore, the PPS had significantly poorer performance (AUC = 0.62) than that seen in the original derivation cohort in a separate study that manually calculated the score on more than 1000 patients.¹⁵

There are important limitations to our study. It was done at a single academic institution using a dataset of VTE-associated, validated research that was well-known to the researchers.²⁰ Another major limitation is the dependence of the algorithm on data available within the first 4 hours of admission and earlier; thus, previous encounters may frequently play an important role. Patients presenting to our health system for the first time would have significantly fewer data available at the time of calculation. Additionally, our data could not reliably tell us the total doses of pharmacologic prophylaxis that a patient received. While most patients will maintain a consistent VTEP regimen once initiated in the hospital, 2 patients with the same LOS may have received differing amounts of pharmacologic prophylaxis. This research study did not assess how much time automatic calculation of VTE risk might save providers, because we did not record the time for each manual abstraction; however, from discussion with the main abstracter, chart review and manual calculation for this study took from 2 to 14 minutes per patient, depending on the number of previous interactions with the health system. Finally, although we chose data elements that are likely to exist at most institutions using an EHR, many institutions’ EHRs do not have EDW capabilities nor programmers who can assist with an automated risk score.

The EHR interventions to assist providers in determining appropriate VTEP have been able to increase rates of VTEP and decrease VTE-associated mortality.^16,21 In addition to automating the calculation of guideline-adherent risk scores, there is a need for wider adoption for clinical decision support for VTE. For this reason, we chose only structured data fields from some of the most common elements within our EHR’s data warehouse to derive APPS (Appendix 1). Our study supports the idea that automated calculation of scores requiring input of more complex data such as diagnoses, recent medical events, and current clinical status remains predictive of hospital-acquired VTE risk. Because it is calculated automatically in the background while the clinician completes his or her assessment, the APPS holds the potential to significantly reduce the burden on providers while making guideline-adherent risk assessment more readily accessible. Further research is required to determine the exact amount of time automatic calculation saves, and, more important, if the relatively high predictive capacity we observed using APPS would be reproducible across institutions and could reduce incidence of hospital-acquired VTE.

 

Disclosures

Dr. Auerbach was supported by NHLBI K24HL098372 during the period of this study. Dr. Khanna, who is an implementation scientist at the University of California San Francisco Center for Digital Health Innovation, is the principal inventor of CareWeb, and may benefit financially from its commercialization. The other authors report no financial conflicts of interest.