Jason Hom, MD

Point-of-care ultrasonography (POCUS) can transform healthcare delivery through its diagnostic and therapeutic expediency.¹ POCUS has been shown to bolster diagnostic accuracy, reduce procedural complications, decrease inpatient length of stay, and improve patient satisfaction by encouraging the physician to be present at the bedside.^2-8

POCUS has become widespread across a variety of clinical settings as more investigations have demonstrated its positive impact on patient care.^1,9-12 This includes the use of POCUS by trainees, who are now utilizing this technology as part of their assessments of patients.^13,14 However, trainees may be performing these examinations with minimal oversight, and outside of emergency medicine, there are few guidelines on how to effectively teach POCUS or measure competency.^13,14 While POCUS is rapidly becoming a part of inpatient care, teaching physicians may have little experience in ultrasound or the expertise to adequately supervise trainees.¹⁴ There is a growing need to study what trainees can learn and how this knowledge is acquired.

Previous investigations have demonstrated that inexperienced users can be taught to use POCUS to identify a variety of pathological states.^2,3,15-23 Most of these curricula used a single lecture series as their pedagogical vehicle, and they variably included junior medical trainees. More importantly, the investigations did not explore whether personal access to handheld ultrasound devices (HUDs) improved learning. In theory, improved access to POCUS devices increases opportunities for authentic and deliberate practice, which may be needed to improve trainee skill with POCUS beyond the classroom setting.¹⁴

This study aimed to address several ongoing gaps in knowledge related to learning POCUS. First, we hypothesized that personal HUD access would improve trainees’ POCUS-­related knowledge and interpretive ability as a result of increased practice opportunities. Second, we hypothesized that trainees who receive personal access to HUDs would be more likely to perform POCUS examinations and feel more confident in their interpretations. Finally, we hypothesized that repeated exposure to POCUS-related lectures would result in greater improvements in knowledge as compared with a single lecture series.

Participants and Setting

The 2017 intern class (n = 47) at an academic internal medicine residency program participated in the study. Control data were obtained from the 2016 intern class (historical control; n = 50) and the 2018 intern class (contemporaneous control; n = 52). The Stanford University Institutional Review Board approved this study.

Study Design

The 2017 intern class (n = 47) received POCUS didactics from June 2017 to June 2018. To evaluate if increased access to HUDs improved learning outcomes, the 2017 interns were randomized 1:1 to receive their own personal HUD that could be used for patient care and/or self-directed learning (n = 24) vs no-HUD (n = 23; Figure). Learning outcomes were assessed over the course of 1 year (see “Outcomes” below) and were compared with the 2016 and 2018 controls. The 2016 intern class had completed a year of training but had not received formalized POCUS didactics (historical control), whereas the 2018 intern class was assessed at the beginning of their year (contemporaneous control; Figure). In order to make comparisons based on intern experience, baseline data for the 2017 intern class were compared with the 2018 intern class, whereas end-of-study data for 2017 interns were compared with 2016 interns.

Outcomes

The primary outcome was the difference in assessment scores at the end of the study period between interns randomized to receive a HUD and those who were not. Secondary outcomes included differences in HUD usage rates, lecture attendance, and assessment scores. To assess whether repeated lecture exposure resulted in greater amounts of learning, this study evaluated for assessment score improvements after each lecture block. Finally, trainee attitudes toward POCUS and their confidence in their interpretative ability were measured at the beginning and end of the study period.

Curriculum Implementation

The lectures were administered as once-weekly didactics of 1-hour duration to interns rotating on the inpatient wards rotation. This rotation is 4 weeks long, and each intern will experience the rotation two to four times per year. Each lecture contained two parts: (1) 20-30 minutes of didactics via Microsoft PowerPoint^TM and (2) 30-40 minutes of supervised practice using HUDs on standardized patients. Four lectures were given each month: (1) introduction to POCUS and ultrasound physics, (2) thoracic/lung ultrasound, (3) echocardiography, and (4) abdominal POCUS. The lectures consisted of contrasting cases of normal/abnormal videos and clinical vignettes. These four lectures were repeated each month as new interns rotated on service. Some interns experienced the same content multiple times, which was intentional in order to assess their rates of learning over time. Lecture contents were based on previously published guidelines and expert consensus for teaching POCUS in internal medicine.^{13, 24-26} Content from the Accreditation Council for Graduate Medical Education (ACGME) and the American College of Emergency Physicians (ACEP) was also incorporated because these organizations had published relevant guidelines for teaching POCUS.^13,26 Further development of the lectures occurred through review of previously described POCUS-relevant curricula.^27-32

Handheld Ultrasound Devices

This study used the Philips Lumify^TM, a United States Food and Drug Administration–approved device. Interns randomized to HUDs received their own device at the start of the rotation. It was at their discretion to use the device outside of the course. All devices were approved for patient use and were encrypted in compliance with our information security office. For privacy reasons, any saved patient images were not reviewed by the researchers. Interns were encouraged to share their findings with supervising physicians during rounds, but actual oversight was not measured. Interns not randomized to HUDs could access a single community device that was shared among all residents and fellows in the hospital. Interns reported the average number of POCUS examinations performed each week via a survey sent during the last week of the rotation.

Assessment Design and Implementation

Assessments evaluating trainee knowledge were administered before, during, and after the study period (Figure). For the 2017 cohort, assessments were also administered at the start and end of the ward month to track knowledge acquisition. Assessment contents were selected from POCUS guidelines for internal medicine and adaptation of the ACGME and ACEP guidelines.^13,24,26 Additional content was obtained from major society POCUS tutorials and deidentified images collected by the study authors.^13,24,33 In keeping with previously described methodology, the images were shown for approximately 12 seconds, followed by five additional seconds to allow the learner to answer the question.³² Final assessment contents were determined by the authors using the Delphi method.³⁴ A sample assessment can be found in the Appendix Material.

Surveys

Surveys were administered alongside the assessments to the 2016-2018 intern classes. These surveys assessed trainee attitudes toward POCUS and were based on previously validated assessments.^27,28,30 Attitudes were measured using 5-point Likert scales.

Statistical Analysis

For the primary outcome, we performed generalized binomial mixed-effect regressions using the survey periods, randomization group, and the interaction of the two as independent variables after adjusting for attendance and controlling of intra-intern correlations. The bivariate unadjusted analysis was performed to display the distribution of overall correctness on the assessments. Wilcoxon signed rank test was used to determine score significance for dependent score variables (R-­Statistical Programming Language, Vienna, Austria).

Baseline Characteristics

There were 149 interns who participated in this study (Figure). Assessment/survey completion rates were as follows: 2016 control: 68.0%; 2017 preintervention: 97.9%; 2017 postintervention: 89.4%; and 2018 control: 100%. The 2017 interns reported similar amounts of prior POCUS exposure in medical school (Table 1).

Primary Outcome: Assessment Scores (HUD vs no HUD)

There were no significant differences in assessment scores at the end of the study between interns randomized to personal HUD access vs those to no-HUD access (Table 1). HUD interns reported performing POCUS assessments on patients a mean 6.8 (standard deviation [SD] 2.2) times per week vs 6.4 (SD 2.9) times per week in the no-HUD arm (P = .66). The mean lecture attendance was 75.0% and did not significantly differ between the HUD arms (Table 1).

Secondary Outcomes

Impact of Repeating Lectures

The 2017 interns demonstrated significant increases in preblock vs postblock assessment scores after first-time exposure to the lectures (median preblock score 0.61 [interquartile range (IQR), 0.53-0.70] vs postblock score 0.81 [IQR, 0.72-0.86]; P < .001; Table 2). However, intern performance on the preblock vs postblock assessments after second-time exposure to the curriculum failed to improve (median second preblock score 0.78 [IQR, 0.69-0.83] vs postblock score 0.81 [IQR, 0.64-0.89]; P = .94). Intern performance on individual domains of knowledge for each block is listed in Appendix Table 1.

Intervention Performance vs Controls

The 2016 historical control had significantly higher scores compared with the 2017 preintervention group (P < .001; Appendix Table 2). The year-long lecture series resulted in significant increases in median scores for the 2017 group (median preintervention score 0.55 [0.41-0.61] vs median postintervention score 0.84 [0.71-0.90]; P = .006; Appendix Table 1). At the end of the study, the 2017 postintervention scores were significantly higher across multiple knowledge domains compared with the 2016 historical control (Appendix Table 2).

Survey Results

Notably, the 2017 intern class at the end of the intervention did not have significantly different assessment scores for several disease-specific domains, compared with the 2016 control (Appendix Table 2). Nonetheless, the 2017 intern class reported higher levels of confidence in these same domains despite similar scores (Supplementary Figure). The HUD group seldomly cited a lack of confidence in their abilities as a barrier to performing POCUS examinations (17.6%), compared with the no-HUD group (50.0%), despite nearly identical assessment scores between the two groups (Table 1).

Previous guidelines have recommended increased HUD access for learners,^13,24,35,36 but there have been few investigations that have evaluated the impact of such access on learning POCUS. One previous investigation found that hospitalists who carried HUDs were more likely to identify heart failure on bedside examination.³⁷ In contrast, our study found no improvement in interpretative ability when randomizing interns to carry HUDs for patient care. Notably, interns did not perform more POCUS examinations when given HUDs. We offer several explanations for this finding. First, time-motion studies have demonstrated that internal medicine interns spend less than 15% of their time toward direct patient care.³⁸ It is possible that the demands of being an intern impeded their ability to perform more POCUS examinations on their patients, regardless of HUD access. Alternatively, the interns randomized to no personal access may have used the community device more frequently as a result of the lecture series. Given the cost of HUDs, further studies are needed to assess the degree to which HUD access will improve trainee interpretive ability, especially as more training programs consider the creation of ultrasound curricula.^{10,11,24,39,40}

This study was unique because it followed interns over a year-long course that repeated the same material to assess rates of learning with repeated exposure. Learners improved their scores after the first, but not second, block. Furthermore, the median scores were nearly identical between the first postblock assessment and second preblock assessment (0.81 vs 0.78), suggesting that knowledge was retained between blocks. Together, these findings suggest there may be limitations of traditional lectures that use standardized patient models for practice. Supplementary pedagogies, such as in-the-moment feedback with actual patients, may be needed to promote mastery.^14,35

Despite no formal curriculum, the 2016 intern class (historical control) had learned POCUS to some degree based on their higher assessment scores compared with the 2017 intern class during the preintervention period. Such learning may be informal, and yet, trainees may feel confident in making clinical decisions without formalized training, accreditation, or oversight. As suggested by this study, adding regular didactics or giving trainees HUDs may not immediately solve this issue. For assessment items in which the 2017 interns did not significantly differ from the controls, they nonetheless reported higher confidence in their abilities. Similarly, interns randomized to HUDs less frequently cited a lack of confidence in their abilities, despite similar scores to the no-HUD group. Such confidence may be incongruent with their actual knowledge or ability to safely use POCUS. This phenomenon of misplaced confidence is known as the Dunning–Kruger effect, and it may be common with ultrasound learning.⁴¹ While confidence can be part of a holistic definition of competency,¹⁴ these results raise the concern that trainees may have difficulty assessing their own competency level with POCUS.³⁵

There are several limitations to this study. It was performed at a single institution with limited sample size. It examined only intern physicians because of funding constraints, which limits the generalizability of these findings among medical trainees. Technical ability assessments (including obtaining and interpreting images) were not included. We were unable to track the timing or location of the devices’ usage, and the interns’ self-reported usage rates may be subject to recall bias. To our knowledge, there were no significant lapses in device availability/functionality. Intern physicians in the HUD arm did not receive formal feedback on personally acquired patient images, which may have limited the intervention’s impact.

In conclusion, internal medicine interns who received personal HUDs were not better at recognizing normal/abnormal findings on image assessments, and they did not report performing more POCUS examinations. Since the minority of a trainee’s time is spent toward direct patient care, offering trainees HUDs without substantial guidance may not be enough to promote mastery. Notably, trainees who received HUDs felt more confident in their abilities, despite no objective increase in their actual skill. Finally, interns who received POCUS-related lectures experienced significant benefit upon first exposure to the material, while repeated exposures did not improve performance. Future investigations should stringently track trainee POCUS usage rates with HUDs and assess whether image acquisition ability improves as a result of personal access.

Point-of-care ultrasonography (POCUS) can transform healthcare delivery through its diagnostic and therapeutic expediency.¹ POCUS has been shown to bolster diagnostic accuracy, reduce procedural complications, decrease inpatient length of stay, and improve patient satisfaction by encouraging the physician to be present at the bedside.^2-8

POCUS has become widespread across a variety of clinical settings as more investigations have demonstrated its positive impact on patient care.^1,9-12 This includes the use of POCUS by trainees, who are now utilizing this technology as part of their assessments of patients.^13,14 However, trainees may be performing these examinations with minimal oversight, and outside of emergency medicine, there are few guidelines on how to effectively teach POCUS or measure competency.^13,14 While POCUS is rapidly becoming a part of inpatient care, teaching physicians may have little experience in ultrasound or the expertise to adequately supervise trainees.¹⁴ There is a growing need to study what trainees can learn and how this knowledge is acquired.

Previous investigations have demonstrated that inexperienced users can be taught to use POCUS to identify a variety of pathological states.^2,3,15-23 Most of these curricula used a single lecture series as their pedagogical vehicle, and they variably included junior medical trainees. More importantly, the investigations did not explore whether personal access to handheld ultrasound devices (HUDs) improved learning. In theory, improved access to POCUS devices increases opportunities for authentic and deliberate practice, which may be needed to improve trainee skill with POCUS beyond the classroom setting.¹⁴

This study aimed to address several ongoing gaps in knowledge related to learning POCUS. First, we hypothesized that personal HUD access would improve trainees’ POCUS-­related knowledge and interpretive ability as a result of increased practice opportunities. Second, we hypothesized that trainees who receive personal access to HUDs would be more likely to perform POCUS examinations and feel more confident in their interpretations. Finally, we hypothesized that repeated exposure to POCUS-related lectures would result in greater improvements in knowledge as compared with a single lecture series.

Participants and Setting

The 2017 intern class (n = 47) at an academic internal medicine residency program participated in the study. Control data were obtained from the 2016 intern class (historical control; n = 50) and the 2018 intern class (contemporaneous control; n = 52). The Stanford University Institutional Review Board approved this study.

Study Design

The 2017 intern class (n = 47) received POCUS didactics from June 2017 to June 2018. To evaluate if increased access to HUDs improved learning outcomes, the 2017 interns were randomized 1:1 to receive their own personal HUD that could be used for patient care and/or self-directed learning (n = 24) vs no-HUD (n = 23; Figure). Learning outcomes were assessed over the course of 1 year (see “Outcomes” below) and were compared with the 2016 and 2018 controls. The 2016 intern class had completed a year of training but had not received formalized POCUS didactics (historical control), whereas the 2018 intern class was assessed at the beginning of their year (contemporaneous control; Figure). In order to make comparisons based on intern experience, baseline data for the 2017 intern class were compared with the 2018 intern class, whereas end-of-study data for 2017 interns were compared with 2016 interns.

Outcomes

The primary outcome was the difference in assessment scores at the end of the study period between interns randomized to receive a HUD and those who were not. Secondary outcomes included differences in HUD usage rates, lecture attendance, and assessment scores. To assess whether repeated lecture exposure resulted in greater amounts of learning, this study evaluated for assessment score improvements after each lecture block. Finally, trainee attitudes toward POCUS and their confidence in their interpretative ability were measured at the beginning and end of the study period.

Curriculum Implementation

The lectures were administered as once-weekly didactics of 1-hour duration to interns rotating on the inpatient wards rotation. This rotation is 4 weeks long, and each intern will experience the rotation two to four times per year. Each lecture contained two parts: (1) 20-30 minutes of didactics via Microsoft PowerPoint^TM and (2) 30-40 minutes of supervised practice using HUDs on standardized patients. Four lectures were given each month: (1) introduction to POCUS and ultrasound physics, (2) thoracic/lung ultrasound, (3) echocardiography, and (4) abdominal POCUS. The lectures consisted of contrasting cases of normal/abnormal videos and clinical vignettes. These four lectures were repeated each month as new interns rotated on service. Some interns experienced the same content multiple times, which was intentional in order to assess their rates of learning over time. Lecture contents were based on previously published guidelines and expert consensus for teaching POCUS in internal medicine.^{13, 24-26} Content from the Accreditation Council for Graduate Medical Education (ACGME) and the American College of Emergency Physicians (ACEP) was also incorporated because these organizations had published relevant guidelines for teaching POCUS.^13,26 Further development of the lectures occurred through review of previously described POCUS-relevant curricula.^27-32

Handheld Ultrasound Devices

This study used the Philips Lumify^TM, a United States Food and Drug Administration–approved device. Interns randomized to HUDs received their own device at the start of the rotation. It was at their discretion to use the device outside of the course. All devices were approved for patient use and were encrypted in compliance with our information security office. For privacy reasons, any saved patient images were not reviewed by the researchers. Interns were encouraged to share their findings with supervising physicians during rounds, but actual oversight was not measured. Interns not randomized to HUDs could access a single community device that was shared among all residents and fellows in the hospital. Interns reported the average number of POCUS examinations performed each week via a survey sent during the last week of the rotation.

Assessment Design and Implementation

Assessments evaluating trainee knowledge were administered before, during, and after the study period (Figure). For the 2017 cohort, assessments were also administered at the start and end of the ward month to track knowledge acquisition. Assessment contents were selected from POCUS guidelines for internal medicine and adaptation of the ACGME and ACEP guidelines.^13,24,26 Additional content was obtained from major society POCUS tutorials and deidentified images collected by the study authors.^13,24,33 In keeping with previously described methodology, the images were shown for approximately 12 seconds, followed by five additional seconds to allow the learner to answer the question.³² Final assessment contents were determined by the authors using the Delphi method.³⁴ A sample assessment can be found in the Appendix Material.

Surveys

Surveys were administered alongside the assessments to the 2016-2018 intern classes. These surveys assessed trainee attitudes toward POCUS and were based on previously validated assessments.^27,28,30 Attitudes were measured using 5-point Likert scales.

Statistical Analysis

For the primary outcome, we performed generalized binomial mixed-effect regressions using the survey periods, randomization group, and the interaction of the two as independent variables after adjusting for attendance and controlling of intra-intern correlations. The bivariate unadjusted analysis was performed to display the distribution of overall correctness on the assessments. Wilcoxon signed rank test was used to determine score significance for dependent score variables (R-­Statistical Programming Language, Vienna, Austria).

Baseline Characteristics

There were 149 interns who participated in this study (Figure). Assessment/survey completion rates were as follows: 2016 control: 68.0%; 2017 preintervention: 97.9%; 2017 postintervention: 89.4%; and 2018 control: 100%. The 2017 interns reported similar amounts of prior POCUS exposure in medical school (Table 1).

Primary Outcome: Assessment Scores (HUD vs no HUD)

There were no significant differences in assessment scores at the end of the study between interns randomized to personal HUD access vs those to no-HUD access (Table 1). HUD interns reported performing POCUS assessments on patients a mean 6.8 (standard deviation [SD] 2.2) times per week vs 6.4 (SD 2.9) times per week in the no-HUD arm (P = .66). The mean lecture attendance was 75.0% and did not significantly differ between the HUD arms (Table 1).

Secondary Outcomes

Impact of Repeating Lectures

The 2017 interns demonstrated significant increases in preblock vs postblock assessment scores after first-time exposure to the lectures (median preblock score 0.61 [interquartile range (IQR), 0.53-0.70] vs postblock score 0.81 [IQR, 0.72-0.86]; P < .001; Table 2). However, intern performance on the preblock vs postblock assessments after second-time exposure to the curriculum failed to improve (median second preblock score 0.78 [IQR, 0.69-0.83] vs postblock score 0.81 [IQR, 0.64-0.89]; P = .94). Intern performance on individual domains of knowledge for each block is listed in Appendix Table 1.

Intervention Performance vs Controls

The 2016 historical control had significantly higher scores compared with the 2017 preintervention group (P < .001; Appendix Table 2). The year-long lecture series resulted in significant increases in median scores for the 2017 group (median preintervention score 0.55 [0.41-0.61] vs median postintervention score 0.84 [0.71-0.90]; P = .006; Appendix Table 1). At the end of the study, the 2017 postintervention scores were significantly higher across multiple knowledge domains compared with the 2016 historical control (Appendix Table 2).

Survey Results

Notably, the 2017 intern class at the end of the intervention did not have significantly different assessment scores for several disease-specific domains, compared with the 2016 control (Appendix Table 2). Nonetheless, the 2017 intern class reported higher levels of confidence in these same domains despite similar scores (Supplementary Figure). The HUD group seldomly cited a lack of confidence in their abilities as a barrier to performing POCUS examinations (17.6%), compared with the no-HUD group (50.0%), despite nearly identical assessment scores between the two groups (Table 1).

Previous guidelines have recommended increased HUD access for learners,^13,24,35,36 but there have been few investigations that have evaluated the impact of such access on learning POCUS. One previous investigation found that hospitalists who carried HUDs were more likely to identify heart failure on bedside examination.³⁷ In contrast, our study found no improvement in interpretative ability when randomizing interns to carry HUDs for patient care. Notably, interns did not perform more POCUS examinations when given HUDs. We offer several explanations for this finding. First, time-motion studies have demonstrated that internal medicine interns spend less than 15% of their time toward direct patient care.³⁸ It is possible that the demands of being an intern impeded their ability to perform more POCUS examinations on their patients, regardless of HUD access. Alternatively, the interns randomized to no personal access may have used the community device more frequently as a result of the lecture series. Given the cost of HUDs, further studies are needed to assess the degree to which HUD access will improve trainee interpretive ability, especially as more training programs consider the creation of ultrasound curricula.^{10,11,24,39,40}

This study was unique because it followed interns over a year-long course that repeated the same material to assess rates of learning with repeated exposure. Learners improved their scores after the first, but not second, block. Furthermore, the median scores were nearly identical between the first postblock assessment and second preblock assessment (0.81 vs 0.78), suggesting that knowledge was retained between blocks. Together, these findings suggest there may be limitations of traditional lectures that use standardized patient models for practice. Supplementary pedagogies, such as in-the-moment feedback with actual patients, may be needed to promote mastery.^14,35

Despite no formal curriculum, the 2016 intern class (historical control) had learned POCUS to some degree based on their higher assessment scores compared with the 2017 intern class during the preintervention period. Such learning may be informal, and yet, trainees may feel confident in making clinical decisions without formalized training, accreditation, or oversight. As suggested by this study, adding regular didactics or giving trainees HUDs may not immediately solve this issue. For assessment items in which the 2017 interns did not significantly differ from the controls, they nonetheless reported higher confidence in their abilities. Similarly, interns randomized to HUDs less frequently cited a lack of confidence in their abilities, despite similar scores to the no-HUD group. Such confidence may be incongruent with their actual knowledge or ability to safely use POCUS. This phenomenon of misplaced confidence is known as the Dunning–Kruger effect, and it may be common with ultrasound learning.⁴¹ While confidence can be part of a holistic definition of competency,¹⁴ these results raise the concern that trainees may have difficulty assessing their own competency level with POCUS.³⁵

There are several limitations to this study. It was performed at a single institution with limited sample size. It examined only intern physicians because of funding constraints, which limits the generalizability of these findings among medical trainees. Technical ability assessments (including obtaining and interpreting images) were not included. We were unable to track the timing or location of the devices’ usage, and the interns’ self-reported usage rates may be subject to recall bias. To our knowledge, there were no significant lapses in device availability/functionality. Intern physicians in the HUD arm did not receive formal feedback on personally acquired patient images, which may have limited the intervention’s impact.

In conclusion, internal medicine interns who received personal HUDs were not better at recognizing normal/abnormal findings on image assessments, and they did not report performing more POCUS examinations. Since the minority of a trainee’s time is spent toward direct patient care, offering trainees HUDs without substantial guidance may not be enough to promote mastery. Notably, trainees who received HUDs felt more confident in their abilities, despite no objective increase in their actual skill. Finally, interns who received POCUS-related lectures experienced significant benefit upon first exposure to the material, while repeated exposures did not improve performance. Future investigations should stringently track trainee POCUS usage rates with HUDs and assess whether image acquisition ability improves as a result of personal access.

Point-of-care ultrasonography (POCUS) can transform healthcare delivery through its diagnostic and therapeutic expediency.¹ POCUS has been shown to bolster diagnostic accuracy, reduce procedural complications, decrease inpatient length of stay, and improve patient satisfaction by encouraging the physician to be present at the bedside.^2-8

POCUS has become widespread across a variety of clinical settings as more investigations have demonstrated its positive impact on patient care.^1,9-12 This includes the use of POCUS by trainees, who are now utilizing this technology as part of their assessments of patients.^13,14 However, trainees may be performing these examinations with minimal oversight, and outside of emergency medicine, there are few guidelines on how to effectively teach POCUS or measure competency.^13,14 While POCUS is rapidly becoming a part of inpatient care, teaching physicians may have little experience in ultrasound or the expertise to adequately supervise trainees.¹⁴ There is a growing need to study what trainees can learn and how this knowledge is acquired.

Previous investigations have demonstrated that inexperienced users can be taught to use POCUS to identify a variety of pathological states.^2,3,15-23 Most of these curricula used a single lecture series as their pedagogical vehicle, and they variably included junior medical trainees. More importantly, the investigations did not explore whether personal access to handheld ultrasound devices (HUDs) improved learning. In theory, improved access to POCUS devices increases opportunities for authentic and deliberate practice, which may be needed to improve trainee skill with POCUS beyond the classroom setting.¹⁴

This study aimed to address several ongoing gaps in knowledge related to learning POCUS. First, we hypothesized that personal HUD access would improve trainees’ POCUS-­related knowledge and interpretive ability as a result of increased practice opportunities. Second, we hypothesized that trainees who receive personal access to HUDs would be more likely to perform POCUS examinations and feel more confident in their interpretations. Finally, we hypothesized that repeated exposure to POCUS-related lectures would result in greater improvements in knowledge as compared with a single lecture series.

Participants and Setting

The 2017 intern class (n = 47) at an academic internal medicine residency program participated in the study. Control data were obtained from the 2016 intern class (historical control; n = 50) and the 2018 intern class (contemporaneous control; n = 52). The Stanford University Institutional Review Board approved this study.

Study Design

The 2017 intern class (n = 47) received POCUS didactics from June 2017 to June 2018. To evaluate if increased access to HUDs improved learning outcomes, the 2017 interns were randomized 1:1 to receive their own personal HUD that could be used for patient care and/or self-directed learning (n = 24) vs no-HUD (n = 23; Figure). Learning outcomes were assessed over the course of 1 year (see “Outcomes” below) and were compared with the 2016 and 2018 controls. The 2016 intern class had completed a year of training but had not received formalized POCUS didactics (historical control), whereas the 2018 intern class was assessed at the beginning of their year (contemporaneous control; Figure). In order to make comparisons based on intern experience, baseline data for the 2017 intern class were compared with the 2018 intern class, whereas end-of-study data for 2017 interns were compared with 2016 interns.

Outcomes

The primary outcome was the difference in assessment scores at the end of the study period between interns randomized to receive a HUD and those who were not. Secondary outcomes included differences in HUD usage rates, lecture attendance, and assessment scores. To assess whether repeated lecture exposure resulted in greater amounts of learning, this study evaluated for assessment score improvements after each lecture block. Finally, trainee attitudes toward POCUS and their confidence in their interpretative ability were measured at the beginning and end of the study period.

Curriculum Implementation

The lectures were administered as once-weekly didactics of 1-hour duration to interns rotating on the inpatient wards rotation. This rotation is 4 weeks long, and each intern will experience the rotation two to four times per year. Each lecture contained two parts: (1) 20-30 minutes of didactics via Microsoft PowerPoint^TM and (2) 30-40 minutes of supervised practice using HUDs on standardized patients. Four lectures were given each month: (1) introduction to POCUS and ultrasound physics, (2) thoracic/lung ultrasound, (3) echocardiography, and (4) abdominal POCUS. The lectures consisted of contrasting cases of normal/abnormal videos and clinical vignettes. These four lectures were repeated each month as new interns rotated on service. Some interns experienced the same content multiple times, which was intentional in order to assess their rates of learning over time. Lecture contents were based on previously published guidelines and expert consensus for teaching POCUS in internal medicine.^{13, 24-26} Content from the Accreditation Council for Graduate Medical Education (ACGME) and the American College of Emergency Physicians (ACEP) was also incorporated because these organizations had published relevant guidelines for teaching POCUS.^13,26 Further development of the lectures occurred through review of previously described POCUS-relevant curricula.^27-32

Handheld Ultrasound Devices

This study used the Philips Lumify^TM, a United States Food and Drug Administration–approved device. Interns randomized to HUDs received their own device at the start of the rotation. It was at their discretion to use the device outside of the course. All devices were approved for patient use and were encrypted in compliance with our information security office. For privacy reasons, any saved patient images were not reviewed by the researchers. Interns were encouraged to share their findings with supervising physicians during rounds, but actual oversight was not measured. Interns not randomized to HUDs could access a single community device that was shared among all residents and fellows in the hospital. Interns reported the average number of POCUS examinations performed each week via a survey sent during the last week of the rotation.

Assessment Design and Implementation

Assessments evaluating trainee knowledge were administered before, during, and after the study period (Figure). For the 2017 cohort, assessments were also administered at the start and end of the ward month to track knowledge acquisition. Assessment contents were selected from POCUS guidelines for internal medicine and adaptation of the ACGME and ACEP guidelines.^13,24,26 Additional content was obtained from major society POCUS tutorials and deidentified images collected by the study authors.^13,24,33 In keeping with previously described methodology, the images were shown for approximately 12 seconds, followed by five additional seconds to allow the learner to answer the question.³² Final assessment contents were determined by the authors using the Delphi method.³⁴ A sample assessment can be found in the Appendix Material.

Surveys

Surveys were administered alongside the assessments to the 2016-2018 intern classes. These surveys assessed trainee attitudes toward POCUS and were based on previously validated assessments.^27,28,30 Attitudes were measured using 5-point Likert scales.

Statistical Analysis

For the primary outcome, we performed generalized binomial mixed-effect regressions using the survey periods, randomization group, and the interaction of the two as independent variables after adjusting for attendance and controlling of intra-intern correlations. The bivariate unadjusted analysis was performed to display the distribution of overall correctness on the assessments. Wilcoxon signed rank test was used to determine score significance for dependent score variables (R-­Statistical Programming Language, Vienna, Austria).

Baseline Characteristics

There were 149 interns who participated in this study (Figure). Assessment/survey completion rates were as follows: 2016 control: 68.0%; 2017 preintervention: 97.9%; 2017 postintervention: 89.4%; and 2018 control: 100%. The 2017 interns reported similar amounts of prior POCUS exposure in medical school (Table 1).

Primary Outcome: Assessment Scores (HUD vs no HUD)

There were no significant differences in assessment scores at the end of the study between interns randomized to personal HUD access vs those to no-HUD access (Table 1). HUD interns reported performing POCUS assessments on patients a mean 6.8 (standard deviation [SD] 2.2) times per week vs 6.4 (SD 2.9) times per week in the no-HUD arm (P = .66). The mean lecture attendance was 75.0% and did not significantly differ between the HUD arms (Table 1).

Secondary Outcomes

Impact of Repeating Lectures

The 2017 interns demonstrated significant increases in preblock vs postblock assessment scores after first-time exposure to the lectures (median preblock score 0.61 [interquartile range (IQR), 0.53-0.70] vs postblock score 0.81 [IQR, 0.72-0.86]; P < .001; Table 2). However, intern performance on the preblock vs postblock assessments after second-time exposure to the curriculum failed to improve (median second preblock score 0.78 [IQR, 0.69-0.83] vs postblock score 0.81 [IQR, 0.64-0.89]; P = .94). Intern performance on individual domains of knowledge for each block is listed in Appendix Table 1.

Intervention Performance vs Controls

The 2016 historical control had significantly higher scores compared with the 2017 preintervention group (P < .001; Appendix Table 2). The year-long lecture series resulted in significant increases in median scores for the 2017 group (median preintervention score 0.55 [0.41-0.61] vs median postintervention score 0.84 [0.71-0.90]; P = .006; Appendix Table 1). At the end of the study, the 2017 postintervention scores were significantly higher across multiple knowledge domains compared with the 2016 historical control (Appendix Table 2).

Survey Results

Notably, the 2017 intern class at the end of the intervention did not have significantly different assessment scores for several disease-specific domains, compared with the 2016 control (Appendix Table 2). Nonetheless, the 2017 intern class reported higher levels of confidence in these same domains despite similar scores (Supplementary Figure). The HUD group seldomly cited a lack of confidence in their abilities as a barrier to performing POCUS examinations (17.6%), compared with the no-HUD group (50.0%), despite nearly identical assessment scores between the two groups (Table 1).

Previous guidelines have recommended increased HUD access for learners,^13,24,35,36 but there have been few investigations that have evaluated the impact of such access on learning POCUS. One previous investigation found that hospitalists who carried HUDs were more likely to identify heart failure on bedside examination.³⁷ In contrast, our study found no improvement in interpretative ability when randomizing interns to carry HUDs for patient care. Notably, interns did not perform more POCUS examinations when given HUDs. We offer several explanations for this finding. First, time-motion studies have demonstrated that internal medicine interns spend less than 15% of their time toward direct patient care.³⁸ It is possible that the demands of being an intern impeded their ability to perform more POCUS examinations on their patients, regardless of HUD access. Alternatively, the interns randomized to no personal access may have used the community device more frequently as a result of the lecture series. Given the cost of HUDs, further studies are needed to assess the degree to which HUD access will improve trainee interpretive ability, especially as more training programs consider the creation of ultrasound curricula.^{10,11,24,39,40}

This study was unique because it followed interns over a year-long course that repeated the same material to assess rates of learning with repeated exposure. Learners improved their scores after the first, but not second, block. Furthermore, the median scores were nearly identical between the first postblock assessment and second preblock assessment (0.81 vs 0.78), suggesting that knowledge was retained between blocks. Together, these findings suggest there may be limitations of traditional lectures that use standardized patient models for practice. Supplementary pedagogies, such as in-the-moment feedback with actual patients, may be needed to promote mastery.^14,35

Despite no formal curriculum, the 2016 intern class (historical control) had learned POCUS to some degree based on their higher assessment scores compared with the 2017 intern class during the preintervention period. Such learning may be informal, and yet, trainees may feel confident in making clinical decisions without formalized training, accreditation, or oversight. As suggested by this study, adding regular didactics or giving trainees HUDs may not immediately solve this issue. For assessment items in which the 2017 interns did not significantly differ from the controls, they nonetheless reported higher confidence in their abilities. Similarly, interns randomized to HUDs less frequently cited a lack of confidence in their abilities, despite similar scores to the no-HUD group. Such confidence may be incongruent with their actual knowledge or ability to safely use POCUS. This phenomenon of misplaced confidence is known as the Dunning–Kruger effect, and it may be common with ultrasound learning.⁴¹ While confidence can be part of a holistic definition of competency,¹⁴ these results raise the concern that trainees may have difficulty assessing their own competency level with POCUS.³⁵

There are several limitations to this study. It was performed at a single institution with limited sample size. It examined only intern physicians because of funding constraints, which limits the generalizability of these findings among medical trainees. Technical ability assessments (including obtaining and interpreting images) were not included. We were unable to track the timing or location of the devices’ usage, and the interns’ self-reported usage rates may be subject to recall bias. To our knowledge, there were no significant lapses in device availability/functionality. Intern physicians in the HUD arm did not receive formal feedback on personally acquired patient images, which may have limited the intervention’s impact.

In conclusion, internal medicine interns who received personal HUDs were not better at recognizing normal/abnormal findings on image assessments, and they did not report performing more POCUS examinations. Since the minority of a trainee’s time is spent toward direct patient care, offering trainees HUDs without substantial guidance may not be enough to promote mastery. Notably, trainees who received HUDs felt more confident in their abilities, despite no objective increase in their actual skill. Finally, interns who received POCUS-related lectures experienced significant benefit upon first exposure to the material, while repeated exposures did not improve performance. Future investigations should stringently track trainee POCUS usage rates with HUDs and assess whether image acquisition ability improves as a result of personal access.

Given that multiple disciplines are often involved in caring for patients admitted to the hospital, timely communication, collaboration, and coordination amongst various disciplines is necessary for safe and effective patient care.¹ With the focus on improving patient satisfaction and throughput in hospitals, it is also important to make more accurate predictions of the discharge date and allow time for patients and their families to prepare for discharge.^2-4

Multidisciplinary rounds (MDR) are defined as structured daily communication amongst key members of the patient’s care team (eg, nurses, physicians, case managers, social workers, pharmacists, and rehabilitation services). MDR have shown to be a useful strategy for ensuring that all members of the care team are updated on the plan of care for the patient.⁵ During MDR, a brief “check-in” discussing the patient’s plan of care, pending needs, and barriers to discharge allows all team members, patients, and families to effectively coordinate care and plan and prepare for discharge.

Multiple studies have reported increased collaboration and improved communication between disciplines with the use of such multidisciplinary rounding.^2,5-7 Additionally, MDR have been shown to improve patient outcomes⁸ and reduce adverse events,⁹ length of stay (LOS),^6,8 cost of care,⁸ and readmissions.¹

We redesigned MDR on the general medicine wards at our institution in October 2014 by using Lean management techniques. Lean is defined as a set of philosophies and methods that aim to create transformation in thinking, behavior, and culture in each process, with the goal of maximizing the value for the patients and providers, adding efficiency, and reducing waste and waits.¹⁰

In this study, we evaluate whether this new model of MDR was associated with a decrease in the LOS. We also evaluate whether this new model of MDR was associated with an increase in discharges before noon, documentation of estimated discharge date (EDD) in our electronic health record (EHR), and patient satisfaction.

Setting, Design, and Patients

The study was conducted on the teaching general medicine service at our institution, an urban, 484-bed academic hospital. The general medicine service has patients on 4 inpatient units (total of 95 beds) and is managed by 5 teaching service teams.

We performed a pre-post study. The preperiod (in which the old model of MDR was followed) included 4000 patients discharged between September 1, 2013, and October 22, 2014. The postperiod (in which the new model of MDR was followed) included 2085 patients discharged between October 23, 2014, and April 30, 2015. We excluded 139 patients that died in the hospital prior to discharge and patients on the nonteaching and/or private practice service.

All data were provided by our institution’s Digital Solutions Department. Our institutional review board issued a letter of determination exempting this study from further review because it was deemed to be a quality improvement initiative.

Use of Lean Management to Redesign our MDR

Our institution has incorporated the Lean management system to continually add value to services through the elimination of waste, thus simultaneously optimizing the quality of patient care, cost, and patient satisfaction.¹¹ Lean, derived from the Toyota Production System, has long been used in manufacturing and in recent decades has spread to healthcare.¹² We leveraged the following 3 key Lean techniques to redesign our MDR: (1) value stream management (VSM), (2) rapid process improvement workshops (RPIW), and (3) active daily management (ADM), as detailed in supplementary Appendix 1.

Interventions

Outcomes

The primary outcome was mean LOS. The secondary outcomes were (1) discharges before noon, (2) recording of the EDD in our EHR within 24 hours of admission (as time stamped on our EHR), and (3) patient satisfaction.

Data for patient satisfaction were obtained using the Press Ganey survey. We used data on patient satisfaction scores for the following 2 relevant questions on this survey: (1) extent to which the patient felt ready to be discharged and (2) how well staff worked together to care for the patient. Proportions of the “top-box” (“very good”) were used for the analysis. These survey data were available on 467 patients (11.7%) in the preperiod and 188 patients (9.0%) in the postperiod.

Data Analysis

A sensitivity analysis was conducted on a second cohort that included a subset of patients from the preperiod between November 1, 2013, and April 30, 2014, and a subset of patients from the postperiod between November 1, 2014, and April 1, 2015, to control for the calendar period (supplementary Appendix 2).

All analyses were conducted in R version 3.3.0, with the linear mixed-effects model lme4 statistical package.^13,14

Table 3 shows the differences in the outcomes between the pre- and postperiods. There was no change in the LOS or LOS adjusted for CMI. There was a 3.9% increase in discharges before noon in the postperiod compared with the preperiod (95% CI, 2.4% to 5.3%; P < .001). There was a 9.9% increase in the percentage of patients for whom the EDD was recorded in our EHR within 24 hours of admission (95% CI, 7.4% to 12.4%; P < .001). There was no change in the “top-box” patient satisfaction scores.

There were only marginal differences in the results between the entire cohort and a second subset cohort used for sensitivity analysis (supplementary Appendix 2).

In our study, there was no change in the mean LOS with the new model of MDR. There was an increase in discharges before noon and in recording of the EDD in our EHR within 24 hours of admission in the postperiod when the Lean-based new model of MDR was utilized. There was no change in patient satisfaction. With no change in staffing, we were able to accommodate the increase in the discharge volume in the postperiod.

We believe our results are attributable to several factors, including clearly defined roles and responsibilities for all participants of MDR, the inclusion of more experienced general medicine attending physician (compared with housestaff), Lean management techniques to identify gaps in the patient’s journey from emergency department to discharge using VSM, the development of appropriate workflows and standard work on how the multidisciplinary teams would work together at RPIWs, and ADM to ensure sustainability and engagement among frontline members and institutional leaders. In order to sustain this, we planned to continue monitoring data in daily, weekly, and monthly forums with senior physician and administrative leaders. Planning for additional interventions is underway, including moving MDR to the bedside, instituting an afternoon “check-in” that would enable more detailed action planning, and addressing barriers in a timely manner for patients ready to discharge the following day.

Our study has a few limitations. First, this is an observational study that cannot determine causation. Second, this is a single-center study conducted on patients only on the general medicine teaching service. Third, there were several concurrent interventions implemented at our institution to improve LOS, throughput, and patient satisfaction in addition to MDR, thus making it difficult to isolate the impact of our intervention. Fourth, in the new model of MDR, rounds took place only 5 days per week, thereby possibly limiting the potential impact on our outcomes. Fifth, while we showed improvements in the discharges before noon and recording of EDD in the post period, we were not able to achieve our target of 25% discharges before noon or 100% recording of EDD in this time period. We believe the limited amount of time between the pre- and postperiods to allow for adoption and learning of the processes might have contributed to the underestimation of the impact of the new model of MDR, thereby limiting our ability to achieve our targets. Sixth, the response rate on the Press Ganey survey was low, and we did not directly survey patients or families for their satisfaction with MDR.

Our study has several strengths. To our knowledge, this is the first study to embed Lean management techniques in the design of MDR in the inpatient setting. While several studies have demonstrated improvements in discharges before noon through the implementation of MDR, they have not incorporated Lean management techniques, which we believe are critical to ensure the sustainability of results.^1,3,5,6,8,15 Second, while it was not measured, there was a high level of provider engagement in the process in the new model of MDR. Third, because the MDR were conducted at the nurse’s station on each inpatient unit in the new model instead of in a conference room, it was well attended by all members of the multidisciplinary team. Fourth, the presence of a visibility board allowed for all team members to have easy access to visual feedback throughout the day, even if they were not present at the MDR. Fifth, we believe that there was also more accurate estimation of the date and time of discharge in the new model of MDR because the discussion was facilitated by the case manager, who is experienced in identifying barriers to discharge (compared with the housestaff in the old model of MDR), and included the more experienced attending physician. Finally, the consistent presence of a multidisciplinary team at MDR allowed for the incorporation of everyone’s concerns at one time, thereby limiting the need for paging multiple disciplines throughout the day, which led to quicker resolution of issues and assignment of pending tasks.

In conclusion, our study shows no change in the mean LOS when the Lean-based model of MDR was utilized. Our study demonstrates an increase in discharges before noon and in recording of EDD on our EHR within 24 hours of admission in the post period when the Lean-based model of MDR was utilized. There was no change in patient satisfaction. While this study was conducted at an academic medical center on the general medicine wards, we believe our new model of MDR, which leveraged Lean management techniques, may successfully impact patient flow in all inpatient clinical services and nonteaching hospitals.

Disclosure

The authors report no financial conflicts of interest and have nothing to disclose.

Given that multiple disciplines are often involved in caring for patients admitted to the hospital, timely communication, collaboration, and coordination amongst various disciplines is necessary for safe and effective patient care.¹ With the focus on improving patient satisfaction and throughput in hospitals, it is also important to make more accurate predictions of the discharge date and allow time for patients and their families to prepare for discharge.^2-4

Multidisciplinary rounds (MDR) are defined as structured daily communication amongst key members of the patient’s care team (eg, nurses, physicians, case managers, social workers, pharmacists, and rehabilitation services). MDR have shown to be a useful strategy for ensuring that all members of the care team are updated on the plan of care for the patient.⁵ During MDR, a brief “check-in” discussing the patient’s plan of care, pending needs, and barriers to discharge allows all team members, patients, and families to effectively coordinate care and plan and prepare for discharge.

Multiple studies have reported increased collaboration and improved communication between disciplines with the use of such multidisciplinary rounding.^2,5-7 Additionally, MDR have been shown to improve patient outcomes⁸ and reduce adverse events,⁹ length of stay (LOS),^6,8 cost of care,⁸ and readmissions.¹

We redesigned MDR on the general medicine wards at our institution in October 2014 by using Lean management techniques. Lean is defined as a set of philosophies and methods that aim to create transformation in thinking, behavior, and culture in each process, with the goal of maximizing the value for the patients and providers, adding efficiency, and reducing waste and waits.¹⁰

In this study, we evaluate whether this new model of MDR was associated with a decrease in the LOS. We also evaluate whether this new model of MDR was associated with an increase in discharges before noon, documentation of estimated discharge date (EDD) in our electronic health record (EHR), and patient satisfaction.

Setting, Design, and Patients

The study was conducted on the teaching general medicine service at our institution, an urban, 484-bed academic hospital. The general medicine service has patients on 4 inpatient units (total of 95 beds) and is managed by 5 teaching service teams.

We performed a pre-post study. The preperiod (in which the old model of MDR was followed) included 4000 patients discharged between September 1, 2013, and October 22, 2014. The postperiod (in which the new model of MDR was followed) included 2085 patients discharged between October 23, 2014, and April 30, 2015. We excluded 139 patients that died in the hospital prior to discharge and patients on the nonteaching and/or private practice service.

All data were provided by our institution’s Digital Solutions Department. Our institutional review board issued a letter of determination exempting this study from further review because it was deemed to be a quality improvement initiative.

Use of Lean Management to Redesign our MDR

Our institution has incorporated the Lean management system to continually add value to services through the elimination of waste, thus simultaneously optimizing the quality of patient care, cost, and patient satisfaction.¹¹ Lean, derived from the Toyota Production System, has long been used in manufacturing and in recent decades has spread to healthcare.¹² We leveraged the following 3 key Lean techniques to redesign our MDR: (1) value stream management (VSM), (2) rapid process improvement workshops (RPIW), and (3) active daily management (ADM), as detailed in supplementary Appendix 1.

Interventions

Outcomes

The primary outcome was mean LOS. The secondary outcomes were (1) discharges before noon, (2) recording of the EDD in our EHR within 24 hours of admission (as time stamped on our EHR), and (3) patient satisfaction.

Data for patient satisfaction were obtained using the Press Ganey survey. We used data on patient satisfaction scores for the following 2 relevant questions on this survey: (1) extent to which the patient felt ready to be discharged and (2) how well staff worked together to care for the patient. Proportions of the “top-box” (“very good”) were used for the analysis. These survey data were available on 467 patients (11.7%) in the preperiod and 188 patients (9.0%) in the postperiod.

Data Analysis

A sensitivity analysis was conducted on a second cohort that included a subset of patients from the preperiod between November 1, 2013, and April 30, 2014, and a subset of patients from the postperiod between November 1, 2014, and April 1, 2015, to control for the calendar period (supplementary Appendix 2).

All analyses were conducted in R version 3.3.0, with the linear mixed-effects model lme4 statistical package.^13,14

Table 3 shows the differences in the outcomes between the pre- and postperiods. There was no change in the LOS or LOS adjusted for CMI. There was a 3.9% increase in discharges before noon in the postperiod compared with the preperiod (95% CI, 2.4% to 5.3%; P < .001). There was a 9.9% increase in the percentage of patients for whom the EDD was recorded in our EHR within 24 hours of admission (95% CI, 7.4% to 12.4%; P < .001). There was no change in the “top-box” patient satisfaction scores.

There were only marginal differences in the results between the entire cohort and a second subset cohort used for sensitivity analysis (supplementary Appendix 2).

In our study, there was no change in the mean LOS with the new model of MDR. There was an increase in discharges before noon and in recording of the EDD in our EHR within 24 hours of admission in the postperiod when the Lean-based new model of MDR was utilized. There was no change in patient satisfaction. With no change in staffing, we were able to accommodate the increase in the discharge volume in the postperiod.

We believe our results are attributable to several factors, including clearly defined roles and responsibilities for all participants of MDR, the inclusion of more experienced general medicine attending physician (compared with housestaff), Lean management techniques to identify gaps in the patient’s journey from emergency department to discharge using VSM, the development of appropriate workflows and standard work on how the multidisciplinary teams would work together at RPIWs, and ADM to ensure sustainability and engagement among frontline members and institutional leaders. In order to sustain this, we planned to continue monitoring data in daily, weekly, and monthly forums with senior physician and administrative leaders. Planning for additional interventions is underway, including moving MDR to the bedside, instituting an afternoon “check-in” that would enable more detailed action planning, and addressing barriers in a timely manner for patients ready to discharge the following day.

Our study has a few limitations. First, this is an observational study that cannot determine causation. Second, this is a single-center study conducted on patients only on the general medicine teaching service. Third, there were several concurrent interventions implemented at our institution to improve LOS, throughput, and patient satisfaction in addition to MDR, thus making it difficult to isolate the impact of our intervention. Fourth, in the new model of MDR, rounds took place only 5 days per week, thereby possibly limiting the potential impact on our outcomes. Fifth, while we showed improvements in the discharges before noon and recording of EDD in the post period, we were not able to achieve our target of 25% discharges before noon or 100% recording of EDD in this time period. We believe the limited amount of time between the pre- and postperiods to allow for adoption and learning of the processes might have contributed to the underestimation of the impact of the new model of MDR, thereby limiting our ability to achieve our targets. Sixth, the response rate on the Press Ganey survey was low, and we did not directly survey patients or families for their satisfaction with MDR.

Our study has several strengths. To our knowledge, this is the first study to embed Lean management techniques in the design of MDR in the inpatient setting. While several studies have demonstrated improvements in discharges before noon through the implementation of MDR, they have not incorporated Lean management techniques, which we believe are critical to ensure the sustainability of results.^1,3,5,6,8,15 Second, while it was not measured, there was a high level of provider engagement in the process in the new model of MDR. Third, because the MDR were conducted at the nurse’s station on each inpatient unit in the new model instead of in a conference room, it was well attended by all members of the multidisciplinary team. Fourth, the presence of a visibility board allowed for all team members to have easy access to visual feedback throughout the day, even if they were not present at the MDR. Fifth, we believe that there was also more accurate estimation of the date and time of discharge in the new model of MDR because the discussion was facilitated by the case manager, who is experienced in identifying barriers to discharge (compared with the housestaff in the old model of MDR), and included the more experienced attending physician. Finally, the consistent presence of a multidisciplinary team at MDR allowed for the incorporation of everyone’s concerns at one time, thereby limiting the need for paging multiple disciplines throughout the day, which led to quicker resolution of issues and assignment of pending tasks.

In conclusion, our study shows no change in the mean LOS when the Lean-based model of MDR was utilized. Our study demonstrates an increase in discharges before noon and in recording of EDD on our EHR within 24 hours of admission in the post period when the Lean-based model of MDR was utilized. There was no change in patient satisfaction. While this study was conducted at an academic medical center on the general medicine wards, we believe our new model of MDR, which leveraged Lean management techniques, may successfully impact patient flow in all inpatient clinical services and nonteaching hospitals.

Disclosure

The authors report no financial conflicts of interest and have nothing to disclose.

Given that multiple disciplines are often involved in caring for patients admitted to the hospital, timely communication, collaboration, and coordination amongst various disciplines is necessary for safe and effective patient care.¹ With the focus on improving patient satisfaction and throughput in hospitals, it is also important to make more accurate predictions of the discharge date and allow time for patients and their families to prepare for discharge.^2-4

Multidisciplinary rounds (MDR) are defined as structured daily communication amongst key members of the patient’s care team (eg, nurses, physicians, case managers, social workers, pharmacists, and rehabilitation services). MDR have shown to be a useful strategy for ensuring that all members of the care team are updated on the plan of care for the patient.⁵ During MDR, a brief “check-in” discussing the patient’s plan of care, pending needs, and barriers to discharge allows all team members, patients, and families to effectively coordinate care and plan and prepare for discharge.

Multiple studies have reported increased collaboration and improved communication between disciplines with the use of such multidisciplinary rounding.^2,5-7 Additionally, MDR have been shown to improve patient outcomes⁸ and reduce adverse events,⁹ length of stay (LOS),^6,8 cost of care,⁸ and readmissions.¹

We redesigned MDR on the general medicine wards at our institution in October 2014 by using Lean management techniques. Lean is defined as a set of philosophies and methods that aim to create transformation in thinking, behavior, and culture in each process, with the goal of maximizing the value for the patients and providers, adding efficiency, and reducing waste and waits.¹⁰

In this study, we evaluate whether this new model of MDR was associated with a decrease in the LOS. We also evaluate whether this new model of MDR was associated with an increase in discharges before noon, documentation of estimated discharge date (EDD) in our electronic health record (EHR), and patient satisfaction.

Setting, Design, and Patients

The study was conducted on the teaching general medicine service at our institution, an urban, 484-bed academic hospital. The general medicine service has patients on 4 inpatient units (total of 95 beds) and is managed by 5 teaching service teams.

We performed a pre-post study. The preperiod (in which the old model of MDR was followed) included 4000 patients discharged between September 1, 2013, and October 22, 2014. The postperiod (in which the new model of MDR was followed) included 2085 patients discharged between October 23, 2014, and April 30, 2015. We excluded 139 patients that died in the hospital prior to discharge and patients on the nonteaching and/or private practice service.

All data were provided by our institution’s Digital Solutions Department. Our institutional review board issued a letter of determination exempting this study from further review because it was deemed to be a quality improvement initiative.

Use of Lean Management to Redesign our MDR

Our institution has incorporated the Lean management system to continually add value to services through the elimination of waste, thus simultaneously optimizing the quality of patient care, cost, and patient satisfaction.¹¹ Lean, derived from the Toyota Production System, has long been used in manufacturing and in recent decades has spread to healthcare.¹² We leveraged the following 3 key Lean techniques to redesign our MDR: (1) value stream management (VSM), (2) rapid process improvement workshops (RPIW), and (3) active daily management (ADM), as detailed in supplementary Appendix 1.

Interventions

Outcomes

The primary outcome was mean LOS. The secondary outcomes were (1) discharges before noon, (2) recording of the EDD in our EHR within 24 hours of admission (as time stamped on our EHR), and (3) patient satisfaction.

Data for patient satisfaction were obtained using the Press Ganey survey. We used data on patient satisfaction scores for the following 2 relevant questions on this survey: (1) extent to which the patient felt ready to be discharged and (2) how well staff worked together to care for the patient. Proportions of the “top-box” (“very good”) were used for the analysis. These survey data were available on 467 patients (11.7%) in the preperiod and 188 patients (9.0%) in the postperiod.

Data Analysis

A sensitivity analysis was conducted on a second cohort that included a subset of patients from the preperiod between November 1, 2013, and April 30, 2014, and a subset of patients from the postperiod between November 1, 2014, and April 1, 2015, to control for the calendar period (supplementary Appendix 2).

All analyses were conducted in R version 3.3.0, with the linear mixed-effects model lme4 statistical package.^13,14

Table 3 shows the differences in the outcomes between the pre- and postperiods. There was no change in the LOS or LOS adjusted for CMI. There was a 3.9% increase in discharges before noon in the postperiod compared with the preperiod (95% CI, 2.4% to 5.3%; P < .001). There was a 9.9% increase in the percentage of patients for whom the EDD was recorded in our EHR within 24 hours of admission (95% CI, 7.4% to 12.4%; P < .001). There was no change in the “top-box” patient satisfaction scores.

There were only marginal differences in the results between the entire cohort and a second subset cohort used for sensitivity analysis (supplementary Appendix 2).

In our study, there was no change in the mean LOS with the new model of MDR. There was an increase in discharges before noon and in recording of the EDD in our EHR within 24 hours of admission in the postperiod when the Lean-based new model of MDR was utilized. There was no change in patient satisfaction. With no change in staffing, we were able to accommodate the increase in the discharge volume in the postperiod.

We believe our results are attributable to several factors, including clearly defined roles and responsibilities for all participants of MDR, the inclusion of more experienced general medicine attending physician (compared with housestaff), Lean management techniques to identify gaps in the patient’s journey from emergency department to discharge using VSM, the development of appropriate workflows and standard work on how the multidisciplinary teams would work together at RPIWs, and ADM to ensure sustainability and engagement among frontline members and institutional leaders. In order to sustain this, we planned to continue monitoring data in daily, weekly, and monthly forums with senior physician and administrative leaders. Planning for additional interventions is underway, including moving MDR to the bedside, instituting an afternoon “check-in” that would enable more detailed action planning, and addressing barriers in a timely manner for patients ready to discharge the following day.

Our study has a few limitations. First, this is an observational study that cannot determine causation. Second, this is a single-center study conducted on patients only on the general medicine teaching service. Third, there were several concurrent interventions implemented at our institution to improve LOS, throughput, and patient satisfaction in addition to MDR, thus making it difficult to isolate the impact of our intervention. Fourth, in the new model of MDR, rounds took place only 5 days per week, thereby possibly limiting the potential impact on our outcomes. Fifth, while we showed improvements in the discharges before noon and recording of EDD in the post period, we were not able to achieve our target of 25% discharges before noon or 100% recording of EDD in this time period. We believe the limited amount of time between the pre- and postperiods to allow for adoption and learning of the processes might have contributed to the underestimation of the impact of the new model of MDR, thereby limiting our ability to achieve our targets. Sixth, the response rate on the Press Ganey survey was low, and we did not directly survey patients or families for their satisfaction with MDR.

Our study has several strengths. To our knowledge, this is the first study to embed Lean management techniques in the design of MDR in the inpatient setting. While several studies have demonstrated improvements in discharges before noon through the implementation of MDR, they have not incorporated Lean management techniques, which we believe are critical to ensure the sustainability of results.^1,3,5,6,8,15 Second, while it was not measured, there was a high level of provider engagement in the process in the new model of MDR. Third, because the MDR were conducted at the nurse’s station on each inpatient unit in the new model instead of in a conference room, it was well attended by all members of the multidisciplinary team. Fourth, the presence of a visibility board allowed for all team members to have easy access to visual feedback throughout the day, even if they were not present at the MDR. Fifth, we believe that there was also more accurate estimation of the date and time of discharge in the new model of MDR because the discussion was facilitated by the case manager, who is experienced in identifying barriers to discharge (compared with the housestaff in the old model of MDR), and included the more experienced attending physician. Finally, the consistent presence of a multidisciplinary team at MDR allowed for the incorporation of everyone’s concerns at one time, thereby limiting the need for paging multiple disciplines throughout the day, which led to quicker resolution of issues and assignment of pending tasks.

In conclusion, our study shows no change in the mean LOS when the Lean-based model of MDR was utilized. Our study demonstrates an increase in discharges before noon and in recording of EDD on our EHR within 24 hours of admission in the post period when the Lean-based model of MDR was utilized. There was no change in patient satisfaction. While this study was conducted at an academic medical center on the general medicine wards, we believe our new model of MDR, which leveraged Lean management techniques, may successfully impact patient flow in all inpatient clinical services and nonteaching hospitals.

Disclosure

The authors report no financial conflicts of interest and have nothing to disclose.

Venous thromboembolism (VTE) affects more than 1 million patients and costs the US healthcare system more than $1.5 billion annually.¹ Inherited and acquired thrombophilias have been perceived as important risk factors in assessing the risk of VTE recurrence and guiding the duration of anticoagulation.

Thrombophilias increase the risk of a first thrombotic event, but existing data have failed to demonstrate the usefulness of routine thrombophilia screening on subsequent management.^2,3 Moreover, thrombophilia testing ordered in the context of an inpatient hospitalization is limited by confounding factors, especially during an acute thrombotic event or in the setting of concurrent anticoagulation.⁴

Recognizing the costliness of routine thrombophilia testing, The American Society of Hematology introduced its Choosing Wisely campaign in 2013 in an effort to reduce test ordering in the setting of provoked VTEs with a major transient risk factor.⁵ In order to define current practice behavior at our institution, we conducted a retrospective study to determine the magnitude and financial impact of potentially inappropriate thrombophilia testing in the inpatient setting.

We performed a retrospective analysis of thrombophilia testing across all inpatient services at a large, quaternary-care academic institution over a 2-year period. Electronic medical record data containing all thrombophilia tests ordered on inpatients from June 2013 to June 2015 were obtained. This study was exempt from institutional review board approval.

Inclusion criteria included any inpatient for which thrombophilia testing occurred. Patients were excluded if testing was ordered in the absence of VTE or arterial thrombosis or if it was ordered as part of a work-up for another medical condition (see Supplementary Material).

Thrombophilia testing was defined as any of the following: inherited thrombophilias (Factor V Leiden or prothrombin 20210 gene mutations, antithrombin, or protein C or S activity levels) or acquired thrombophilias (lupus anticoagulant [Testing refers to the activated partial thromboplastin time lupus assay.], beta-2 glycoprotein 1 immunoglobulins M and G, anticardiolipin immunoglobulins M and G, dilute Russell’s viper venom time, or JAK2 V617F mutations).

Extracted data included patient age, sex, type of thrombophilia test ordered, ordering primary service, admission diagnosis, and objective confirmation of thrombotic events. The indication for test ordering was determined via medical record review of the patient’s corresponding hospitalization. Each test was evaluated in the context of the patient’s presenting history, hospital course, active medications, accompanying laboratory and radiographic studies, and consultant recommendations to arrive at a conclusion regarding both the test’s reason for ordering and whether its indication was “inappropriate,” “appropriate,” or “equivocal.” Cost data were obtained through the Centers for Medicare & Medicaid Services (CMS) Clinical Laboratory Fee Schedule for 2016 (see Supplementary Material).⁶

The criteria for defining test appropriateness were formulated by utilizing a combination of major society guidelines and literature review.^5,7-10 The criteria placed emphasis upon the ordered tests’ clinical relevance and reliability and were subsequently reviewed by a senior hematologist with specific expertise in thrombosis (see Supplementary Material).

Two internal medicine resident physician data reviewers independently evaluated the ordered tests. To ensure consistency between reviewers, a sample of identical test orders was compared for concordance, and a Cohen’s kappa coefficient was calculated. For purposes of analysis, equivocal orders were included under the appropriate category, as this study focused on the quantification of potentially inappropriate ordering practices. Pearson chi-square testing was performed in order to compare ordering practices between services using Stata.¹¹

In total, we reviewed 2179 individual tests, of which 362 (16.6%) were excluded. The remaining 1817 tests involved 299 patients across 26 primary specialties. Fifty-two (2.9% of orders) were ultimately deemed equivocal. The Table illustrates the overall proportion and cost of inappropriate test ordering as well as testing characteristics of the most commonly encountered thrombotic diagnoses. The Figure illustrates the proportion of potentially inappropriate test ordering with its associated cost by test type.

Orders for Factor V Leiden, prothrombin 20210, and protein C and S activity levels were most commonly deemed inappropriate due to the test results’ failure to alter clinical management (97.3%, 99.2%, 99.4% of their inappropriate orders, respectively). Antithrombin testing (59.4%) was deemed inappropriate most commonly in the setting of acute thrombosis. The lupus anticoagulant (82.8%) was inappropriately ordered most frequently in the setting of concurrent anticoagulation.

This retrospective analysis represents one of the largest examinations of inpatient thrombophilia testing practices to date. Our results illustrate the high prevalence and significant financial impact of potentially inappropriate thrombophilia testing conducted in the inpatient setting. The data confirm that, per our defined criteria, more than 90% of inherited thrombophilia testing was potentially inappropriate while the majority of acquired thrombophilia testing was appropriate, with the exception of the lupus anticoagulant.

Even when appropriately ordered, studies suggest that positive thrombophilia screening results fail to impact outcomes in most patients with VTE. In an effort to evaluate positive results’ potential to provide a basis from which to extend the duration of anticoagulation, and therefore reduce the risk of a recurrent VTE, a case-control analysis was performed on a series of patients with a first-VTE event (Multiple Environmental and Genetic Assessment of risk factors for venous thrombosis [MEGA] study).³ In examining the odds ratio (OR) for recurrence between patients who did or did not undergo testing for Factor V Leiden, antithrombin, or protein C or S activity, the data failed to show an impact of testing on the risk of VTE recurrence (OR 1.2; confidence interval, 0.8-1.8). In fact, decision making has increasingly relied on patients’ clinical characteristics rather than thrombophilia test results to guide anticoagulation duration after incident VTEs. A 2017 study illustrated that when using a clinical decision rule (Clinical Decision Rule Validation Study to Predict Low Recurrent Risk in Patients With Unprovoked Venous Thromboembolism [REVERSE criteria]) in patients with a first, unprovoked VTE, routine thrombophilia screening added little to determining the need for prolonged anticoagulation.¹² These findings support the limited clinical utility of current test ordering practices for the prediction and management of recurrent venous thrombosis.

Regarding the acquired thrombophilias, antiphospholipid antibody testing was predominantly ordered in a justified manner, which is consistent with the notion that test results could affect clinical management, such as anticoagulation duration or choice of anticoagulant.¹³ However, the validity of lupus anticoagulant testing was limited by the frequency of patients on concurrent anticoagulation.

Financially, the cumulative cost associated with inappropriate ordering was substantial, regardless of the thrombotic event in question. Moreover, our calculated costs are derived from CMS reimbursement rates and likely underestimate the true financial impact of errant testing given that commercial laboratories frequently charge at rates several-fold higher. On a national scale, prior analyses have suggested that the annual cost of thrombophilia testing, based on typical commercial rates, ranges from $300 million to $672 million.¹⁴

Researchers in prior studies have similarly examined the frequency of inappropriate thrombophilia testing and methods to reduce it. Researchers in a 2014 study demonstrated initially high rates of inappropriate inherited thrombophilia testing, and then showed marked reductions in testing and cost savings across multiple specialties following the introduction of a flowchart on a preprinted order form.¹⁵ Our findings provide motivation to perform similar endeavors.

The proportional difference of inappropriate ordering observed between nonteaching- and teaching-medicine services indicates a potential role for educational interventions. We recently completed a series of lectures on high-value thrombophilia ordering for residents and are actively analyzing its impact on subsequent ordering practices. We are also piloting an electronic best practice advisory for thrombophilia test ordering. Though the advisory may be overridden, providers are asked to provide justification for doing so on a voluntary basis. We plan to evaluate its effect on our findings reported in this study.

We acknowledge that our exclusion criteria resulted in the omission of testing across a spectrum of nonthrombotic clinical conditions, raising the question of selection bias. Because there are no established guidelines to determine the appropriateness of testing in these scenarios, we chose to limit the analysis of errant ordering to the context of thrombotic events. Other limitations of this study include the analysis of equivocal orders as appropriate. However, because equivocal ordering represented less than 3% of all analyzed orders, including these as inappropriate would not have significantly altered our findings.

A review of thrombophilia testing practices at our institution demonstrated that inappropriate testing in the inpatient setting is a frequent phenomenon associated with a significant financial impact. This effect was more pronounced in inherited versus acquired thrombophilia testing. Testing was frequently confounded and often failed to impact patients’ short- or long-term clinical management, regardless of the result.

These findings serve as a strong impetus to reduce the burden of routine thrombophilia testing during hospital admissions. Our data demonstrate a need for institution-wide changes such as implementing best practice advisories, introducing ordering restrictions, and conducting educational interventions in order to reduce unnecessary expenditures and improve patient care.

Disclosure

The authors have nothing to disclose.

Venous thromboembolism (VTE) affects more than 1 million patients and costs the US healthcare system more than $1.5 billion annually.¹ Inherited and acquired thrombophilias have been perceived as important risk factors in assessing the risk of VTE recurrence and guiding the duration of anticoagulation.

Thrombophilias increase the risk of a first thrombotic event, but existing data have failed to demonstrate the usefulness of routine thrombophilia screening on subsequent management.^2,3 Moreover, thrombophilia testing ordered in the context of an inpatient hospitalization is limited by confounding factors, especially during an acute thrombotic event or in the setting of concurrent anticoagulation.⁴

Recognizing the costliness of routine thrombophilia testing, The American Society of Hematology introduced its Choosing Wisely campaign in 2013 in an effort to reduce test ordering in the setting of provoked VTEs with a major transient risk factor.⁵ In order to define current practice behavior at our institution, we conducted a retrospective study to determine the magnitude and financial impact of potentially inappropriate thrombophilia testing in the inpatient setting.

We performed a retrospective analysis of thrombophilia testing across all inpatient services at a large, quaternary-care academic institution over a 2-year period. Electronic medical record data containing all thrombophilia tests ordered on inpatients from June 2013 to June 2015 were obtained. This study was exempt from institutional review board approval.

Inclusion criteria included any inpatient for which thrombophilia testing occurred. Patients were excluded if testing was ordered in the absence of VTE or arterial thrombosis or if it was ordered as part of a work-up for another medical condition (see Supplementary Material).

Thrombophilia testing was defined as any of the following: inherited thrombophilias (Factor V Leiden or prothrombin 20210 gene mutations, antithrombin, or protein C or S activity levels) or acquired thrombophilias (lupus anticoagulant [Testing refers to the activated partial thromboplastin time lupus assay.], beta-2 glycoprotein 1 immunoglobulins M and G, anticardiolipin immunoglobulins M and G, dilute Russell’s viper venom time, or JAK2 V617F mutations).

Extracted data included patient age, sex, type of thrombophilia test ordered, ordering primary service, admission diagnosis, and objective confirmation of thrombotic events. The indication for test ordering was determined via medical record review of the patient’s corresponding hospitalization. Each test was evaluated in the context of the patient’s presenting history, hospital course, active medications, accompanying laboratory and radiographic studies, and consultant recommendations to arrive at a conclusion regarding both the test’s reason for ordering and whether its indication was “inappropriate,” “appropriate,” or “equivocal.” Cost data were obtained through the Centers for Medicare & Medicaid Services (CMS) Clinical Laboratory Fee Schedule for 2016 (see Supplementary Material).⁶

The criteria for defining test appropriateness were formulated by utilizing a combination of major society guidelines and literature review.^5,7-10 The criteria placed emphasis upon the ordered tests’ clinical relevance and reliability and were subsequently reviewed by a senior hematologist with specific expertise in thrombosis (see Supplementary Material).

Two internal medicine resident physician data reviewers independently evaluated the ordered tests. To ensure consistency between reviewers, a sample of identical test orders was compared for concordance, and a Cohen’s kappa coefficient was calculated. For purposes of analysis, equivocal orders were included under the appropriate category, as this study focused on the quantification of potentially inappropriate ordering practices. Pearson chi-square testing was performed in order to compare ordering practices between services using Stata.¹¹

In total, we reviewed 2179 individual tests, of which 362 (16.6%) were excluded. The remaining 1817 tests involved 299 patients across 26 primary specialties. Fifty-two (2.9% of orders) were ultimately deemed equivocal. The Table illustrates the overall proportion and cost of inappropriate test ordering as well as testing characteristics of the most commonly encountered thrombotic diagnoses. The Figure illustrates the proportion of potentially inappropriate test ordering with its associated cost by test type.

Orders for Factor V Leiden, prothrombin 20210, and protein C and S activity levels were most commonly deemed inappropriate due to the test results’ failure to alter clinical management (97.3%, 99.2%, 99.4% of their inappropriate orders, respectively). Antithrombin testing (59.4%) was deemed inappropriate most commonly in the setting of acute thrombosis. The lupus anticoagulant (82.8%) was inappropriately ordered most frequently in the setting of concurrent anticoagulation.

This retrospective analysis represents one of the largest examinations of inpatient thrombophilia testing practices to date. Our results illustrate the high prevalence and significant financial impact of potentially inappropriate thrombophilia testing conducted in the inpatient setting. The data confirm that, per our defined criteria, more than 90% of inherited thrombophilia testing was potentially inappropriate while the majority of acquired thrombophilia testing was appropriate, with the exception of the lupus anticoagulant.

Even when appropriately ordered, studies suggest that positive thrombophilia screening results fail to impact outcomes in most patients with VTE. In an effort to evaluate positive results’ potential to provide a basis from which to extend the duration of anticoagulation, and therefore reduce the risk of a recurrent VTE, a case-control analysis was performed on a series of patients with a first-VTE event (Multiple Environmental and Genetic Assessment of risk factors for venous thrombosis [MEGA] study).³ In examining the odds ratio (OR) for recurrence between patients who did or did not undergo testing for Factor V Leiden, antithrombin, or protein C or S activity, the data failed to show an impact of testing on the risk of VTE recurrence (OR 1.2; confidence interval, 0.8-1.8). In fact, decision making has increasingly relied on patients’ clinical characteristics rather than thrombophilia test results to guide anticoagulation duration after incident VTEs. A 2017 study illustrated that when using a clinical decision rule (Clinical Decision Rule Validation Study to Predict Low Recurrent Risk in Patients With Unprovoked Venous Thromboembolism [REVERSE criteria]) in patients with a first, unprovoked VTE, routine thrombophilia screening added little to determining the need for prolonged anticoagulation.¹² These findings support the limited clinical utility of current test ordering practices for the prediction and management of recurrent venous thrombosis.

Regarding the acquired thrombophilias, antiphospholipid antibody testing was predominantly ordered in a justified manner, which is consistent with the notion that test results could affect clinical management, such as anticoagulation duration or choice of anticoagulant.¹³ However, the validity of lupus anticoagulant testing was limited by the frequency of patients on concurrent anticoagulation.

Financially, the cumulative cost associated with inappropriate ordering was substantial, regardless of the thrombotic event in question. Moreover, our calculated costs are derived from CMS reimbursement rates and likely underestimate the true financial impact of errant testing given that commercial laboratories frequently charge at rates several-fold higher. On a national scale, prior analyses have suggested that the annual cost of thrombophilia testing, based on typical commercial rates, ranges from $300 million to $672 million.¹⁴

Researchers in prior studies have similarly examined the frequency of inappropriate thrombophilia testing and methods to reduce it. Researchers in a 2014 study demonstrated initially high rates of inappropriate inherited thrombophilia testing, and then showed marked reductions in testing and cost savings across multiple specialties following the introduction of a flowchart on a preprinted order form.¹⁵ Our findings provide motivation to perform similar endeavors.

The proportional difference of inappropriate ordering observed between nonteaching- and teaching-medicine services indicates a potential role for educational interventions. We recently completed a series of lectures on high-value thrombophilia ordering for residents and are actively analyzing its impact on subsequent ordering practices. We are also piloting an electronic best practice advisory for thrombophilia test ordering. Though the advisory may be overridden, providers are asked to provide justification for doing so on a voluntary basis. We plan to evaluate its effect on our findings reported in this study.

We acknowledge that our exclusion criteria resulted in the omission of testing across a spectrum of nonthrombotic clinical conditions, raising the question of selection bias. Because there are no established guidelines to determine the appropriateness of testing in these scenarios, we chose to limit the analysis of errant ordering to the context of thrombotic events. Other limitations of this study include the analysis of equivocal orders as appropriate. However, because equivocal ordering represented less than 3% of all analyzed orders, including these as inappropriate would not have significantly altered our findings.

A review of thrombophilia testing practices at our institution demonstrated that inappropriate testing in the inpatient setting is a frequent phenomenon associated with a significant financial impact. This effect was more pronounced in inherited versus acquired thrombophilia testing. Testing was frequently confounded and often failed to impact patients’ short- or long-term clinical management, regardless of the result.

These findings serve as a strong impetus to reduce the burden of routine thrombophilia testing during hospital admissions. Our data demonstrate a need for institution-wide changes such as implementing best practice advisories, introducing ordering restrictions, and conducting educational interventions in order to reduce unnecessary expenditures and improve patient care.

Disclosure

The authors have nothing to disclose.

Venous thromboembolism (VTE) affects more than 1 million patients and costs the US healthcare system more than $1.5 billion annually.¹ Inherited and acquired thrombophilias have been perceived as important risk factors in assessing the risk of VTE recurrence and guiding the duration of anticoagulation.

Thrombophilias increase the risk of a first thrombotic event, but existing data have failed to demonstrate the usefulness of routine thrombophilia screening on subsequent management.^2,3 Moreover, thrombophilia testing ordered in the context of an inpatient hospitalization is limited by confounding factors, especially during an acute thrombotic event or in the setting of concurrent anticoagulation.⁴

Recognizing the costliness of routine thrombophilia testing, The American Society of Hematology introduced its Choosing Wisely campaign in 2013 in an effort to reduce test ordering in the setting of provoked VTEs with a major transient risk factor.⁵ In order to define current practice behavior at our institution, we conducted a retrospective study to determine the magnitude and financial impact of potentially inappropriate thrombophilia testing in the inpatient setting.

We performed a retrospective analysis of thrombophilia testing across all inpatient services at a large, quaternary-care academic institution over a 2-year period. Electronic medical record data containing all thrombophilia tests ordered on inpatients from June 2013 to June 2015 were obtained. This study was exempt from institutional review board approval.

Inclusion criteria included any inpatient for which thrombophilia testing occurred. Patients were excluded if testing was ordered in the absence of VTE or arterial thrombosis or if it was ordered as part of a work-up for another medical condition (see Supplementary Material).

Thrombophilia testing was defined as any of the following: inherited thrombophilias (Factor V Leiden or prothrombin 20210 gene mutations, antithrombin, or protein C or S activity levels) or acquired thrombophilias (lupus anticoagulant [Testing refers to the activated partial thromboplastin time lupus assay.], beta-2 glycoprotein 1 immunoglobulins M and G, anticardiolipin immunoglobulins M and G, dilute Russell’s viper venom time, or JAK2 V617F mutations).

Extracted data included patient age, sex, type of thrombophilia test ordered, ordering primary service, admission diagnosis, and objective confirmation of thrombotic events. The indication for test ordering was determined via medical record review of the patient’s corresponding hospitalization. Each test was evaluated in the context of the patient’s presenting history, hospital course, active medications, accompanying laboratory and radiographic studies, and consultant recommendations to arrive at a conclusion regarding both the test’s reason for ordering and whether its indication was “inappropriate,” “appropriate,” or “equivocal.” Cost data were obtained through the Centers for Medicare & Medicaid Services (CMS) Clinical Laboratory Fee Schedule for 2016 (see Supplementary Material).⁶

The criteria for defining test appropriateness were formulated by utilizing a combination of major society guidelines and literature review.^5,7-10 The criteria placed emphasis upon the ordered tests’ clinical relevance and reliability and were subsequently reviewed by a senior hematologist with specific expertise in thrombosis (see Supplementary Material).

Two internal medicine resident physician data reviewers independently evaluated the ordered tests. To ensure consistency between reviewers, a sample of identical test orders was compared for concordance, and a Cohen’s kappa coefficient was calculated. For purposes of analysis, equivocal orders were included under the appropriate category, as this study focused on the quantification of potentially inappropriate ordering practices. Pearson chi-square testing was performed in order to compare ordering practices between services using Stata.¹¹

In total, we reviewed 2179 individual tests, of which 362 (16.6%) were excluded. The remaining 1817 tests involved 299 patients across 26 primary specialties. Fifty-two (2.9% of orders) were ultimately deemed equivocal. The Table illustrates the overall proportion and cost of inappropriate test ordering as well as testing characteristics of the most commonly encountered thrombotic diagnoses. The Figure illustrates the proportion of potentially inappropriate test ordering with its associated cost by test type.

Orders for Factor V Leiden, prothrombin 20210, and protein C and S activity levels were most commonly deemed inappropriate due to the test results’ failure to alter clinical management (97.3%, 99.2%, 99.4% of their inappropriate orders, respectively). Antithrombin testing (59.4%) was deemed inappropriate most commonly in the setting of acute thrombosis. The lupus anticoagulant (82.8%) was inappropriately ordered most frequently in the setting of concurrent anticoagulation.

This retrospective analysis represents one of the largest examinations of inpatient thrombophilia testing practices to date. Our results illustrate the high prevalence and significant financial impact of potentially inappropriate thrombophilia testing conducted in the inpatient setting. The data confirm that, per our defined criteria, more than 90% of inherited thrombophilia testing was potentially inappropriate while the majority of acquired thrombophilia testing was appropriate, with the exception of the lupus anticoagulant.

Even when appropriately ordered, studies suggest that positive thrombophilia screening results fail to impact outcomes in most patients with VTE. In an effort to evaluate positive results’ potential to provide a basis from which to extend the duration of anticoagulation, and therefore reduce the risk of a recurrent VTE, a case-control analysis was performed on a series of patients with a first-VTE event (Multiple Environmental and Genetic Assessment of risk factors for venous thrombosis [MEGA] study).³ In examining the odds ratio (OR) for recurrence between patients who did or did not undergo testing for Factor V Leiden, antithrombin, or protein C or S activity, the data failed to show an impact of testing on the risk of VTE recurrence (OR 1.2; confidence interval, 0.8-1.8). In fact, decision making has increasingly relied on patients’ clinical characteristics rather than thrombophilia test results to guide anticoagulation duration after incident VTEs. A 2017 study illustrated that when using a clinical decision rule (Clinical Decision Rule Validation Study to Predict Low Recurrent Risk in Patients With Unprovoked Venous Thromboembolism [REVERSE criteria]) in patients with a first, unprovoked VTE, routine thrombophilia screening added little to determining the need for prolonged anticoagulation.¹² These findings support the limited clinical utility of current test ordering practices for the prediction and management of recurrent venous thrombosis.

Regarding the acquired thrombophilias, antiphospholipid antibody testing was predominantly ordered in a justified manner, which is consistent with the notion that test results could affect clinical management, such as anticoagulation duration or choice of anticoagulant.¹³ However, the validity of lupus anticoagulant testing was limited by the frequency of patients on concurrent anticoagulation.

Financially, the cumulative cost associated with inappropriate ordering was substantial, regardless of the thrombotic event in question. Moreover, our calculated costs are derived from CMS reimbursement rates and likely underestimate the true financial impact of errant testing given that commercial laboratories frequently charge at rates several-fold higher. On a national scale, prior analyses have suggested that the annual cost of thrombophilia testing, based on typical commercial rates, ranges from $300 million to $672 million.¹⁴

Researchers in prior studies have similarly examined the frequency of inappropriate thrombophilia testing and methods to reduce it. Researchers in a 2014 study demonstrated initially high rates of inappropriate inherited thrombophilia testing, and then showed marked reductions in testing and cost savings across multiple specialties following the introduction of a flowchart on a preprinted order form.¹⁵ Our findings provide motivation to perform similar endeavors.

The proportional difference of inappropriate ordering observed between nonteaching- and teaching-medicine services indicates a potential role for educational interventions. We recently completed a series of lectures on high-value thrombophilia ordering for residents and are actively analyzing its impact on subsequent ordering practices. We are also piloting an electronic best practice advisory for thrombophilia test ordering. Though the advisory may be overridden, providers are asked to provide justification for doing so on a voluntary basis. We plan to evaluate its effect on our findings reported in this study.

We acknowledge that our exclusion criteria resulted in the omission of testing across a spectrum of nonthrombotic clinical conditions, raising the question of selection bias. Because there are no established guidelines to determine the appropriateness of testing in these scenarios, we chose to limit the analysis of errant ordering to the context of thrombotic events. Other limitations of this study include the analysis of equivocal orders as appropriate. However, because equivocal ordering represented less than 3% of all analyzed orders, including these as inappropriate would not have significantly altered our findings.

A review of thrombophilia testing practices at our institution demonstrated that inappropriate testing in the inpatient setting is a frequent phenomenon associated with a significant financial impact. This effect was more pronounced in inherited versus acquired thrombophilia testing. Testing was frequently confounded and often failed to impact patients’ short- or long-term clinical management, regardless of the result.

These findings serve as a strong impetus to reduce the burden of routine thrombophilia testing during hospital admissions. Our data demonstrate a need for institution-wide changes such as implementing best practice advisories, introducing ordering restrictions, and conducting educational interventions in order to reduce unnecessary expenditures and improve patient care.

Disclosure

The authors have nothing to disclose.