Elaine R. Cohen, BA

Inpatient falls are the most common type of inpatient adverse event,1 persist as a significant problem nationally, and result in patient injury, increased length of stay, healthcare costs, and litigation.27 Inpatient falls remain a main focus of patient safety and a measure of quality in this era of healthcare reform and quality improvement.8 Inpatient fall rates per 1000 patient‐days range from 1.4 to 18.2.4, 9 The absolute percentage of inpatients that fall ranges from 1.3% to 7%.4, 5, 9, 10 Of inpatient falls, almost all data suggest that roughly one‐third result in some type of injury while 3%‐8% result in serious injury or death.9, 1113

Fall prevention interventions have largely been aimed at modifiable risk factors such as getting out of bed with bed alarms, toileting needs with bedside commodes, and reducing delirium through reorientation techniques. There have been several attempts at decreasing fall rates in hospitals surrounding a multidisciplinary, team‐based approach. Two Cochrane reviews and 2 meta‐analyses have partially examined this issue with mixed results.1417 However, none of these reviews focused on the acute care inpatient population. In fact, the majority of the data analyzed for inpatients was from rehabilitation wards and long‐term care wards. Additionally, there exists almost no data examining fall prevention with single interventions in the acute inpatient population, likely due to the belief that falls are multifactorial in etiology and require more comprehensive interventions.

The aim of this article is to determine the impact of team‐based, multidisciplinary quality improvement efforts to reduce inpatient falls in acute care inpatient hospitals and identify key features that determine their effectiveness.

METHODS

RESULTS

Fall Rates

Dykes et al22 and Williams et al24 found a statistically significant reduction in fall rate with falls reduced by 1.16 per 1000‐patient days and 1.5 per 1000‐patient days, respectively. Mitchell and Jones25 demonstrated a large fall reduction but had an extremely small sample size. Brandis21 found an extremely small reduction in fall rates and failed to report a P‐value. Krauss et al23 showed a trend towards reducing falls, and even showed a statistically significant reduction over the first 5 months of the study, but lost significance in the final 4 months. Similarly, Schwendimann et al9 saw more impressive fall reductions in the first year of the study that dissipated in the final 3 years of data collection.

Results from the meta‐analysis of the 6 studies comparing odds ratios are displayed quantitatively and as a forest plot in Figure 1. The figure shows results with 95% CI for each individual study and overall. There was no statistical evidence of heterogeneity between the studies or study designs. Although, due to the small number of studies included, there is poor power to detect true heterogeneity among studies. The magnitude of boxes shown is a relative sample size indicator. Using the random‐effects model, the summary odds ratio is 0.90 (95% CI, 0.83 to 0.99) (P = 0.02) (I² = 0%).27

Figure 1

DISCUSSION

The frequency and morbidity associated with inpatient falls is well established, based on reproduced epidemiologic data. Reducing these adverse events could reduce morbidity, mortality, and healthcare costs, and has become the focus of most hospitals quality and patient safety initiatives. The focus of this review was to examine multidisciplinary efforts to reduce falls in acute care inpatient hospitals. Despite the importance and scope of the problem, there is a paucity of research available on this topic, with a wide literature search yielding only 6 primary research studies.

Our major finding is that multidisciplinary fall prevention strategies have a statistically significant impact on fall rates with a combined OR of 0.90. While this review demonstrates a significant benefit to multidisciplinary fall prevention strategies in the acute inpatient population, the clinical impact of these efforts may be limited. Based on rates ranging from 1.7 to 9.5 falls per 1000‐patient days, multidisciplinary interventions would reduce falls by 1 to 10 falls per 10,000‐patient days using the combined OR calculated of 0.9. Using other available incidence data regarding inpatient falls,4, 9 a reasonable baseline frequency to consider would be 8 falls per 1000 patient‐days. Assuming that prevalence, the number needed to treat (NNT) to prevent a single inpatient fall is 1250 patient days. Furthermore, based on available data, only approximately one‐third of these falls result in injury and only a minor fraction of these results in serious injury.9, 1113 The magnitude of this apparent benefit in the context of fall incidence rates raises some concerns about cost‐effectiveness given the high staffing and systems needs that multidisciplinary prevention programs require. This also suggests that there are limitations when using inpatient falls as a measure of healthcare quality given the absence of high‐quality evidence demonstrating a viable solution to the problem. At present, the Center for Medicare and Medicaid services limit reimbursement for fall‐related injuries if they occur during an acute inpatient hospitalization.28

The complexity of the interventions used may help explain the limited impact. Krauss et al23 examined compliance to their interventions and found less than ideal results. They found only 36.4% of intervention floor patients had maintained a toileting schedule compared to 24.6% on control floors. Additionally, a greater proportion of patients on control floors had a physical or occupational therapy consult, and only 1.8% more patients on intervention floors had walking aids provided. These were all strategies emphasized on the intervention floors. Similarly, Schwendimann et al9 questioned their staff's adherence to protocol after fall prevention committee audits. This may help explain why a potential benefit lost statistical significance with time, based on a natural tendency towards more participation at the beginning of a new policy. Williams et al24 reported only a 64% compliance rate with fall care plan forms and 77% rate of missing information on fall care plans. A multidisciplinary fall prevention study that did not meet inclusion criteria (based on study population) yielded strongly positive results for which the authors commented mostly on changing of the hospital culture surrounding fall prevention as a key to their success.29 Adoptability of a multidisciplinary intervention will clearly impact adherence and the intervention's ultimate effectiveness.

Single intervention strategies, not analyzed in this review, are simpler to execute and adhere to. While these types of interventions may be superior, there is extremely limited data supporting or refuting patient fall benefits in the acute care inpatient population when using simple single interventions. However, some data generated partially on acute care geriatrics wards targeting patient education only showed benefit.30

Dykes et al22 was able to improve compliance rates by removing steps in the process of executing interventions with the FPTK built into the EMR. Importantly, the FPTK was compared against very similar fall prevention strategies, the difference being that patients randomized to the FPTK arm had the assessment and interventions automatically prompted on admission in the EMR. Adherence was measured through Morse Fall Scale completion rates (81% in control units versus 94% in intervention units).22 In many ways, the utility of this study was displaying a fall risk reduction by simply enhancing compliance using health information technology with automated alerts. Additionally, both arms of the study reported low fall rates compared to previously reported data, and there may have been larger benefit seen if the FPTK was compared against no fall prevention strategy. This diminishing of effect size may have been present in all studies reviewed, as usual hospital care commonly includes basic patient safety measures.

Another potential problem with the multidisciplinary fall prevention programs included in the meta‐analysis is the inability to target interventions. Each study employed a fall risk score in an attempt to focus resources on a select group of high‐risk patients. This method is problematic given that countless risk factors for inpatient falls have been identified in the literature. Factors that have been described range from clinical characteristics to laboratory tests.31 The most consistently reproducible patient‐related risks are altered mental status (including cognitive impairment and depression), altered mobility (particularly lower limb weakness), a history of falls, and toileting needs.13, 3236 Less consistency is seen with other traditional risk factors such as age, sedating medication, and length of stay.5, 13, 32, 3638 Attempting to risk‐stratify patients using simple and accurate assessment tools developed from these risk factors has proven to be very difficult. Many tools have been developed based on identified risk factors, but perform very poorly when trying to identify patients who will fall with reasonable specificity and positive predictive value.34, 3944 In fact, it has been demonstrated that using a nurse's judgment, a physician's opinion based on a patient's likelihood to wander or a simple 2‐question tool have all performed better than sophisticated risk calculators.33, 45, 46 Therefore, it is possible that interventions could benefit from including all patients, with de‐emphasis on unproven risk stratification tools.

In contrast to our findings, a modest risk reduction has been demonstrated in several primary articles and meta‐analyses in the subacute, rehabilitation, and long‐term care populations.15, 16, 4750 Additionally, a recent study has described a 63.9% risk reduction in a population that included medical, surgical, psychiatric, and rehabilitation wards.29 One important difference between these settings and the acute inpatient populations may be the amount of time and energy that can be dedicated to fall prevention and overall care planning. Another likely factor is the added challenge of preventing falls in patients with more active medical illnesses. In the acute care setting, a patient's chief complaint may not be completely addressed at the time of first mobilization and ambulation. This may be most relevant in patients who are admitted with syncope, seizure, vertigo, and dehydration.

Our study has several limitations; most notably, the available evidence is limited in quality and quantity. Furthermore, omission of unpublished data may also lead to effect bias, though this would likely be in the direction of ineffective interventions supporting a conclusion that multidisciplinary efforts have had only a small impact on fall rates. Ideally, future studies can limit confounding variables through randomization. However, it is difficult to adequately blind when studying a multidisciplinary fall intervention that depends on patient and provider participation. As a result, none of the papers reviewed met criteria for high quality. However, almost all available data examined in this review came from large sample sizes in which thoughtful interventions were used. Since an inpatient fall will not affect the majority of patients, it was crucial for these studies to recruit a large sample size to have adequate power to detect a difference in fall rates. However, each study used risk assessment tools, which are poor indicators of who will and will not fall in the hospital.34, 39, 42 This may suggest a need for improved risk assessment tools, or be further evidence to include all patients in fall prevention regardless of risk. Quantitative synthesis of multidisciplinary fall interventions has the added limitation of comparing complex, multifaceted treatments that are not perfectly uniform. It is our opinion that interventions are semi‐standardized using the grouping methods employed in Table 3.

Preventing inpatient falls remains a difficult issue to address while convincing data is lacking. Based on current evidence, multidisciplinary fall prevention efforts on acutely ill inpatients show a possible small benefit and should be explored from a cost‐effectiveness standpoint to ensure they garner appropriate investment. Many resources are required to run such teams including nursing staff, equipment, physical and occupational therapy staff, pharmacists, and specialized staff training. We are unaware of any such cost‐effectiveness data available. Effective interventions may be those that maximize compliance through health information technology, maintain staff dedication, increase staff availability, improve risk assessment, or include all patients regardless of calculated fall risk, and take the patient's chief complaint into account in the fall prevention strategy. Where resources are limited, it appears most reasonable to focus on major risk factors for inpatient falls that have independently been shown to be detrimental to outcomes, such as delirium.51 Additionally, using inpatient fall rates as a hospital quality measure may be premature, given the lack of proven efforts to lower fall rates. Multidisciplinary fall prevention efforts on acutely ill inpatients should be further studied using high‐quality, randomized trials. It remains to be seen whether these large programs are cost‐effective, or on balance clinically effective.

Inpatient falls are the most common type of inpatient adverse event,1 persist as a significant problem nationally, and result in patient injury, increased length of stay, healthcare costs, and litigation.27 Inpatient falls remain a main focus of patient safety and a measure of quality in this era of healthcare reform and quality improvement.8 Inpatient fall rates per 1000 patient‐days range from 1.4 to 18.2.4, 9 The absolute percentage of inpatients that fall ranges from 1.3% to 7%.4, 5, 9, 10 Of inpatient falls, almost all data suggest that roughly one‐third result in some type of injury while 3%‐8% result in serious injury or death.9, 1113

Fall prevention interventions have largely been aimed at modifiable risk factors such as getting out of bed with bed alarms, toileting needs with bedside commodes, and reducing delirium through reorientation techniques. There have been several attempts at decreasing fall rates in hospitals surrounding a multidisciplinary, team‐based approach. Two Cochrane reviews and 2 meta‐analyses have partially examined this issue with mixed results.1417 However, none of these reviews focused on the acute care inpatient population. In fact, the majority of the data analyzed for inpatients was from rehabilitation wards and long‐term care wards. Additionally, there exists almost no data examining fall prevention with single interventions in the acute inpatient population, likely due to the belief that falls are multifactorial in etiology and require more comprehensive interventions.

The aim of this article is to determine the impact of team‐based, multidisciplinary quality improvement efforts to reduce inpatient falls in acute care inpatient hospitals and identify key features that determine their effectiveness.

METHODS

RESULTS

Fall Rates

Dykes et al22 and Williams et al24 found a statistically significant reduction in fall rate with falls reduced by 1.16 per 1000‐patient days and 1.5 per 1000‐patient days, respectively. Mitchell and Jones25 demonstrated a large fall reduction but had an extremely small sample size. Brandis21 found an extremely small reduction in fall rates and failed to report a P‐value. Krauss et al23 showed a trend towards reducing falls, and even showed a statistically significant reduction over the first 5 months of the study, but lost significance in the final 4 months. Similarly, Schwendimann et al9 saw more impressive fall reductions in the first year of the study that dissipated in the final 3 years of data collection.

Results from the meta‐analysis of the 6 studies comparing odds ratios are displayed quantitatively and as a forest plot in Figure 1. The figure shows results with 95% CI for each individual study and overall. There was no statistical evidence of heterogeneity between the studies or study designs. Although, due to the small number of studies included, there is poor power to detect true heterogeneity among studies. The magnitude of boxes shown is a relative sample size indicator. Using the random‐effects model, the summary odds ratio is 0.90 (95% CI, 0.83 to 0.99) (P = 0.02) (I² = 0%).27

Figure 1

DISCUSSION

The frequency and morbidity associated with inpatient falls is well established, based on reproduced epidemiologic data. Reducing these adverse events could reduce morbidity, mortality, and healthcare costs, and has become the focus of most hospitals quality and patient safety initiatives. The focus of this review was to examine multidisciplinary efforts to reduce falls in acute care inpatient hospitals. Despite the importance and scope of the problem, there is a paucity of research available on this topic, with a wide literature search yielding only 6 primary research studies.

Our major finding is that multidisciplinary fall prevention strategies have a statistically significant impact on fall rates with a combined OR of 0.90. While this review demonstrates a significant benefit to multidisciplinary fall prevention strategies in the acute inpatient population, the clinical impact of these efforts may be limited. Based on rates ranging from 1.7 to 9.5 falls per 1000‐patient days, multidisciplinary interventions would reduce falls by 1 to 10 falls per 10,000‐patient days using the combined OR calculated of 0.9. Using other available incidence data regarding inpatient falls,4, 9 a reasonable baseline frequency to consider would be 8 falls per 1000 patient‐days. Assuming that prevalence, the number needed to treat (NNT) to prevent a single inpatient fall is 1250 patient days. Furthermore, based on available data, only approximately one‐third of these falls result in injury and only a minor fraction of these results in serious injury.9, 1113 The magnitude of this apparent benefit in the context of fall incidence rates raises some concerns about cost‐effectiveness given the high staffing and systems needs that multidisciplinary prevention programs require. This also suggests that there are limitations when using inpatient falls as a measure of healthcare quality given the absence of high‐quality evidence demonstrating a viable solution to the problem. At present, the Center for Medicare and Medicaid services limit reimbursement for fall‐related injuries if they occur during an acute inpatient hospitalization.28

The complexity of the interventions used may help explain the limited impact. Krauss et al23 examined compliance to their interventions and found less than ideal results. They found only 36.4% of intervention floor patients had maintained a toileting schedule compared to 24.6% on control floors. Additionally, a greater proportion of patients on control floors had a physical or occupational therapy consult, and only 1.8% more patients on intervention floors had walking aids provided. These were all strategies emphasized on the intervention floors. Similarly, Schwendimann et al9 questioned their staff's adherence to protocol after fall prevention committee audits. This may help explain why a potential benefit lost statistical significance with time, based on a natural tendency towards more participation at the beginning of a new policy. Williams et al24 reported only a 64% compliance rate with fall care plan forms and 77% rate of missing information on fall care plans. A multidisciplinary fall prevention study that did not meet inclusion criteria (based on study population) yielded strongly positive results for which the authors commented mostly on changing of the hospital culture surrounding fall prevention as a key to their success.29 Adoptability of a multidisciplinary intervention will clearly impact adherence and the intervention's ultimate effectiveness.

Single intervention strategies, not analyzed in this review, are simpler to execute and adhere to. While these types of interventions may be superior, there is extremely limited data supporting or refuting patient fall benefits in the acute care inpatient population when using simple single interventions. However, some data generated partially on acute care geriatrics wards targeting patient education only showed benefit.30

Dykes et al22 was able to improve compliance rates by removing steps in the process of executing interventions with the FPTK built into the EMR. Importantly, the FPTK was compared against very similar fall prevention strategies, the difference being that patients randomized to the FPTK arm had the assessment and interventions automatically prompted on admission in the EMR. Adherence was measured through Morse Fall Scale completion rates (81% in control units versus 94% in intervention units).22 In many ways, the utility of this study was displaying a fall risk reduction by simply enhancing compliance using health information technology with automated alerts. Additionally, both arms of the study reported low fall rates compared to previously reported data, and there may have been larger benefit seen if the FPTK was compared against no fall prevention strategy. This diminishing of effect size may have been present in all studies reviewed, as usual hospital care commonly includes basic patient safety measures.

Another potential problem with the multidisciplinary fall prevention programs included in the meta‐analysis is the inability to target interventions. Each study employed a fall risk score in an attempt to focus resources on a select group of high‐risk patients. This method is problematic given that countless risk factors for inpatient falls have been identified in the literature. Factors that have been described range from clinical characteristics to laboratory tests.31 The most consistently reproducible patient‐related risks are altered mental status (including cognitive impairment and depression), altered mobility (particularly lower limb weakness), a history of falls, and toileting needs.13, 3236 Less consistency is seen with other traditional risk factors such as age, sedating medication, and length of stay.5, 13, 32, 3638 Attempting to risk‐stratify patients using simple and accurate assessment tools developed from these risk factors has proven to be very difficult. Many tools have been developed based on identified risk factors, but perform very poorly when trying to identify patients who will fall with reasonable specificity and positive predictive value.34, 3944 In fact, it has been demonstrated that using a nurse's judgment, a physician's opinion based on a patient's likelihood to wander or a simple 2‐question tool have all performed better than sophisticated risk calculators.33, 45, 46 Therefore, it is possible that interventions could benefit from including all patients, with de‐emphasis on unproven risk stratification tools.

In contrast to our findings, a modest risk reduction has been demonstrated in several primary articles and meta‐analyses in the subacute, rehabilitation, and long‐term care populations.15, 16, 4750 Additionally, a recent study has described a 63.9% risk reduction in a population that included medical, surgical, psychiatric, and rehabilitation wards.29 One important difference between these settings and the acute inpatient populations may be the amount of time and energy that can be dedicated to fall prevention and overall care planning. Another likely factor is the added challenge of preventing falls in patients with more active medical illnesses. In the acute care setting, a patient's chief complaint may not be completely addressed at the time of first mobilization and ambulation. This may be most relevant in patients who are admitted with syncope, seizure, vertigo, and dehydration.

Our study has several limitations; most notably, the available evidence is limited in quality and quantity. Furthermore, omission of unpublished data may also lead to effect bias, though this would likely be in the direction of ineffective interventions supporting a conclusion that multidisciplinary efforts have had only a small impact on fall rates. Ideally, future studies can limit confounding variables through randomization. However, it is difficult to adequately blind when studying a multidisciplinary fall intervention that depends on patient and provider participation. As a result, none of the papers reviewed met criteria for high quality. However, almost all available data examined in this review came from large sample sizes in which thoughtful interventions were used. Since an inpatient fall will not affect the majority of patients, it was crucial for these studies to recruit a large sample size to have adequate power to detect a difference in fall rates. However, each study used risk assessment tools, which are poor indicators of who will and will not fall in the hospital.34, 39, 42 This may suggest a need for improved risk assessment tools, or be further evidence to include all patients in fall prevention regardless of risk. Quantitative synthesis of multidisciplinary fall interventions has the added limitation of comparing complex, multifaceted treatments that are not perfectly uniform. It is our opinion that interventions are semi‐standardized using the grouping methods employed in Table 3.

Preventing inpatient falls remains a difficult issue to address while convincing data is lacking. Based on current evidence, multidisciplinary fall prevention efforts on acutely ill inpatients show a possible small benefit and should be explored from a cost‐effectiveness standpoint to ensure they garner appropriate investment. Many resources are required to run such teams including nursing staff, equipment, physical and occupational therapy staff, pharmacists, and specialized staff training. We are unaware of any such cost‐effectiveness data available. Effective interventions may be those that maximize compliance through health information technology, maintain staff dedication, increase staff availability, improve risk assessment, or include all patients regardless of calculated fall risk, and take the patient's chief complaint into account in the fall prevention strategy. Where resources are limited, it appears most reasonable to focus on major risk factors for inpatient falls that have independently been shown to be detrimental to outcomes, such as delirium.51 Additionally, using inpatient fall rates as a hospital quality measure may be premature, given the lack of proven efforts to lower fall rates. Multidisciplinary fall prevention efforts on acutely ill inpatients should be further studied using high‐quality, randomized trials. It remains to be seen whether these large programs are cost‐effective, or on balance clinically effective.

Inpatient falls are the most common type of inpatient adverse event,1 persist as a significant problem nationally, and result in patient injury, increased length of stay, healthcare costs, and litigation.27 Inpatient falls remain a main focus of patient safety and a measure of quality in this era of healthcare reform and quality improvement.8 Inpatient fall rates per 1000 patient‐days range from 1.4 to 18.2.4, 9 The absolute percentage of inpatients that fall ranges from 1.3% to 7%.4, 5, 9, 10 Of inpatient falls, almost all data suggest that roughly one‐third result in some type of injury while 3%‐8% result in serious injury or death.9, 1113

Fall prevention interventions have largely been aimed at modifiable risk factors such as getting out of bed with bed alarms, toileting needs with bedside commodes, and reducing delirium through reorientation techniques. There have been several attempts at decreasing fall rates in hospitals surrounding a multidisciplinary, team‐based approach. Two Cochrane reviews and 2 meta‐analyses have partially examined this issue with mixed results.1417 However, none of these reviews focused on the acute care inpatient population. In fact, the majority of the data analyzed for inpatients was from rehabilitation wards and long‐term care wards. Additionally, there exists almost no data examining fall prevention with single interventions in the acute inpatient population, likely due to the belief that falls are multifactorial in etiology and require more comprehensive interventions.

The aim of this article is to determine the impact of team‐based, multidisciplinary quality improvement efforts to reduce inpatient falls in acute care inpatient hospitals and identify key features that determine their effectiveness.

METHODS

RESULTS

Fall Rates

Dykes et al22 and Williams et al24 found a statistically significant reduction in fall rate with falls reduced by 1.16 per 1000‐patient days and 1.5 per 1000‐patient days, respectively. Mitchell and Jones25 demonstrated a large fall reduction but had an extremely small sample size. Brandis21 found an extremely small reduction in fall rates and failed to report a P‐value. Krauss et al23 showed a trend towards reducing falls, and even showed a statistically significant reduction over the first 5 months of the study, but lost significance in the final 4 months. Similarly, Schwendimann et al9 saw more impressive fall reductions in the first year of the study that dissipated in the final 3 years of data collection.

Results from the meta‐analysis of the 6 studies comparing odds ratios are displayed quantitatively and as a forest plot in Figure 1. The figure shows results with 95% CI for each individual study and overall. There was no statistical evidence of heterogeneity between the studies or study designs. Although, due to the small number of studies included, there is poor power to detect true heterogeneity among studies. The magnitude of boxes shown is a relative sample size indicator. Using the random‐effects model, the summary odds ratio is 0.90 (95% CI, 0.83 to 0.99) (P = 0.02) (I² = 0%).27

Figure 1

DISCUSSION

The frequency and morbidity associated with inpatient falls is well established, based on reproduced epidemiologic data. Reducing these adverse events could reduce morbidity, mortality, and healthcare costs, and has become the focus of most hospitals quality and patient safety initiatives. The focus of this review was to examine multidisciplinary efforts to reduce falls in acute care inpatient hospitals. Despite the importance and scope of the problem, there is a paucity of research available on this topic, with a wide literature search yielding only 6 primary research studies.

Our major finding is that multidisciplinary fall prevention strategies have a statistically significant impact on fall rates with a combined OR of 0.90. While this review demonstrates a significant benefit to multidisciplinary fall prevention strategies in the acute inpatient population, the clinical impact of these efforts may be limited. Based on rates ranging from 1.7 to 9.5 falls per 1000‐patient days, multidisciplinary interventions would reduce falls by 1 to 10 falls per 10,000‐patient days using the combined OR calculated of 0.9. Using other available incidence data regarding inpatient falls,4, 9 a reasonable baseline frequency to consider would be 8 falls per 1000 patient‐days. Assuming that prevalence, the number needed to treat (NNT) to prevent a single inpatient fall is 1250 patient days. Furthermore, based on available data, only approximately one‐third of these falls result in injury and only a minor fraction of these results in serious injury.9, 1113 The magnitude of this apparent benefit in the context of fall incidence rates raises some concerns about cost‐effectiveness given the high staffing and systems needs that multidisciplinary prevention programs require. This also suggests that there are limitations when using inpatient falls as a measure of healthcare quality given the absence of high‐quality evidence demonstrating a viable solution to the problem. At present, the Center for Medicare and Medicaid services limit reimbursement for fall‐related injuries if they occur during an acute inpatient hospitalization.28

The complexity of the interventions used may help explain the limited impact. Krauss et al23 examined compliance to their interventions and found less than ideal results. They found only 36.4% of intervention floor patients had maintained a toileting schedule compared to 24.6% on control floors. Additionally, a greater proportion of patients on control floors had a physical or occupational therapy consult, and only 1.8% more patients on intervention floors had walking aids provided. These were all strategies emphasized on the intervention floors. Similarly, Schwendimann et al9 questioned their staff's adherence to protocol after fall prevention committee audits. This may help explain why a potential benefit lost statistical significance with time, based on a natural tendency towards more participation at the beginning of a new policy. Williams et al24 reported only a 64% compliance rate with fall care plan forms and 77% rate of missing information on fall care plans. A multidisciplinary fall prevention study that did not meet inclusion criteria (based on study population) yielded strongly positive results for which the authors commented mostly on changing of the hospital culture surrounding fall prevention as a key to their success.29 Adoptability of a multidisciplinary intervention will clearly impact adherence and the intervention's ultimate effectiveness.

Single intervention strategies, not analyzed in this review, are simpler to execute and adhere to. While these types of interventions may be superior, there is extremely limited data supporting or refuting patient fall benefits in the acute care inpatient population when using simple single interventions. However, some data generated partially on acute care geriatrics wards targeting patient education only showed benefit.30

Dykes et al22 was able to improve compliance rates by removing steps in the process of executing interventions with the FPTK built into the EMR. Importantly, the FPTK was compared against very similar fall prevention strategies, the difference being that patients randomized to the FPTK arm had the assessment and interventions automatically prompted on admission in the EMR. Adherence was measured through Morse Fall Scale completion rates (81% in control units versus 94% in intervention units).22 In many ways, the utility of this study was displaying a fall risk reduction by simply enhancing compliance using health information technology with automated alerts. Additionally, both arms of the study reported low fall rates compared to previously reported data, and there may have been larger benefit seen if the FPTK was compared against no fall prevention strategy. This diminishing of effect size may have been present in all studies reviewed, as usual hospital care commonly includes basic patient safety measures.

Another potential problem with the multidisciplinary fall prevention programs included in the meta‐analysis is the inability to target interventions. Each study employed a fall risk score in an attempt to focus resources on a select group of high‐risk patients. This method is problematic given that countless risk factors for inpatient falls have been identified in the literature. Factors that have been described range from clinical characteristics to laboratory tests.31 The most consistently reproducible patient‐related risks are altered mental status (including cognitive impairment and depression), altered mobility (particularly lower limb weakness), a history of falls, and toileting needs.13, 3236 Less consistency is seen with other traditional risk factors such as age, sedating medication, and length of stay.5, 13, 32, 3638 Attempting to risk‐stratify patients using simple and accurate assessment tools developed from these risk factors has proven to be very difficult. Many tools have been developed based on identified risk factors, but perform very poorly when trying to identify patients who will fall with reasonable specificity and positive predictive value.34, 3944 In fact, it has been demonstrated that using a nurse's judgment, a physician's opinion based on a patient's likelihood to wander or a simple 2‐question tool have all performed better than sophisticated risk calculators.33, 45, 46 Therefore, it is possible that interventions could benefit from including all patients, with de‐emphasis on unproven risk stratification tools.

In contrast to our findings, a modest risk reduction has been demonstrated in several primary articles and meta‐analyses in the subacute, rehabilitation, and long‐term care populations.15, 16, 4750 Additionally, a recent study has described a 63.9% risk reduction in a population that included medical, surgical, psychiatric, and rehabilitation wards.29 One important difference between these settings and the acute inpatient populations may be the amount of time and energy that can be dedicated to fall prevention and overall care planning. Another likely factor is the added challenge of preventing falls in patients with more active medical illnesses. In the acute care setting, a patient's chief complaint may not be completely addressed at the time of first mobilization and ambulation. This may be most relevant in patients who are admitted with syncope, seizure, vertigo, and dehydration.

Our study has several limitations; most notably, the available evidence is limited in quality and quantity. Furthermore, omission of unpublished data may also lead to effect bias, though this would likely be in the direction of ineffective interventions supporting a conclusion that multidisciplinary efforts have had only a small impact on fall rates. Ideally, future studies can limit confounding variables through randomization. However, it is difficult to adequately blind when studying a multidisciplinary fall intervention that depends on patient and provider participation. As a result, none of the papers reviewed met criteria for high quality. However, almost all available data examined in this review came from large sample sizes in which thoughtful interventions were used. Since an inpatient fall will not affect the majority of patients, it was crucial for these studies to recruit a large sample size to have adequate power to detect a difference in fall rates. However, each study used risk assessment tools, which are poor indicators of who will and will not fall in the hospital.34, 39, 42 This may suggest a need for improved risk assessment tools, or be further evidence to include all patients in fall prevention regardless of risk. Quantitative synthesis of multidisciplinary fall interventions has the added limitation of comparing complex, multifaceted treatments that are not perfectly uniform. It is our opinion that interventions are semi‐standardized using the grouping methods employed in Table 3.

Preventing inpatient falls remains a difficult issue to address while convincing data is lacking. Based on current evidence, multidisciplinary fall prevention efforts on acutely ill inpatients show a possible small benefit and should be explored from a cost‐effectiveness standpoint to ensure they garner appropriate investment. Many resources are required to run such teams including nursing staff, equipment, physical and occupational therapy staff, pharmacists, and specialized staff training. We are unaware of any such cost‐effectiveness data available. Effective interventions may be those that maximize compliance through health information technology, maintain staff dedication, increase staff availability, improve risk assessment, or include all patients regardless of calculated fall risk, and take the patient's chief complaint into account in the fall prevention strategy. Where resources are limited, it appears most reasonable to focus on major risk factors for inpatient falls that have independently been shown to be detrimental to outcomes, such as delirium.51 Additionally, using inpatient fall rates as a hospital quality measure may be premature, given the lack of proven efforts to lower fall rates. Multidisciplinary fall prevention efforts on acutely ill inpatients should be further studied using high‐quality, randomized trials. It remains to be seen whether these large programs are cost‐effective, or on balance clinically effective.

Central venous catheter (CVC) insertions are commonly performed at the bedside in medical intensive care unit (MICU) settings. Internal medicine residents are required to demonstrate knowledge regarding CVC indications, complications, and sterile technique,1 and often perform the procedure during training. Education in CVC insertion is needed because many internal medicine residents are uncomfortable performing this procedure.2 CVC insertion also carries the risk of potentially life‐threatening complications including infection, pneumothorax, arterial puncture, deep vein thrombosis, and bleeding. Education and training may also contribute to improved patient care because increased physician experience with CVC insertion reduces complication risk.3, 4 Similarly, a higher number of needle passes or attempts during CVC insertion correlates with mechanical complications such as pneumothorax or arterial punctures.48 Pneumothorax rates for internal jugular (IJ) CVCs have been reported to range from 0% to 0.2% and for subclavian (SC) CVCs from 1.5% to 3.1%.4, 5 The arterial puncture rate for IJ CVCs ranges from 5.0% to 9.4% and for SC CVCs from 3.1% to 4.9%.4, 5 Proper use of ultrasound to assist with IJ CVC insertion has been shown to decrease these mechanical complications.4, 5 However, studies of ultrasound use with SC CVC insertion have mixed results.4

Simulation‐based training has been used in medical education to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.9, 10 We previously used simulation‐based mastery learning to improve the thoracentesis and advanced cardiac life support (ACLS) skills of internal medicine residents.11, 12 Although a few small studies have linked simulation‐based interventions to improved quality of care,1319 more work is needed to show that results from a simulated environment transfer to actual patient care.

This study had 2 aims. The first was to expand our simulation‐based mastery learning to CVC insertion using a CVC simulator and ultrasound device. The second was to assess quality indicators (number of needle passes, pneumothorax, arterial punctures, and need for catheter adjustment) and resident confidence related to actual CVC insertions in the MICU before and after an educational intervention.

Materials and Methods

Design

This was a cohort study20 of IJ and SC CVC insertions by 41 second‐ and third‐year internal medicine residents rotating through the MICU in a university‐affiliated program from October 2006 to February 2007. The Northwestern University Institutional Review Board approved the study. All study participants were required to give informed consent prior to participation.

Thirteen residents rotated through the MICU during a 6‐week preintervention phase. These residents served as a traditionally trained group that did not receive CVC insertion simulator training. Simultaneously, 28 residents who rotated through the MICU later in the study period received simulation‐based training in CVC insertion and served as the simulator‐trained group (Figure 1). Demographic data were obtained from the participants including age, gender, ethnicity, year of training, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2.

Figure 1

Simulator‐trained residents underwent baseline skill assessment (pretest) using a 27‐item checklist in IJ and SC CVC insertions (see Appendix). Checklists were developed by one author (J.H.B.) using appropriate references4, 5 and a step‐by‐step process,21 and reviewed for completeness by another author with expertise in checklist development (D.B.W.). Each skill or other action was listed in order and given equal weight. A dichotomous scoring scale of 1 = done correctly and 0 = done incorrectly/not done was imposed for each item. Assessments were performed using Simulab's CentralLineMan. This model features realistic tissue with ultrasound compatibility, an arterial pulse, and self‐sealing veins and skins. Needles, dilators, and guidewires can be inserted and realistic venous and arterial pressures demonstrated (Figure 2).

Figure 2

Residents in the simulator‐trained group received two, 2‐hour education sessions featuring a lecture, ultrasound training, deliberate practice with the CVC simulator, and feedback.22 Education sessions contained standardized didactic material on CVC indications and complications, as well as a stepwise demonstration of IJ and SC CVC insertions using ultrasound and landmark techniques. These sessions were supervised by a senior hospitalist faculty member with expertise in CVC insertions (J.H.B.). Residents were expected to use the ultrasound device for all IJ CVC insertions. However, its use was optional for SC CVC insertion. After training, residents were retested (posttest) and required to meet or exceed a minimum passing score (MPS) set by an expert panel for both IJ and SC procedures.23 This 11 member expert panel provided item‐based (Angoff) and group‐based (Hofstee) judgments on the 27‐item checklists as described previously.23

Residents who did not achieve the MPS had more deliberate practice and were retested until the MPS was reached; the key feature of mastery learning.24 After completing simulation‐based mastery learning in CVC insertion, the 28 simulator‐trained residents rotated through the MICU.

Results

All eligible residents participated in the study and completed the entire protocol. There was no significant difference in age, gender, ethnicity, year of training, or USMLE Step 1 and 2 scores between the traditionally‐trained and simulator‐trained groups.

Interrater reliability measured by the mean kappa coefficient was very high (_n = 0.94) across the 27 IJ and SC checklist items. No resident met the MPS (79.1%) for CVC insertion at baseline testing. In the simulator‐trained group, 25 of 28 (89%) residents achieved SC skill mastery and 27 of 28 (96%) achieved IJ skill mastery within the standard four hour curriculum. All residents subsequently reached the MPS with less than one hour of additional practice time. A graphic portrait of the residents' pretest and posttest performance on the simulated CVC clinical skills examination with descriptive statistics is shown in Figure 3. After the educational intervention, posttest scores significantly improved (p < 0.001), to meet or exceed the MPS.

Figure 3

Traditionally trained and simulator‐trained residents independently inserted 46 CVCs during the study period. Simulator‐trained residents required significantly fewer needle passes to insert all actual CVCs in the MICU compared to traditionally trained residents: mean (M) = 1.79, standard deviation (SD) = 1.03 versus M = 2.78, SD = 1.77 (p = 0.04). As shown in Table 1, the groups did not differ in pneumothorax, arterial puncture, or mean number of CVC adjustments. In addition, the groups did not differ in use of ultrasound for IJ or SC CVC insertions. One IJ CVC was inserted without ultrasound in the traditionally‐trained group; 2 were inserted without ultrasound in the simulator‐trained group. Ultrasound was not used during any SC CVC insertions in the traditionally‐trained group and was used for 1 SC CVC insertion in the simulator‐trained group.

Simulator‐trained residents displayed more self‐confidence about their procedural skills than traditionally‐trained residents (M = 81, SD = 11 versus M = 68, SD = 20, p = 0.02). Spearman correlations showed no practical association between resident self‐confidence and performance on CVC insertion quality indicators.

Discussion

This study demonstrates the use of a mastery learning model to develop CVC insertion skills to a high achievement level among internal medicine residents. Our data support prior work showing that procedural skills that are poor at baseline can be increased significantly using simulation‐based training and deliberate practice.1118, 28 This report on CVC insertion adds to the growing body of literature showing that simulation training complements standard medical education,1119, 28 and expands the clinical application of the mastery model beyond thoracentesis and ACLS.11, 12 Use of the mastery model described in this study also has important implications for patients. In our training program, residents are required to demonstrate procedural mastery in a simulated environment before independently performing a CVC insertion on an actual patient. This is in sharp contrast to the traditional clinical model of procedural training at the bedside, and may be used in other training programs and with other invasive procedures.

The second aim of our study was to determine the impact of simulation‐based training on actual clinical practice by residents in the MICU. To our knowledge, no prior study has demonstrated that simulation‐based training in CVC insertion improves patient outcomes. We believe our results advance what is known about the impact of simulation‐based training because simulator‐trained residents in this study performed actual CVC insertions in the MICU using significantly fewer needle passes. Needle passes have been used by other investigators as a surrogate measure for reduced CVC‐associated complications because mechanical complications rise exponentially with more than two insertion attempts.47, 29 We believe this finding demonstrates transfer of skill acquired from simulation‐based training to the actual clinical environment. It is possible that ultrasound training accounts for the improvement in the simulator‐trained group. However, we do not believe that ultrasound training is entirely responsible as prior work has shown that deliberate practice using mastery learning without ultrasound significantly improved resident performance of thoracentesis11 and ACLS12, 19 procedures. We did not show a significant reduction in complications such as pneumothorax or arterial puncture. This is likely due to the small sample size and the low number of procedures and complications during the study period.

Our results also show that resident self‐confidence regarding actual CVC insertions improved after simulation training. These findings are similar to prior reports linking improved confidence among trainees after simulation‐based training in CVC insertion.29, 30 Our results did not reveal a correlation between improved self‐confidence and clinical skill acquisition. Linking improved self‐confidence to improved clinical skill is important because self‐assessment does not always correlate with performance ability.31, 32

More study is needed to evaluate the impact of simulation‐based training on the quality of CVC insertions by trainees. Mechanisms shown to decrease complications of CVC placement include use of ultrasound,4, 7, 3336 full sterile barrier technique,3739 chlorhexidine skin preparations,4042 and nurse‐physician education.43 Our simulation‐training program incorporates each of these elements. We plan to expand our simulation‐based training intervention to a larger sample size to determine its impact on mechanical and infectious complication rates linked to CVC insertion.

This study has several limitations. It was performed at a single institution over a short time period. However, demonstration of significantly fewer needle passes and improved resident self‐confidence after simulator training are important findings that warrant further study. It was impossible to blind raters during the skills assessment examination about whether the resident was performing a pretest or posttest. This was accounted for by using a second rater, who was blind to the pretest and posttest status of the examinee. The arterial puncture rate of 7% among simulator‐trained residents was higher than expected, although it remains within published ranges.4, 5 Also, a low total number of CVCs were evaluated during the study. This is likely due to strict exclusion criteria employed in order to study the impact of simulation training. For example, CVC insertions were only evaluated if they were actually performed by study residents (supervised insertions were excluded) and femoral catheters were not evaluated. We did not track clinical experience with CVC insertion by residents before the study. Residents who were simulator‐trained may have had more clinical experience with CVC insertion and this may have impacted their performance. However, residents did not differ in year of training or clinical rotations, and there is clear evidence that clinical training is not a proxy for skill acquisition.44 Finally, outcome data were measured via resident questionnaires that relied on resident recall about CVC insertion rather than observer ratings. This method was selected because observer ratings could not be standardized given the large number of clinical supervisors in the MICU over the study period. Information about needle passes and arterial puncture also may not be documented in procedural notes and could not be obtained by medical record review. We attempted to minimize recall bias by surveying residents within 24 hours of CVC placement.

In conclusion, this study demonstrates that simulation‐based training and deliberate practice in a mastery learning setting improves performance of both simulated and actual CVC insertions by internal medicine residents. Procedural training remains an important component of internal medicine training, although internists are performing fewer invasive procedures now than in years past.45, 46 Use of a mastery model of CVC insertion requires that trainees demonstrate skill in a simulated environment before independently performing this invasive procedure on patients. Further study is needed to assess clinical outcomes such as reduced CVC‐related infections and mechanical complications after simulation‐based training.

Central venous catheter (CVC) insertions are commonly performed at the bedside in medical intensive care unit (MICU) settings. Internal medicine residents are required to demonstrate knowledge regarding CVC indications, complications, and sterile technique,1 and often perform the procedure during training. Education in CVC insertion is needed because many internal medicine residents are uncomfortable performing this procedure.2 CVC insertion also carries the risk of potentially life‐threatening complications including infection, pneumothorax, arterial puncture, deep vein thrombosis, and bleeding. Education and training may also contribute to improved patient care because increased physician experience with CVC insertion reduces complication risk.3, 4 Similarly, a higher number of needle passes or attempts during CVC insertion correlates with mechanical complications such as pneumothorax or arterial punctures.48 Pneumothorax rates for internal jugular (IJ) CVCs have been reported to range from 0% to 0.2% and for subclavian (SC) CVCs from 1.5% to 3.1%.4, 5 The arterial puncture rate for IJ CVCs ranges from 5.0% to 9.4% and for SC CVCs from 3.1% to 4.9%.4, 5 Proper use of ultrasound to assist with IJ CVC insertion has been shown to decrease these mechanical complications.4, 5 However, studies of ultrasound use with SC CVC insertion have mixed results.4

Simulation‐based training has been used in medical education to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.9, 10 We previously used simulation‐based mastery learning to improve the thoracentesis and advanced cardiac life support (ACLS) skills of internal medicine residents.11, 12 Although a few small studies have linked simulation‐based interventions to improved quality of care,1319 more work is needed to show that results from a simulated environment transfer to actual patient care.

This study had 2 aims. The first was to expand our simulation‐based mastery learning to CVC insertion using a CVC simulator and ultrasound device. The second was to assess quality indicators (number of needle passes, pneumothorax, arterial punctures, and need for catheter adjustment) and resident confidence related to actual CVC insertions in the MICU before and after an educational intervention.

Materials and Methods

Design

This was a cohort study20 of IJ and SC CVC insertions by 41 second‐ and third‐year internal medicine residents rotating through the MICU in a university‐affiliated program from October 2006 to February 2007. The Northwestern University Institutional Review Board approved the study. All study participants were required to give informed consent prior to participation.

Thirteen residents rotated through the MICU during a 6‐week preintervention phase. These residents served as a traditionally trained group that did not receive CVC insertion simulator training. Simultaneously, 28 residents who rotated through the MICU later in the study period received simulation‐based training in CVC insertion and served as the simulator‐trained group (Figure 1). Demographic data were obtained from the participants including age, gender, ethnicity, year of training, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2.

Figure 1

Simulator‐trained residents underwent baseline skill assessment (pretest) using a 27‐item checklist in IJ and SC CVC insertions (see Appendix). Checklists were developed by one author (J.H.B.) using appropriate references4, 5 and a step‐by‐step process,21 and reviewed for completeness by another author with expertise in checklist development (D.B.W.). Each skill or other action was listed in order and given equal weight. A dichotomous scoring scale of 1 = done correctly and 0 = done incorrectly/not done was imposed for each item. Assessments were performed using Simulab's CentralLineMan. This model features realistic tissue with ultrasound compatibility, an arterial pulse, and self‐sealing veins and skins. Needles, dilators, and guidewires can be inserted and realistic venous and arterial pressures demonstrated (Figure 2).

Figure 2

Residents in the simulator‐trained group received two, 2‐hour education sessions featuring a lecture, ultrasound training, deliberate practice with the CVC simulator, and feedback.22 Education sessions contained standardized didactic material on CVC indications and complications, as well as a stepwise demonstration of IJ and SC CVC insertions using ultrasound and landmark techniques. These sessions were supervised by a senior hospitalist faculty member with expertise in CVC insertions (J.H.B.). Residents were expected to use the ultrasound device for all IJ CVC insertions. However, its use was optional for SC CVC insertion. After training, residents were retested (posttest) and required to meet or exceed a minimum passing score (MPS) set by an expert panel for both IJ and SC procedures.23 This 11 member expert panel provided item‐based (Angoff) and group‐based (Hofstee) judgments on the 27‐item checklists as described previously.23

Residents who did not achieve the MPS had more deliberate practice and were retested until the MPS was reached; the key feature of mastery learning.24 After completing simulation‐based mastery learning in CVC insertion, the 28 simulator‐trained residents rotated through the MICU.

Results

All eligible residents participated in the study and completed the entire protocol. There was no significant difference in age, gender, ethnicity, year of training, or USMLE Step 1 and 2 scores between the traditionally‐trained and simulator‐trained groups.

Interrater reliability measured by the mean kappa coefficient was very high (_n = 0.94) across the 27 IJ and SC checklist items. No resident met the MPS (79.1%) for CVC insertion at baseline testing. In the simulator‐trained group, 25 of 28 (89%) residents achieved SC skill mastery and 27 of 28 (96%) achieved IJ skill mastery within the standard four hour curriculum. All residents subsequently reached the MPS with less than one hour of additional practice time. A graphic portrait of the residents' pretest and posttest performance on the simulated CVC clinical skills examination with descriptive statistics is shown in Figure 3. After the educational intervention, posttest scores significantly improved (p < 0.001), to meet or exceed the MPS.

Figure 3

Traditionally trained and simulator‐trained residents independently inserted 46 CVCs during the study period. Simulator‐trained residents required significantly fewer needle passes to insert all actual CVCs in the MICU compared to traditionally trained residents: mean (M) = 1.79, standard deviation (SD) = 1.03 versus M = 2.78, SD = 1.77 (p = 0.04). As shown in Table 1, the groups did not differ in pneumothorax, arterial puncture, or mean number of CVC adjustments. In addition, the groups did not differ in use of ultrasound for IJ or SC CVC insertions. One IJ CVC was inserted without ultrasound in the traditionally‐trained group; 2 were inserted without ultrasound in the simulator‐trained group. Ultrasound was not used during any SC CVC insertions in the traditionally‐trained group and was used for 1 SC CVC insertion in the simulator‐trained group.

Simulator‐trained residents displayed more self‐confidence about their procedural skills than traditionally‐trained residents (M = 81, SD = 11 versus M = 68, SD = 20, p = 0.02). Spearman correlations showed no practical association between resident self‐confidence and performance on CVC insertion quality indicators.

Discussion

This study demonstrates the use of a mastery learning model to develop CVC insertion skills to a high achievement level among internal medicine residents. Our data support prior work showing that procedural skills that are poor at baseline can be increased significantly using simulation‐based training and deliberate practice.1118, 28 This report on CVC insertion adds to the growing body of literature showing that simulation training complements standard medical education,1119, 28 and expands the clinical application of the mastery model beyond thoracentesis and ACLS.11, 12 Use of the mastery model described in this study also has important implications for patients. In our training program, residents are required to demonstrate procedural mastery in a simulated environment before independently performing a CVC insertion on an actual patient. This is in sharp contrast to the traditional clinical model of procedural training at the bedside, and may be used in other training programs and with other invasive procedures.

The second aim of our study was to determine the impact of simulation‐based training on actual clinical practice by residents in the MICU. To our knowledge, no prior study has demonstrated that simulation‐based training in CVC insertion improves patient outcomes. We believe our results advance what is known about the impact of simulation‐based training because simulator‐trained residents in this study performed actual CVC insertions in the MICU using significantly fewer needle passes. Needle passes have been used by other investigators as a surrogate measure for reduced CVC‐associated complications because mechanical complications rise exponentially with more than two insertion attempts.47, 29 We believe this finding demonstrates transfer of skill acquired from simulation‐based training to the actual clinical environment. It is possible that ultrasound training accounts for the improvement in the simulator‐trained group. However, we do not believe that ultrasound training is entirely responsible as prior work has shown that deliberate practice using mastery learning without ultrasound significantly improved resident performance of thoracentesis11 and ACLS12, 19 procedures. We did not show a significant reduction in complications such as pneumothorax or arterial puncture. This is likely due to the small sample size and the low number of procedures and complications during the study period.

Our results also show that resident self‐confidence regarding actual CVC insertions improved after simulation training. These findings are similar to prior reports linking improved confidence among trainees after simulation‐based training in CVC insertion.29, 30 Our results did not reveal a correlation between improved self‐confidence and clinical skill acquisition. Linking improved self‐confidence to improved clinical skill is important because self‐assessment does not always correlate with performance ability.31, 32

More study is needed to evaluate the impact of simulation‐based training on the quality of CVC insertions by trainees. Mechanisms shown to decrease complications of CVC placement include use of ultrasound,4, 7, 3336 full sterile barrier technique,3739 chlorhexidine skin preparations,4042 and nurse‐physician education.43 Our simulation‐training program incorporates each of these elements. We plan to expand our simulation‐based training intervention to a larger sample size to determine its impact on mechanical and infectious complication rates linked to CVC insertion.

This study has several limitations. It was performed at a single institution over a short time period. However, demonstration of significantly fewer needle passes and improved resident self‐confidence after simulator training are important findings that warrant further study. It was impossible to blind raters during the skills assessment examination about whether the resident was performing a pretest or posttest. This was accounted for by using a second rater, who was blind to the pretest and posttest status of the examinee. The arterial puncture rate of 7% among simulator‐trained residents was higher than expected, although it remains within published ranges.4, 5 Also, a low total number of CVCs were evaluated during the study. This is likely due to strict exclusion criteria employed in order to study the impact of simulation training. For example, CVC insertions were only evaluated if they were actually performed by study residents (supervised insertions were excluded) and femoral catheters were not evaluated. We did not track clinical experience with CVC insertion by residents before the study. Residents who were simulator‐trained may have had more clinical experience with CVC insertion and this may have impacted their performance. However, residents did not differ in year of training or clinical rotations, and there is clear evidence that clinical training is not a proxy for skill acquisition.44 Finally, outcome data were measured via resident questionnaires that relied on resident recall about CVC insertion rather than observer ratings. This method was selected because observer ratings could not be standardized given the large number of clinical supervisors in the MICU over the study period. Information about needle passes and arterial puncture also may not be documented in procedural notes and could not be obtained by medical record review. We attempted to minimize recall bias by surveying residents within 24 hours of CVC placement.

In conclusion, this study demonstrates that simulation‐based training and deliberate practice in a mastery learning setting improves performance of both simulated and actual CVC insertions by internal medicine residents. Procedural training remains an important component of internal medicine training, although internists are performing fewer invasive procedures now than in years past.45, 46 Use of a mastery model of CVC insertion requires that trainees demonstrate skill in a simulated environment before independently performing this invasive procedure on patients. Further study is needed to assess clinical outcomes such as reduced CVC‐related infections and mechanical complications after simulation‐based training.

Central venous catheter (CVC) insertions are commonly performed at the bedside in medical intensive care unit (MICU) settings. Internal medicine residents are required to demonstrate knowledge regarding CVC indications, complications, and sterile technique,1 and often perform the procedure during training. Education in CVC insertion is needed because many internal medicine residents are uncomfortable performing this procedure.2 CVC insertion also carries the risk of potentially life‐threatening complications including infection, pneumothorax, arterial puncture, deep vein thrombosis, and bleeding. Education and training may also contribute to improved patient care because increased physician experience with CVC insertion reduces complication risk.3, 4 Similarly, a higher number of needle passes or attempts during CVC insertion correlates with mechanical complications such as pneumothorax or arterial punctures.48 Pneumothorax rates for internal jugular (IJ) CVCs have been reported to range from 0% to 0.2% and for subclavian (SC) CVCs from 1.5% to 3.1%.4, 5 The arterial puncture rate for IJ CVCs ranges from 5.0% to 9.4% and for SC CVCs from 3.1% to 4.9%.4, 5 Proper use of ultrasound to assist with IJ CVC insertion has been shown to decrease these mechanical complications.4, 5 However, studies of ultrasound use with SC CVC insertion have mixed results.4

Simulation‐based training has been used in medical education to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.9, 10 We previously used simulation‐based mastery learning to improve the thoracentesis and advanced cardiac life support (ACLS) skills of internal medicine residents.11, 12 Although a few small studies have linked simulation‐based interventions to improved quality of care,1319 more work is needed to show that results from a simulated environment transfer to actual patient care.

This study had 2 aims. The first was to expand our simulation‐based mastery learning to CVC insertion using a CVC simulator and ultrasound device. The second was to assess quality indicators (number of needle passes, pneumothorax, arterial punctures, and need for catheter adjustment) and resident confidence related to actual CVC insertions in the MICU before and after an educational intervention.

Materials and Methods

Design

This was a cohort study20 of IJ and SC CVC insertions by 41 second‐ and third‐year internal medicine residents rotating through the MICU in a university‐affiliated program from October 2006 to February 2007. The Northwestern University Institutional Review Board approved the study. All study participants were required to give informed consent prior to participation.

Thirteen residents rotated through the MICU during a 6‐week preintervention phase. These residents served as a traditionally trained group that did not receive CVC insertion simulator training. Simultaneously, 28 residents who rotated through the MICU later in the study period received simulation‐based training in CVC insertion and served as the simulator‐trained group (Figure 1). Demographic data were obtained from the participants including age, gender, ethnicity, year of training, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2.

Figure 1

Simulator‐trained residents underwent baseline skill assessment (pretest) using a 27‐item checklist in IJ and SC CVC insertions (see Appendix). Checklists were developed by one author (J.H.B.) using appropriate references4, 5 and a step‐by‐step process,21 and reviewed for completeness by another author with expertise in checklist development (D.B.W.). Each skill or other action was listed in order and given equal weight. A dichotomous scoring scale of 1 = done correctly and 0 = done incorrectly/not done was imposed for each item. Assessments were performed using Simulab's CentralLineMan. This model features realistic tissue with ultrasound compatibility, an arterial pulse, and self‐sealing veins and skins. Needles, dilators, and guidewires can be inserted and realistic venous and arterial pressures demonstrated (Figure 2).

Figure 2

Residents in the simulator‐trained group received two, 2‐hour education sessions featuring a lecture, ultrasound training, deliberate practice with the CVC simulator, and feedback.22 Education sessions contained standardized didactic material on CVC indications and complications, as well as a stepwise demonstration of IJ and SC CVC insertions using ultrasound and landmark techniques. These sessions were supervised by a senior hospitalist faculty member with expertise in CVC insertions (J.H.B.). Residents were expected to use the ultrasound device for all IJ CVC insertions. However, its use was optional for SC CVC insertion. After training, residents were retested (posttest) and required to meet or exceed a minimum passing score (MPS) set by an expert panel for both IJ and SC procedures.23 This 11 member expert panel provided item‐based (Angoff) and group‐based (Hofstee) judgments on the 27‐item checklists as described previously.23

Residents who did not achieve the MPS had more deliberate practice and were retested until the MPS was reached; the key feature of mastery learning.24 After completing simulation‐based mastery learning in CVC insertion, the 28 simulator‐trained residents rotated through the MICU.

Results

All eligible residents participated in the study and completed the entire protocol. There was no significant difference in age, gender, ethnicity, year of training, or USMLE Step 1 and 2 scores between the traditionally‐trained and simulator‐trained groups.

Interrater reliability measured by the mean kappa coefficient was very high (_n = 0.94) across the 27 IJ and SC checklist items. No resident met the MPS (79.1%) for CVC insertion at baseline testing. In the simulator‐trained group, 25 of 28 (89%) residents achieved SC skill mastery and 27 of 28 (96%) achieved IJ skill mastery within the standard four hour curriculum. All residents subsequently reached the MPS with less than one hour of additional practice time. A graphic portrait of the residents' pretest and posttest performance on the simulated CVC clinical skills examination with descriptive statistics is shown in Figure 3. After the educational intervention, posttest scores significantly improved (p < 0.001), to meet or exceed the MPS.

Figure 3

Traditionally trained and simulator‐trained residents independently inserted 46 CVCs during the study period. Simulator‐trained residents required significantly fewer needle passes to insert all actual CVCs in the MICU compared to traditionally trained residents: mean (M) = 1.79, standard deviation (SD) = 1.03 versus M = 2.78, SD = 1.77 (p = 0.04). As shown in Table 1, the groups did not differ in pneumothorax, arterial puncture, or mean number of CVC adjustments. In addition, the groups did not differ in use of ultrasound for IJ or SC CVC insertions. One IJ CVC was inserted without ultrasound in the traditionally‐trained group; 2 were inserted without ultrasound in the simulator‐trained group. Ultrasound was not used during any SC CVC insertions in the traditionally‐trained group and was used for 1 SC CVC insertion in the simulator‐trained group.

Simulator‐trained residents displayed more self‐confidence about their procedural skills than traditionally‐trained residents (M = 81, SD = 11 versus M = 68, SD = 20, p = 0.02). Spearman correlations showed no practical association between resident self‐confidence and performance on CVC insertion quality indicators.

Discussion

This study demonstrates the use of a mastery learning model to develop CVC insertion skills to a high achievement level among internal medicine residents. Our data support prior work showing that procedural skills that are poor at baseline can be increased significantly using simulation‐based training and deliberate practice.1118, 28 This report on CVC insertion adds to the growing body of literature showing that simulation training complements standard medical education,1119, 28 and expands the clinical application of the mastery model beyond thoracentesis and ACLS.11, 12 Use of the mastery model described in this study also has important implications for patients. In our training program, residents are required to demonstrate procedural mastery in a simulated environment before independently performing a CVC insertion on an actual patient. This is in sharp contrast to the traditional clinical model of procedural training at the bedside, and may be used in other training programs and with other invasive procedures.

The second aim of our study was to determine the impact of simulation‐based training on actual clinical practice by residents in the MICU. To our knowledge, no prior study has demonstrated that simulation‐based training in CVC insertion improves patient outcomes. We believe our results advance what is known about the impact of simulation‐based training because simulator‐trained residents in this study performed actual CVC insertions in the MICU using significantly fewer needle passes. Needle passes have been used by other investigators as a surrogate measure for reduced CVC‐associated complications because mechanical complications rise exponentially with more than two insertion attempts.47, 29 We believe this finding demonstrates transfer of skill acquired from simulation‐based training to the actual clinical environment. It is possible that ultrasound training accounts for the improvement in the simulator‐trained group. However, we do not believe that ultrasound training is entirely responsible as prior work has shown that deliberate practice using mastery learning without ultrasound significantly improved resident performance of thoracentesis11 and ACLS12, 19 procedures. We did not show a significant reduction in complications such as pneumothorax or arterial puncture. This is likely due to the small sample size and the low number of procedures and complications during the study period.

Our results also show that resident self‐confidence regarding actual CVC insertions improved after simulation training. These findings are similar to prior reports linking improved confidence among trainees after simulation‐based training in CVC insertion.29, 30 Our results did not reveal a correlation between improved self‐confidence and clinical skill acquisition. Linking improved self‐confidence to improved clinical skill is important because self‐assessment does not always correlate with performance ability.31, 32

More study is needed to evaluate the impact of simulation‐based training on the quality of CVC insertions by trainees. Mechanisms shown to decrease complications of CVC placement include use of ultrasound,4, 7, 3336 full sterile barrier technique,3739 chlorhexidine skin preparations,4042 and nurse‐physician education.43 Our simulation‐training program incorporates each of these elements. We plan to expand our simulation‐based training intervention to a larger sample size to determine its impact on mechanical and infectious complication rates linked to CVC insertion.

This study has several limitations. It was performed at a single institution over a short time period. However, demonstration of significantly fewer needle passes and improved resident self‐confidence after simulator training are important findings that warrant further study. It was impossible to blind raters during the skills assessment examination about whether the resident was performing a pretest or posttest. This was accounted for by using a second rater, who was blind to the pretest and posttest status of the examinee. The arterial puncture rate of 7% among simulator‐trained residents was higher than expected, although it remains within published ranges.4, 5 Also, a low total number of CVCs were evaluated during the study. This is likely due to strict exclusion criteria employed in order to study the impact of simulation training. For example, CVC insertions were only evaluated if they were actually performed by study residents (supervised insertions were excluded) and femoral catheters were not evaluated. We did not track clinical experience with CVC insertion by residents before the study. Residents who were simulator‐trained may have had more clinical experience with CVC insertion and this may have impacted their performance. However, residents did not differ in year of training or clinical rotations, and there is clear evidence that clinical training is not a proxy for skill acquisition.44 Finally, outcome data were measured via resident questionnaires that relied on resident recall about CVC insertion rather than observer ratings. This method was selected because observer ratings could not be standardized given the large number of clinical supervisors in the MICU over the study period. Information about needle passes and arterial puncture also may not be documented in procedural notes and could not be obtained by medical record review. We attempted to minimize recall bias by surveying residents within 24 hours of CVC placement.

In conclusion, this study demonstrates that simulation‐based training and deliberate practice in a mastery learning setting improves performance of both simulated and actual CVC insertions by internal medicine residents. Procedural training remains an important component of internal medicine training, although internists are performing fewer invasive procedures now than in years past.45, 46 Use of a mastery model of CVC insertion requires that trainees demonstrate skill in a simulated environment before independently performing this invasive procedure on patients. Further study is needed to assess clinical outcomes such as reduced CVC‐related infections and mechanical complications after simulation‐based training.