Article Type
Article
Changed
Sun, 05/28/2017 - 22:22
Display Headline
Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice
Author(s)
Diane B. Wayne, MD
Jeffrey H. Barsuk, MD
Kevin J. O'Leary, MD
Monica J. Fudala, BA
William C. McGaghie, PhD

In a supplement to its inaugural issue, the Journal of Hospital Medicine published core competencies for hospitalists covering 3 areas: clinical conditions, systems in health care, and procedures.1 Completion of a traditional internal medicine residency may not provide hospitalists with the skills necessary to safely perform necessary procedures such as thoracentesis. A recent article reported that most internal medicine residents surveyed were uncomfortable performing common procedures, and their discomfort was higher for thoracentesis than for central line insertion, lumbar puncture, or paracentesis.2 This confirmed a previous report that family practice residents had low confidence in performing thoracenteses.3 Thoracentesis also carries the risk of the potentially life‐threatening complication of pneumothorax, which may be increased when performed by physicians‐in‐training.4

One method for improving training and assessment is the use of simulation technology. Simulation has been used to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.5, 6 Simulation has also been advocated for assessing competence in procedures including carotid angiography,7 emergency airway management,8 basic bronchoscopy,9 and advanced cardiac life support (ACLS).10, 11

Recently, we used simulation technology to help residents reach mastery learning standards for ACLS.11 Mastery learning,12 an extreme form of competency‐based education,13 implies that learners have acquired the clinical knowledge and skill measured against rigorous achievement standards. In mastery learning, educational results are equivalent, whereas educational practice time differs. To demonstrate mastery learning, we first documented a 38% improvement in skill after a simulation‐based educational intervention10 and used a multidisciplinary panel to determine mastery achievement standards for ACLS skills in 6 clinical scenarios.14 These standards were used in a study in which the amount of time needed to achieve skill mastery was allowed to vary while the skill outcomes of the residents were identical clinically.11

The present study had 4 aims. The first was to assess the baseline skill and knowledge of third‐year residents in thoracentesis. The second was to compare the thoracentesis‐related knowledge and skills of residents before and after an educational intervention. The third was to assess the correlation of medical knowledge and clinical experience with performance on a clinical skills examination after simulation training. The last was to document the feasibility of incorporating simulation‐based education into a training program.

METHODS

Objectives and Design

The study, which had a pretestposttest design without a control group,15 was of a simulation‐based, mastery learning educational intervention in thoracentesis. Primary measurements were obtained at baseline (pretest) and after the educational intervention (posttest).

Participants

Study participants were all 40 third‐year residents in the internal medicine residency program at Northwestern University's Chicago campus from January to May 2006. The Northwestern University Institutional Review Board approved the study. Participants provided informed consent before baseline assessment.

This residency program is based at Northwestern Memorial Hospital (NMH) and the Jesse Brown Veteran's Affairs Medical Center. Residents perform thoracenteses under the supervision of second‐ or third‐year residents or faculty members who are credentialed to perform the procedure. A didactic lecture on thoracentesis is part of the annual lecture series.

Procedure

The residents were kept as an intact group during the study period. The research procedure had 2 phases. First, the knowledge and clinical skills of participants at baseline were measured. Second, residents received two 2‐hour education sessions featuring didactic content and deliberate practice using a thoracentesis model. Between 4 and 6 weeks after the pretest, all residents were retested and were expected to meet or exceed a minimum passing score (MPS) on the clinical skills exam. Those who scored below the MPS engaged in more clinical skills practice until the mastery standard was reached. The amount of extra time needed to achieve the MPS was documented.

Educational Intervention

The intervention was designed to help residents acquire the knowledge and skills needed to perform a competent thoracentesis. The necessary components for mastery skill development were contained in the intervention. These included deliberate practice, rigorous skills assessment, and the provision of feedback in a supportive environment.16

The study was conducted in the Northwestern University Center for Advanced Surgical Education (N‐CASE) using the thoracentesis simulator developed by MediSim Inc. (Alton, Ontario) (http://www.medisim.ca/product.php?id=13). The model features realistic skin texture, ribs, and a fluid filled reservoir. Needles of various sizes can be inserted and fluid withdrawn. The model also accommodates the catheter/needle apparatus found in the thoracentesis kits (Tyco Healthcare, Pembroke, Bermuda) used at NMH.

Teaching and testing sessions were standardized. In teaching sessions, groups of 2‐4 residents had 4 hours to practice and ask questions, and to receive structured education and feedback from 1 of 2 hospitalist faculty instructors (J.H.B., K.J.O.). One of the 4 hours was devoted to the presentation of didactic material on indications, complications, and interpretation of results and a step‐by‐step demonstration of a thoracentesis. This presentation was videotaped to ensure standardization of content. The remaining 3 hours were devoted to clinical skills exam education, deliberate practice, and feedback.

One resident was present at each pretest and posttest session with 1 of the 2 faculty instructors who gave standardized instructions. The resident was expected to obtain a relevant history; perform a limited physical examination; review PA, lateral, and decubitus chest radiographs; perform a simulated thoracentesis; and order appropriate diagnostic tests. Written examinations were completed at the pretest and posttest sessions.

Measurements

A 25‐item checklist was developed for the thoracentesis procedure using relevant sources17, 18 and rigorous step‐by‐step procedures.19 Each skill or other action was listed in order and given equal weight. Each skill or action was scored dichotomouslyeither 0 = done correctly or 1 = done incorrectly. Checklists were reviewed for completeness and accuracy by 2 authors who frequently perform and supervise thoracenteses (J.H.B., K.J.O.), 2 authors with expertise in checklist design (D.B.W., W.C.M.), and the physician director of the medical intensive care unit at NMH. The checklist was used in a pilot clinical skills examination of 4 chief medical residents to estimate checklist reliability and face validity.

The MPS for the thoracentesis clinical skills examination was determined by 10 clinical experts using the Angoff and Hofstee standard setting methods. The panel was composed of clinical pulmonary critical care medicine faculty (n = 7) and senior fellows (n = 3). Each panel member was given instruction on standard setting and asked to use the Angoff and Hofstee methods to assign pass/fail standards. The Angoff method asks expert judges to estimate the percentage of borderline examinees who would answer each test item correctly. The Hofstee method requires judges to estimate 4 properties of an evaluation's passing scores and failure rates. The panel was asked to repeat their judgments 6 weeks later to assure stability of the MPS. Details about the use of a standard setting exercise to set an MPS for clinical skills examinations have been published previously.14, 20

Evaluation of each resident's skill was recorded on the checklist by 1 of the 2 faculty raters at the pretest and posttest sessions. A random sample of 50% of the pretest sessions was rescored by a third rater with expertise in scoring clinical skills examinations (D.B.W.) to assess interrater reliability. The rescorer was blinded to the results of the first evaluation.

A multiple choice written examination was prepared according to examination development guidelines21 using appropriate reference articles and texts.17, 18, 22 The examination was prepared by 1 author (J.H.B.) and reviewed for accuracy and clarity by 2 others (K.J.O., D.B.W.) and by the director of the medical intensive care unit at NMH. The examination had questions on knowledge and comprehension of the procedure as well as data interpretation and application. It was administered to 9 fourth‐year medical students and 5 pulmonary/critical care fellows to obtain pilot data. Results of the pilot allowed creation of a pretest and a posttest that were equivalent in content and difficulty.23 The Kuder Richardson Formula 20 (KR‐20) reliability coefficients for the 20‐item pretest and the 20‐item posttest were .72 and .74, respectively.

Demographic data were obtained from the participants including age, gender, ethnicity, medical school, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Each resident's experience performing the procedure was also collected at pretest.

Primary outcome measures were performance on the posttest written and clinical examinations. Secondary outcome measures were the total training time needed to reach the MPS (minimum = 240 minutes) and a course evaluation questionnaire.

Data Analysis

Checklist score reliability was estimated by calculating interrater reliability, the preferred method for assessments that depend on human judges,24 using the kappa () coefficient25 adjusted using the formula of Brennan and Prediger.26 Within‐group differences from pretest (baseline) to posttest (outcome) were analyzed using paired t tests. Multiple regression analysis was used to assess the correlation of posttest performance on thoracentesis skills with (1) performance on pretest thoracentesis skills, (2) medical knowledge measured by the thoracentesis pretest and posttest and USMLE Steps 1 and 2, (3) clinical experience in performing thoracentesis, (4) clinical self‐confidence about performing thoracentesis, and (5) whether additional training was needed to master the procedure.

RESULTS

All residents consented to participate and completed the entire training protocol. Table 1 presents demographic data about the residents. Most had limited experience performing and supervising thoracenteses.

Baseline Demographic Data from 40 Internal Medicine PGY3 Residents Participating in a Simulation‐Based Training Program on Thoracentesis
CharacteristicPGY‐3 Resident
Age (years), mean (SD)28.88 (1.57)
Male23 (57.5%)
Female17 (42.5%)
African American1 (2.5%)
White21 (52.5%)
Asian14 (35.0%)
Other4 (10.0%)
U.S. medical school graduate39 (97.5%)
Foreign medical school graduate1 (2.5%)
Number of thoracentesis procedures 
Performed as an intern 
0‐127.5%
2‐460.0%
512.5%
Performed as a PGY‐2 and PGY‐3 resident 
0‐125.0%
2‐455.0%
520.0%
Supervised others as a PGY‐2 and PGY‐3 resident 
0‐127.5%
2‐457.5%
515.0%

Interrater reliability for the thoracentesis checklist data was calculated at pretest. Across the 25 checklist items, the mean kappa coefficient was very high (n = .94). The MPS used as the mastery achievement standard was 80% (eg, 20 of 25 checklist items). This was the mean of the Angoff and Hofstee ratings obtained from the first judgment of the expert panel and is displayed in Figure 1.

Figure 1
Performance on thoracentesis written exam and clinical skills exam performance (MPS, minimum passing score).

No resident achieved mastery at pretest. However, 37 of the 40 medicine residents (93%) achieved mastery within the standard 4‐hour thoracentesis curriculum. The remaining 3 residents (7%) needed extra time ranging from 20 to 90 minutes to reach mastery.

Figure 1 is a graphic portrait with descriptive statistics of the residents' pretest and posttest performance on the thoracentesis written and clinical skills exams. For the written exam, the mean score rose from 57.63% to 89.75%, a statistically significant improvement of 56% from pretest to posttest (t[39] = 17.0, P < .0001). The clinical skills exam also showed a highly significant 71% pretest‐to‐posttest gain, as the mean score rose from 51.70% to 88.3% (t[39] = 15.6, P < .0001).

Results from the regression analysis indicate that neither pretest performance, medical knowledge measured by local or USMLE examinations, nor thoracentesis clinical experience was correlated with the posttest measure of thoracentesis clinical skills. However, the need for additional practice to reach the mastery standard on the posttest was a powerful negative predictor of posttest performance: b = .27 (95% CI = .46 to .09; P < .006; r2 = .28). For those residents who required extra practice time, the initial clinical skills posttest score was 20% lower than that of their peers. Although the need for extra deliberate practice was associated with relatively lower initial posttest scores, all residents ultimately met or exceeded the rigorous thoracentesis MPS.

The responses of the 40 residents on a course evaluation questionnaire were uniformly positive. Responses were recorded on a Likert scale where 1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, and 5 = strongly agree (Table 2). The data show that residents strongly agreed that practice with the medical simulator boosts clinical skills and self‐confidence, that they received useful feedback from the training sessions, and that deliberate practice using the simulator is a valuable educational experience. Residents were uncertain whether practice with the medical simulator has more educational value than patient care.

Course Evaluations Provided by All Residents (n = 40) after Simulation‐Based Educational Program
 MeanSD
Practice with the thoracentesis model boosts my skills to perform this procedure.4.30.8
I receive useful educational feedback from the training sessions.4.00.6
Practice with the thoracentesis model boosts my clinical self‐confidence.4.10.9
Practice with the thoracentesis model has more educational value than patient care experience.2.31.0
The Skills Center staff are competent.4.30.6
Practice sessions in the Skills Center are a good use of my time.3.71.0
Practice sessions using procedural models should be a required component of residency education.3.80.8
Deliberate practice using models is a valuable educational experience.4.00.9
Practice sessions using models are hard work.2.10.7
Increasing the difficulty of simulated clinical problems helps me become a better doctor.3.90.7
The controlled environment in the Skills Center helps me focus on clinical education problems.3.90.8
Practice with the thoracentesis model has helped to prepare me to perform the procedure better than clinical experience alone.4.01.0

DISCUSSION

This study demonstrates the use of a mastery learning model to develop the thoracentesis skills of internal medicine residents to a high level. Use of a thoracentesis model in a structured educational program offering an opportunity for deliberate practice with feedback produced large and consistent improvements in residents' skills. An important finding of our study is that despite having completed most of their internal medicine training, residents displayed poor knowledge and clinical skill in thoracentesis procedures at baseline. This is similar to previous studies showing that the procedural skills and knowledge of physicians at all stages of training are often poor. Examples of areas in which significant gaps were found include basic skills such as chest radiography,27 emergency airway management,8 and pulmonary auscultation.28 In contrast, after the mastery learning program, all the residents met or exceeded the MPS for the thoracentesis clinical procedure and scored much higher on the posttest written examination.

Our data also demonstrate that medical knowledge measured by procedure‐specific pretests and posttests and USMLE Steps 1 and 2 scores were not correlated with thoracentesis skill acquisition. This reinforces findings from our previous studies of ACLS skill acquisition10, 11 and supports the difference between professional and academic achievement. Pretest skill performance and clinical experience also were not correlated with posttest outcomes. However, the amount of deliberate practice needed to reach the mastery standard was a powerful negative predictor of posttest thoracentesis skill scores, replicating our research on ACLS.11 We believe that clinical experience was not correlated with posttest outcomes because residents infrequently performed thoracenteses procedures during their training.

This project demonstrates a practical model for outcomes‐based education, certification, and program accreditation. Given the need to move procedural training in internal medicine beyond such historical methods as see one, do one, teach one,29 extension of the mastery model to other invasive procedures deserves further study. At our institution we have been encouraged by the ability of simulation‐based education in ACLS to promote long‐term skill retention30 and improvement in the quality of actual patient care.31 In addition to studying these outcomes for thoracentesis, we plan to incorporate the use of ultrasound when training residents to perform procedures such as thoracentesis and central venous catheter insertion.

Given concerns about the quality of resident preparation to perform invasive procedures, programs such as this should be considered as part of the procedural certification process. As shown by our experience with several classes of residents (n = 158), use of simulation technology to reach high procedural skill levels is effective and feasible in internal medicine residency training. In addition, our residents have consistently enjoyed participating in the simulated training programs. Postcourse questionnaires show that residents agree that deliberate practice with simulation technology complements but does not replace patient care in graduate medical education.5, 10

An important question needing more research is whether performance in a simulated environment transfers to actual clinical settings. Several small studies have demonstrated such a relationship,8, 9, 31, 32 yet the transfer of simulated training to clinical practice requires further study. More work should also be done to assess long‐term retention of skills30 and to determine the utility and benefit of simulation‐based training in procedural certification and credentialing.

This study had several limitations. It was conducted in 1 training program at a single medical center. The sample size (n = 40) was relatively small. The thoracentesis model was used for both education and testing, potentially confounding the events. However, these limitations do not diminish the pronounced impact that the simulation‐based training had on the skills and knowledge of our residents.

In conclusion, this study has demonstrated the ability of deliberate practice using a thoracentesis model to produce high‐level performance of simulated thoracenteses. The project received high ratings from learners and provides reliable assessments of procedural competence. Although internists are performing fewer invasive procedures now than in years past, procedural training is still an important component of internal medicine training.29, 33 Attainment of high procedural skill levels may be especially important for residents who plan to practice hospital medicine. We believe that simulation‐based training using deliberate practice should be a key contributor to future internal medicine residency education, certification, and accreditation.

Acknowledgements

The authors thank Charles Watts, MD, and J. Larry Jameson, MD, PhD, for their support of this work. We recognize and appreciate the Northwestern University internal medicine residents for their dedication to patient care and education.

References
  1. Dressler DD,Pistoria MJ,Budnitz TL,McKean SC,Amin AN.Core competencies in hospital medicine: development and methodology.J Hosp Med.2006;1:48–56.
  2. Huang GC,Smith CC,Gordon CE, et al.Beyond the comfort zone: residents assess their comfort performing inpatient medical procedures.Am J Med.2006;119:71.e17–71.e24.
  3. Sharp LK,Wang R,Lipsky MS.Perception of competency to perform procedures and future practice intent: a national survey of family practice residents.Acad Med.2003;78:926–932.
  4. Bartter T,Mayo PD,Pratter MR,Santarelli RJ,Leeds WM,Akers SM.Lower risk and higher yield for thoracentesis when performed by experienced operators.Chest.1993;103:1873–1876.
  5. Issenberg SB,McGaghie WC,Hart IR, et al.Simulation technology for health care professional skills training and assessment.JAMA.1999;282:861–866.
  6. Boulet JR,Murray D,Kras J, et al.Reliability and validity of a simulation‐based acute care skills assessment for medical students and residents.Anesthesiology.2003;99:1270–1280.
  7. Patel AD,Gallagher AG,Nicholson WJ,Cates CU.Learning curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiography.J Am Coll Cardiol.2006;47:1796–1802.
  8. Mayo PH,Hackney JE,Mueck T,Ribaudo V,Schneider RF.Achieving house staff competence in emergency airway management: results of a teaching program using a computerized patient simulator.Crit Care Med.2004;32:2422–2427.
  9. Blum MG,Powers TW,Sundaresan S.Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy.Ann Thorac Surg.2004;78:287–291.
  10. Wayne DB,Butter J,Siddall VJ, et al.Simulation‐based training of internal medicine residents in advanced cardiac life support protocols: a randomized trial.Teach Learn Med.2005;17:210–216.
  11. Wayne DB,Butter J,Siddall VJ, et al.Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice.J Gen Intern Med.2006;21:251–256.
  12. Block JH, ed.Mastery Learning: Theory and Practice.New York:Holt, Rinehart and Winston;1971.
  13. McGaghie WC,Miller GE,Sajid A,Telder TV.Competency‐Based Curriculum Development in Medical Education. Public Health Paper No. 68.Geneva, Switzerland:World Health Organization;1978.
  14. Wayne DB,Fudala MJ,Butter J, et al.Comparison of two standard‐setting methods for advanced cardiac life support training.Acad Med.2005;80(10 Suppl):S63–S66.
  15. Shadish WR,Cook TD,Campbell DT.Experimental and Quasi‐Experimental Designs for Generalized Causal Inference.Boston:Houghton Mifflin;2002.
  16. Ericsson KA.Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains.Acad Med.2004;79(10 Suppl):S70–S81.
  17. Sokolowski JW,Burgher LW,Jones FL,Patterson JR,Selecky PA.Guidelines for thoracentesis and needle biopsy of the pleura. This position paper of the American Thoracic Society was adopted by the ATS Board of Directors June 1988.Am Rev Resp Dis.1989;140:257–258.
  18. Light RW.Clinical practice. Pleural effusion.N Engl J Med2002;346:1971–1977.
  19. Stufflebeam DL. The Checklists Development Checklist. Western Michigan University Evaluation Center, July 2000. Available at: http://www.wmich.edu/evalctr/checklists/cdc.htm. Accessed December 15,2005.
  20. Downing SM,Tekian A,Yudkowsky R.Procedures for establishing defensible absolute passing scores on performance examinations in health professions education.Teach Learn Med2006;18:50–57.
  21. Linn RL,Gronlund NE.Measurement and Assessment in Teaching.8th ed.Upper Saddle River, NJ:Prentice Hall;2000.
  22. Light RW.Pleural Diseases.4th ed.Philadelphia, PA:Lippincott Williams 2000:821–829.
  23. Downing SM.Reliability: on the reproducibility of assessment data.Med Educ.2004;38:1006–1012.
  24. Fleiss JL,Levin B,Paik MC.Statistical Methods for Rates and Proportions.3rd ed.New York:John Wiley 2003.
  25. Brennan RL,Prediger DJ.Coefficient kappa: some uses, misuses, and alternatives.Educ Psychol Meas.1981;41:687–699.
  26. Eisen LA,Berger JS,Hegde A,Schneider RF.Competency in chest radiography: a comparison of medical students, residents and fellows.J Gen Intern Med.2006;21:460–465.
  27. Mangione S,Nieman LZ.Pulmonary auscultatory skills during training in internal medicine and family practice.Am J Resp Crit Care Med.1999;159:1119–1124.
  28. Duffy FD,Holmboe ES.What procedures should internists do?Ann Intern Med.2007;146:392–3.
  29. Wayne DB,Siddall VJ,Butter J, et al.A longitudinal study of internal medicine residents' retention of advanced cardiac life support (ACLS) skills.Acad Med.2006;81(10 Suppl):S9–S12.
  30. Wayne DB,Didwania A,Feinglass J,Barsuk J,Fudala M,McGaghie WC.Simulation‐based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case‐control study.Chest.2008;[Epub ahead of print].
  31. Seymour NE,Gallagher AG,Roman SA, et al.Virtual reality training improves operating room performance: results of a randomized, double‐blinded study.Ann Surg.2002;236:458–464.
  32. Wigton RS,Alguire P.The declining number and variety of procedures done by general internists: a resurvey of members of the American College of Physicians.Ann Intern Med.2007;146:355–360.
Article PDF
268_ftp.pdf
Issue
Journal of Hospital Medicine - 3(1)
Publications
Journal of Hospital Medicine
Page Number
48-54
Legacy Keywords
thoracentesis, residency education, mastery learning, simulation‐based education
Sections
Original Research
Author(s)
Diane B. Wayne, MD
Jeffrey H. Barsuk, MD
Kevin J. O'Leary, MD
Monica J. Fudala, BA
William C. McGaghie, PhD
Author(s)
Diane B. Wayne, MD
Jeffrey H. Barsuk, MD
Kevin J. O'Leary, MD
Monica J. Fudala, BA
William C. McGaghie, PhD
Article PDF
268_ftp.pdf
Article PDF
268_ftp.pdf

In a supplement to its inaugural issue, the Journal of Hospital Medicine published core competencies for hospitalists covering 3 areas: clinical conditions, systems in health care, and procedures.1 Completion of a traditional internal medicine residency may not provide hospitalists with the skills necessary to safely perform necessary procedures such as thoracentesis. A recent article reported that most internal medicine residents surveyed were uncomfortable performing common procedures, and their discomfort was higher for thoracentesis than for central line insertion, lumbar puncture, or paracentesis.2 This confirmed a previous report that family practice residents had low confidence in performing thoracenteses.3 Thoracentesis also carries the risk of the potentially life‐threatening complication of pneumothorax, which may be increased when performed by physicians‐in‐training.4

One method for improving training and assessment is the use of simulation technology. Simulation has been used to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.5, 6 Simulation has also been advocated for assessing competence in procedures including carotid angiography,7 emergency airway management,8 basic bronchoscopy,9 and advanced cardiac life support (ACLS).10, 11

Recently, we used simulation technology to help residents reach mastery learning standards for ACLS.11 Mastery learning,12 an extreme form of competency‐based education,13 implies that learners have acquired the clinical knowledge and skill measured against rigorous achievement standards. In mastery learning, educational results are equivalent, whereas educational practice time differs. To demonstrate mastery learning, we first documented a 38% improvement in skill after a simulation‐based educational intervention10 and used a multidisciplinary panel to determine mastery achievement standards for ACLS skills in 6 clinical scenarios.14 These standards were used in a study in which the amount of time needed to achieve skill mastery was allowed to vary while the skill outcomes of the residents were identical clinically.11

The present study had 4 aims. The first was to assess the baseline skill and knowledge of third‐year residents in thoracentesis. The second was to compare the thoracentesis‐related knowledge and skills of residents before and after an educational intervention. The third was to assess the correlation of medical knowledge and clinical experience with performance on a clinical skills examination after simulation training. The last was to document the feasibility of incorporating simulation‐based education into a training program.

METHODS

Objectives and Design

The study, which had a pretestposttest design without a control group,15 was of a simulation‐based, mastery learning educational intervention in thoracentesis. Primary measurements were obtained at baseline (pretest) and after the educational intervention (posttest).

Participants

Study participants were all 40 third‐year residents in the internal medicine residency program at Northwestern University's Chicago campus from January to May 2006. The Northwestern University Institutional Review Board approved the study. Participants provided informed consent before baseline assessment.

This residency program is based at Northwestern Memorial Hospital (NMH) and the Jesse Brown Veteran's Affairs Medical Center. Residents perform thoracenteses under the supervision of second‐ or third‐year residents or faculty members who are credentialed to perform the procedure. A didactic lecture on thoracentesis is part of the annual lecture series.

Procedure

The residents were kept as an intact group during the study period. The research procedure had 2 phases. First, the knowledge and clinical skills of participants at baseline were measured. Second, residents received two 2‐hour education sessions featuring didactic content and deliberate practice using a thoracentesis model. Between 4 and 6 weeks after the pretest, all residents were retested and were expected to meet or exceed a minimum passing score (MPS) on the clinical skills exam. Those who scored below the MPS engaged in more clinical skills practice until the mastery standard was reached. The amount of extra time needed to achieve the MPS was documented.

Educational Intervention

The intervention was designed to help residents acquire the knowledge and skills needed to perform a competent thoracentesis. The necessary components for mastery skill development were contained in the intervention. These included deliberate practice, rigorous skills assessment, and the provision of feedback in a supportive environment.16

The study was conducted in the Northwestern University Center for Advanced Surgical Education (N‐CASE) using the thoracentesis simulator developed by MediSim Inc. (Alton, Ontario) (http://www.medisim.ca/product.php?id=13). The model features realistic skin texture, ribs, and a fluid filled reservoir. Needles of various sizes can be inserted and fluid withdrawn. The model also accommodates the catheter/needle apparatus found in the thoracentesis kits (Tyco Healthcare, Pembroke, Bermuda) used at NMH.

Teaching and testing sessions were standardized. In teaching sessions, groups of 2‐4 residents had 4 hours to practice and ask questions, and to receive structured education and feedback from 1 of 2 hospitalist faculty instructors (J.H.B., K.J.O.). One of the 4 hours was devoted to the presentation of didactic material on indications, complications, and interpretation of results and a step‐by‐step demonstration of a thoracentesis. This presentation was videotaped to ensure standardization of content. The remaining 3 hours were devoted to clinical skills exam education, deliberate practice, and feedback.

One resident was present at each pretest and posttest session with 1 of the 2 faculty instructors who gave standardized instructions. The resident was expected to obtain a relevant history; perform a limited physical examination; review PA, lateral, and decubitus chest radiographs; perform a simulated thoracentesis; and order appropriate diagnostic tests. Written examinations were completed at the pretest and posttest sessions.

Measurements

A 25‐item checklist was developed for the thoracentesis procedure using relevant sources17, 18 and rigorous step‐by‐step procedures.19 Each skill or other action was listed in order and given equal weight. Each skill or action was scored dichotomouslyeither 0 = done correctly or 1 = done incorrectly. Checklists were reviewed for completeness and accuracy by 2 authors who frequently perform and supervise thoracenteses (J.H.B., K.J.O.), 2 authors with expertise in checklist design (D.B.W., W.C.M.), and the physician director of the medical intensive care unit at NMH. The checklist was used in a pilot clinical skills examination of 4 chief medical residents to estimate checklist reliability and face validity.

The MPS for the thoracentesis clinical skills examination was determined by 10 clinical experts using the Angoff and Hofstee standard setting methods. The panel was composed of clinical pulmonary critical care medicine faculty (n = 7) and senior fellows (n = 3). Each panel member was given instruction on standard setting and asked to use the Angoff and Hofstee methods to assign pass/fail standards. The Angoff method asks expert judges to estimate the percentage of borderline examinees who would answer each test item correctly. The Hofstee method requires judges to estimate 4 properties of an evaluation's passing scores and failure rates. The panel was asked to repeat their judgments 6 weeks later to assure stability of the MPS. Details about the use of a standard setting exercise to set an MPS for clinical skills examinations have been published previously.14, 20

Evaluation of each resident's skill was recorded on the checklist by 1 of the 2 faculty raters at the pretest and posttest sessions. A random sample of 50% of the pretest sessions was rescored by a third rater with expertise in scoring clinical skills examinations (D.B.W.) to assess interrater reliability. The rescorer was blinded to the results of the first evaluation.

A multiple choice written examination was prepared according to examination development guidelines21 using appropriate reference articles and texts.17, 18, 22 The examination was prepared by 1 author (J.H.B.) and reviewed for accuracy and clarity by 2 others (K.J.O., D.B.W.) and by the director of the medical intensive care unit at NMH. The examination had questions on knowledge and comprehension of the procedure as well as data interpretation and application. It was administered to 9 fourth‐year medical students and 5 pulmonary/critical care fellows to obtain pilot data. Results of the pilot allowed creation of a pretest and a posttest that were equivalent in content and difficulty.23 The Kuder Richardson Formula 20 (KR‐20) reliability coefficients for the 20‐item pretest and the 20‐item posttest were .72 and .74, respectively.

Demographic data were obtained from the participants including age, gender, ethnicity, medical school, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Each resident's experience performing the procedure was also collected at pretest.

Primary outcome measures were performance on the posttest written and clinical examinations. Secondary outcome measures were the total training time needed to reach the MPS (minimum = 240 minutes) and a course evaluation questionnaire.

Data Analysis

Checklist score reliability was estimated by calculating interrater reliability, the preferred method for assessments that depend on human judges,24 using the kappa () coefficient25 adjusted using the formula of Brennan and Prediger.26 Within‐group differences from pretest (baseline) to posttest (outcome) were analyzed using paired t tests. Multiple regression analysis was used to assess the correlation of posttest performance on thoracentesis skills with (1) performance on pretest thoracentesis skills, (2) medical knowledge measured by the thoracentesis pretest and posttest and USMLE Steps 1 and 2, (3) clinical experience in performing thoracentesis, (4) clinical self‐confidence about performing thoracentesis, and (5) whether additional training was needed to master the procedure.

RESULTS

All residents consented to participate and completed the entire training protocol. Table 1 presents demographic data about the residents. Most had limited experience performing and supervising thoracenteses.

Baseline Demographic Data from 40 Internal Medicine PGY3 Residents Participating in a Simulation‐Based Training Program on Thoracentesis
CharacteristicPGY‐3 Resident
Age (years), mean (SD)28.88 (1.57)
Male23 (57.5%)
Female17 (42.5%)
African American1 (2.5%)
White21 (52.5%)
Asian14 (35.0%)
Other4 (10.0%)
U.S. medical school graduate39 (97.5%)
Foreign medical school graduate1 (2.5%)
Number of thoracentesis procedures 
Performed as an intern 
0‐127.5%
2‐460.0%
512.5%
Performed as a PGY‐2 and PGY‐3 resident 
0‐125.0%
2‐455.0%
520.0%
Supervised others as a PGY‐2 and PGY‐3 resident 
0‐127.5%
2‐457.5%
515.0%

Interrater reliability for the thoracentesis checklist data was calculated at pretest. Across the 25 checklist items, the mean kappa coefficient was very high (n = .94). The MPS used as the mastery achievement standard was 80% (eg, 20 of 25 checklist items). This was the mean of the Angoff and Hofstee ratings obtained from the first judgment of the expert panel and is displayed in Figure 1.

Figure 1
Performance on thoracentesis written exam and clinical skills exam performance (MPS, minimum passing score).

No resident achieved mastery at pretest. However, 37 of the 40 medicine residents (93%) achieved mastery within the standard 4‐hour thoracentesis curriculum. The remaining 3 residents (7%) needed extra time ranging from 20 to 90 minutes to reach mastery.

Figure 1 is a graphic portrait with descriptive statistics of the residents' pretest and posttest performance on the thoracentesis written and clinical skills exams. For the written exam, the mean score rose from 57.63% to 89.75%, a statistically significant improvement of 56% from pretest to posttest (t[39] = 17.0, P < .0001). The clinical skills exam also showed a highly significant 71% pretest‐to‐posttest gain, as the mean score rose from 51.70% to 88.3% (t[39] = 15.6, P < .0001).

Results from the regression analysis indicate that neither pretest performance, medical knowledge measured by local or USMLE examinations, nor thoracentesis clinical experience was correlated with the posttest measure of thoracentesis clinical skills. However, the need for additional practice to reach the mastery standard on the posttest was a powerful negative predictor of posttest performance: b = .27 (95% CI = .46 to .09; P < .006; r2 = .28). For those residents who required extra practice time, the initial clinical skills posttest score was 20% lower than that of their peers. Although the need for extra deliberate practice was associated with relatively lower initial posttest scores, all residents ultimately met or exceeded the rigorous thoracentesis MPS.

The responses of the 40 residents on a course evaluation questionnaire were uniformly positive. Responses were recorded on a Likert scale where 1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, and 5 = strongly agree (Table 2). The data show that residents strongly agreed that practice with the medical simulator boosts clinical skills and self‐confidence, that they received useful feedback from the training sessions, and that deliberate practice using the simulator is a valuable educational experience. Residents were uncertain whether practice with the medical simulator has more educational value than patient care.

Course Evaluations Provided by All Residents (n = 40) after Simulation‐Based Educational Program
 MeanSD
Practice with the thoracentesis model boosts my skills to perform this procedure.4.30.8
I receive useful educational feedback from the training sessions.4.00.6
Practice with the thoracentesis model boosts my clinical self‐confidence.4.10.9
Practice with the thoracentesis model has more educational value than patient care experience.2.31.0
The Skills Center staff are competent.4.30.6
Practice sessions in the Skills Center are a good use of my time.3.71.0
Practice sessions using procedural models should be a required component of residency education.3.80.8
Deliberate practice using models is a valuable educational experience.4.00.9
Practice sessions using models are hard work.2.10.7
Increasing the difficulty of simulated clinical problems helps me become a better doctor.3.90.7
The controlled environment in the Skills Center helps me focus on clinical education problems.3.90.8
Practice with the thoracentesis model has helped to prepare me to perform the procedure better than clinical experience alone.4.01.0

DISCUSSION

This study demonstrates the use of a mastery learning model to develop the thoracentesis skills of internal medicine residents to a high level. Use of a thoracentesis model in a structured educational program offering an opportunity for deliberate practice with feedback produced large and consistent improvements in residents' skills. An important finding of our study is that despite having completed most of their internal medicine training, residents displayed poor knowledge and clinical skill in thoracentesis procedures at baseline. This is similar to previous studies showing that the procedural skills and knowledge of physicians at all stages of training are often poor. Examples of areas in which significant gaps were found include basic skills such as chest radiography,27 emergency airway management,8 and pulmonary auscultation.28 In contrast, after the mastery learning program, all the residents met or exceeded the MPS for the thoracentesis clinical procedure and scored much higher on the posttest written examination.

Our data also demonstrate that medical knowledge measured by procedure‐specific pretests and posttests and USMLE Steps 1 and 2 scores were not correlated with thoracentesis skill acquisition. This reinforces findings from our previous studies of ACLS skill acquisition10, 11 and supports the difference between professional and academic achievement. Pretest skill performance and clinical experience also were not correlated with posttest outcomes. However, the amount of deliberate practice needed to reach the mastery standard was a powerful negative predictor of posttest thoracentesis skill scores, replicating our research on ACLS.11 We believe that clinical experience was not correlated with posttest outcomes because residents infrequently performed thoracenteses procedures during their training.

This project demonstrates a practical model for outcomes‐based education, certification, and program accreditation. Given the need to move procedural training in internal medicine beyond such historical methods as see one, do one, teach one,29 extension of the mastery model to other invasive procedures deserves further study. At our institution we have been encouraged by the ability of simulation‐based education in ACLS to promote long‐term skill retention30 and improvement in the quality of actual patient care.31 In addition to studying these outcomes for thoracentesis, we plan to incorporate the use of ultrasound when training residents to perform procedures such as thoracentesis and central venous catheter insertion.

Given concerns about the quality of resident preparation to perform invasive procedures, programs such as this should be considered as part of the procedural certification process. As shown by our experience with several classes of residents (n = 158), use of simulation technology to reach high procedural skill levels is effective and feasible in internal medicine residency training. In addition, our residents have consistently enjoyed participating in the simulated training programs. Postcourse questionnaires show that residents agree that deliberate practice with simulation technology complements but does not replace patient care in graduate medical education.5, 10

An important question needing more research is whether performance in a simulated environment transfers to actual clinical settings. Several small studies have demonstrated such a relationship,8, 9, 31, 32 yet the transfer of simulated training to clinical practice requires further study. More work should also be done to assess long‐term retention of skills30 and to determine the utility and benefit of simulation‐based training in procedural certification and credentialing.

This study had several limitations. It was conducted in 1 training program at a single medical center. The sample size (n = 40) was relatively small. The thoracentesis model was used for both education and testing, potentially confounding the events. However, these limitations do not diminish the pronounced impact that the simulation‐based training had on the skills and knowledge of our residents.

In conclusion, this study has demonstrated the ability of deliberate practice using a thoracentesis model to produce high‐level performance of simulated thoracenteses. The project received high ratings from learners and provides reliable assessments of procedural competence. Although internists are performing fewer invasive procedures now than in years past, procedural training is still an important component of internal medicine training.29, 33 Attainment of high procedural skill levels may be especially important for residents who plan to practice hospital medicine. We believe that simulation‐based training using deliberate practice should be a key contributor to future internal medicine residency education, certification, and accreditation.

Acknowledgements

The authors thank Charles Watts, MD, and J. Larry Jameson, MD, PhD, for their support of this work. We recognize and appreciate the Northwestern University internal medicine residents for their dedication to patient care and education.

In a supplement to its inaugural issue, the Journal of Hospital Medicine published core competencies for hospitalists covering 3 areas: clinical conditions, systems in health care, and procedures.1 Completion of a traditional internal medicine residency may not provide hospitalists with the skills necessary to safely perform necessary procedures such as thoracentesis. A recent article reported that most internal medicine residents surveyed were uncomfortable performing common procedures, and their discomfort was higher for thoracentesis than for central line insertion, lumbar puncture, or paracentesis.2 This confirmed a previous report that family practice residents had low confidence in performing thoracenteses.3 Thoracentesis also carries the risk of the potentially life‐threatening complication of pneumothorax, which may be increased when performed by physicians‐in‐training.4

One method for improving training and assessment is the use of simulation technology. Simulation has been used to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.5, 6 Simulation has also been advocated for assessing competence in procedures including carotid angiography,7 emergency airway management,8 basic bronchoscopy,9 and advanced cardiac life support (ACLS).10, 11

Recently, we used simulation technology to help residents reach mastery learning standards for ACLS.11 Mastery learning,12 an extreme form of competency‐based education,13 implies that learners have acquired the clinical knowledge and skill measured against rigorous achievement standards. In mastery learning, educational results are equivalent, whereas educational practice time differs. To demonstrate mastery learning, we first documented a 38% improvement in skill after a simulation‐based educational intervention10 and used a multidisciplinary panel to determine mastery achievement standards for ACLS skills in 6 clinical scenarios.14 These standards were used in a study in which the amount of time needed to achieve skill mastery was allowed to vary while the skill outcomes of the residents were identical clinically.11

The present study had 4 aims. The first was to assess the baseline skill and knowledge of third‐year residents in thoracentesis. The second was to compare the thoracentesis‐related knowledge and skills of residents before and after an educational intervention. The third was to assess the correlation of medical knowledge and clinical experience with performance on a clinical skills examination after simulation training. The last was to document the feasibility of incorporating simulation‐based education into a training program.

METHODS

Objectives and Design

The study, which had a pretestposttest design without a control group,15 was of a simulation‐based, mastery learning educational intervention in thoracentesis. Primary measurements were obtained at baseline (pretest) and after the educational intervention (posttest).

Participants

Study participants were all 40 third‐year residents in the internal medicine residency program at Northwestern University's Chicago campus from January to May 2006. The Northwestern University Institutional Review Board approved the study. Participants provided informed consent before baseline assessment.

This residency program is based at Northwestern Memorial Hospital (NMH) and the Jesse Brown Veteran's Affairs Medical Center. Residents perform thoracenteses under the supervision of second‐ or third‐year residents or faculty members who are credentialed to perform the procedure. A didactic lecture on thoracentesis is part of the annual lecture series.

Procedure

The residents were kept as an intact group during the study period. The research procedure had 2 phases. First, the knowledge and clinical skills of participants at baseline were measured. Second, residents received two 2‐hour education sessions featuring didactic content and deliberate practice using a thoracentesis model. Between 4 and 6 weeks after the pretest, all residents were retested and were expected to meet or exceed a minimum passing score (MPS) on the clinical skills exam. Those who scored below the MPS engaged in more clinical skills practice until the mastery standard was reached. The amount of extra time needed to achieve the MPS was documented.

Educational Intervention

The intervention was designed to help residents acquire the knowledge and skills needed to perform a competent thoracentesis. The necessary components for mastery skill development were contained in the intervention. These included deliberate practice, rigorous skills assessment, and the provision of feedback in a supportive environment.16

The study was conducted in the Northwestern University Center for Advanced Surgical Education (N‐CASE) using the thoracentesis simulator developed by MediSim Inc. (Alton, Ontario) (http://www.medisim.ca/product.php?id=13). The model features realistic skin texture, ribs, and a fluid filled reservoir. Needles of various sizes can be inserted and fluid withdrawn. The model also accommodates the catheter/needle apparatus found in the thoracentesis kits (Tyco Healthcare, Pembroke, Bermuda) used at NMH.

Teaching and testing sessions were standardized. In teaching sessions, groups of 2‐4 residents had 4 hours to practice and ask questions, and to receive structured education and feedback from 1 of 2 hospitalist faculty instructors (J.H.B., K.J.O.). One of the 4 hours was devoted to the presentation of didactic material on indications, complications, and interpretation of results and a step‐by‐step demonstration of a thoracentesis. This presentation was videotaped to ensure standardization of content. The remaining 3 hours were devoted to clinical skills exam education, deliberate practice, and feedback.

One resident was present at each pretest and posttest session with 1 of the 2 faculty instructors who gave standardized instructions. The resident was expected to obtain a relevant history; perform a limited physical examination; review PA, lateral, and decubitus chest radiographs; perform a simulated thoracentesis; and order appropriate diagnostic tests. Written examinations were completed at the pretest and posttest sessions.

Measurements

A 25‐item checklist was developed for the thoracentesis procedure using relevant sources17, 18 and rigorous step‐by‐step procedures.19 Each skill or other action was listed in order and given equal weight. Each skill or action was scored dichotomouslyeither 0 = done correctly or 1 = done incorrectly. Checklists were reviewed for completeness and accuracy by 2 authors who frequently perform and supervise thoracenteses (J.H.B., K.J.O.), 2 authors with expertise in checklist design (D.B.W., W.C.M.), and the physician director of the medical intensive care unit at NMH. The checklist was used in a pilot clinical skills examination of 4 chief medical residents to estimate checklist reliability and face validity.

The MPS for the thoracentesis clinical skills examination was determined by 10 clinical experts using the Angoff and Hofstee standard setting methods. The panel was composed of clinical pulmonary critical care medicine faculty (n = 7) and senior fellows (n = 3). Each panel member was given instruction on standard setting and asked to use the Angoff and Hofstee methods to assign pass/fail standards. The Angoff method asks expert judges to estimate the percentage of borderline examinees who would answer each test item correctly. The Hofstee method requires judges to estimate 4 properties of an evaluation's passing scores and failure rates. The panel was asked to repeat their judgments 6 weeks later to assure stability of the MPS. Details about the use of a standard setting exercise to set an MPS for clinical skills examinations have been published previously.14, 20

Evaluation of each resident's skill was recorded on the checklist by 1 of the 2 faculty raters at the pretest and posttest sessions. A random sample of 50% of the pretest sessions was rescored by a third rater with expertise in scoring clinical skills examinations (D.B.W.) to assess interrater reliability. The rescorer was blinded to the results of the first evaluation.

A multiple choice written examination was prepared according to examination development guidelines21 using appropriate reference articles and texts.17, 18, 22 The examination was prepared by 1 author (J.H.B.) and reviewed for accuracy and clarity by 2 others (K.J.O., D.B.W.) and by the director of the medical intensive care unit at NMH. The examination had questions on knowledge and comprehension of the procedure as well as data interpretation and application. It was administered to 9 fourth‐year medical students and 5 pulmonary/critical care fellows to obtain pilot data. Results of the pilot allowed creation of a pretest and a posttest that were equivalent in content and difficulty.23 The Kuder Richardson Formula 20 (KR‐20) reliability coefficients for the 20‐item pretest and the 20‐item posttest were .72 and .74, respectively.

Demographic data were obtained from the participants including age, gender, ethnicity, medical school, and scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Each resident's experience performing the procedure was also collected at pretest.

Primary outcome measures were performance on the posttest written and clinical examinations. Secondary outcome measures were the total training time needed to reach the MPS (minimum = 240 minutes) and a course evaluation questionnaire.

Data Analysis

Checklist score reliability was estimated by calculating interrater reliability, the preferred method for assessments that depend on human judges,24 using the kappa () coefficient25 adjusted using the formula of Brennan and Prediger.26 Within‐group differences from pretest (baseline) to posttest (outcome) were analyzed using paired t tests. Multiple regression analysis was used to assess the correlation of posttest performance on thoracentesis skills with (1) performance on pretest thoracentesis skills, (2) medical knowledge measured by the thoracentesis pretest and posttest and USMLE Steps 1 and 2, (3) clinical experience in performing thoracentesis, (4) clinical self‐confidence about performing thoracentesis, and (5) whether additional training was needed to master the procedure.

RESULTS

All residents consented to participate and completed the entire training protocol. Table 1 presents demographic data about the residents. Most had limited experience performing and supervising thoracenteses.

Baseline Demographic Data from 40 Internal Medicine PGY3 Residents Participating in a Simulation‐Based Training Program on Thoracentesis
CharacteristicPGY‐3 Resident
Age (years), mean (SD)28.88 (1.57)
Male23 (57.5%)
Female17 (42.5%)
African American1 (2.5%)
White21 (52.5%)
Asian14 (35.0%)
Other4 (10.0%)
U.S. medical school graduate39 (97.5%)
Foreign medical school graduate1 (2.5%)
Number of thoracentesis procedures 
Performed as an intern 
0‐127.5%
2‐460.0%
512.5%
Performed as a PGY‐2 and PGY‐3 resident 
0‐125.0%
2‐455.0%
520.0%
Supervised others as a PGY‐2 and PGY‐3 resident 
0‐127.5%
2‐457.5%
515.0%

Interrater reliability for the thoracentesis checklist data was calculated at pretest. Across the 25 checklist items, the mean kappa coefficient was very high (n = .94). The MPS used as the mastery achievement standard was 80% (eg, 20 of 25 checklist items). This was the mean of the Angoff and Hofstee ratings obtained from the first judgment of the expert panel and is displayed in Figure 1.

Figure 1
Performance on thoracentesis written exam and clinical skills exam performance (MPS, minimum passing score).

No resident achieved mastery at pretest. However, 37 of the 40 medicine residents (93%) achieved mastery within the standard 4‐hour thoracentesis curriculum. The remaining 3 residents (7%) needed extra time ranging from 20 to 90 minutes to reach mastery.

Figure 1 is a graphic portrait with descriptive statistics of the residents' pretest and posttest performance on the thoracentesis written and clinical skills exams. For the written exam, the mean score rose from 57.63% to 89.75%, a statistically significant improvement of 56% from pretest to posttest (t[39] = 17.0, P < .0001). The clinical skills exam also showed a highly significant 71% pretest‐to‐posttest gain, as the mean score rose from 51.70% to 88.3% (t[39] = 15.6, P < .0001).

Results from the regression analysis indicate that neither pretest performance, medical knowledge measured by local or USMLE examinations, nor thoracentesis clinical experience was correlated with the posttest measure of thoracentesis clinical skills. However, the need for additional practice to reach the mastery standard on the posttest was a powerful negative predictor of posttest performance: b = .27 (95% CI = .46 to .09; P < .006; r2 = .28). For those residents who required extra practice time, the initial clinical skills posttest score was 20% lower than that of their peers. Although the need for extra deliberate practice was associated with relatively lower initial posttest scores, all residents ultimately met or exceeded the rigorous thoracentesis MPS.

The responses of the 40 residents on a course evaluation questionnaire were uniformly positive. Responses were recorded on a Likert scale where 1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, and 5 = strongly agree (Table 2). The data show that residents strongly agreed that practice with the medical simulator boosts clinical skills and self‐confidence, that they received useful feedback from the training sessions, and that deliberate practice using the simulator is a valuable educational experience. Residents were uncertain whether practice with the medical simulator has more educational value than patient care.

Course Evaluations Provided by All Residents (n = 40) after Simulation‐Based Educational Program
 MeanSD
Practice with the thoracentesis model boosts my skills to perform this procedure.4.30.8
I receive useful educational feedback from the training sessions.4.00.6
Practice with the thoracentesis model boosts my clinical self‐confidence.4.10.9
Practice with the thoracentesis model has more educational value than patient care experience.2.31.0
The Skills Center staff are competent.4.30.6
Practice sessions in the Skills Center are a good use of my time.3.71.0
Practice sessions using procedural models should be a required component of residency education.3.80.8
Deliberate practice using models is a valuable educational experience.4.00.9
Practice sessions using models are hard work.2.10.7
Increasing the difficulty of simulated clinical problems helps me become a better doctor.3.90.7
The controlled environment in the Skills Center helps me focus on clinical education problems.3.90.8
Practice with the thoracentesis model has helped to prepare me to perform the procedure better than clinical experience alone.4.01.0

DISCUSSION

This study demonstrates the use of a mastery learning model to develop the thoracentesis skills of internal medicine residents to a high level. Use of a thoracentesis model in a structured educational program offering an opportunity for deliberate practice with feedback produced large and consistent improvements in residents' skills. An important finding of our study is that despite having completed most of their internal medicine training, residents displayed poor knowledge and clinical skill in thoracentesis procedures at baseline. This is similar to previous studies showing that the procedural skills and knowledge of physicians at all stages of training are often poor. Examples of areas in which significant gaps were found include basic skills such as chest radiography,27 emergency airway management,8 and pulmonary auscultation.28 In contrast, after the mastery learning program, all the residents met or exceeded the MPS for the thoracentesis clinical procedure and scored much higher on the posttest written examination.

Our data also demonstrate that medical knowledge measured by procedure‐specific pretests and posttests and USMLE Steps 1 and 2 scores were not correlated with thoracentesis skill acquisition. This reinforces findings from our previous studies of ACLS skill acquisition10, 11 and supports the difference between professional and academic achievement. Pretest skill performance and clinical experience also were not correlated with posttest outcomes. However, the amount of deliberate practice needed to reach the mastery standard was a powerful negative predictor of posttest thoracentesis skill scores, replicating our research on ACLS.11 We believe that clinical experience was not correlated with posttest outcomes because residents infrequently performed thoracenteses procedures during their training.

This project demonstrates a practical model for outcomes‐based education, certification, and program accreditation. Given the need to move procedural training in internal medicine beyond such historical methods as see one, do one, teach one,29 extension of the mastery model to other invasive procedures deserves further study. At our institution we have been encouraged by the ability of simulation‐based education in ACLS to promote long‐term skill retention30 and improvement in the quality of actual patient care.31 In addition to studying these outcomes for thoracentesis, we plan to incorporate the use of ultrasound when training residents to perform procedures such as thoracentesis and central venous catheter insertion.

Given concerns about the quality of resident preparation to perform invasive procedures, programs such as this should be considered as part of the procedural certification process. As shown by our experience with several classes of residents (n = 158), use of simulation technology to reach high procedural skill levels is effective and feasible in internal medicine residency training. In addition, our residents have consistently enjoyed participating in the simulated training programs. Postcourse questionnaires show that residents agree that deliberate practice with simulation technology complements but does not replace patient care in graduate medical education.5, 10

An important question needing more research is whether performance in a simulated environment transfers to actual clinical settings. Several small studies have demonstrated such a relationship,8, 9, 31, 32 yet the transfer of simulated training to clinical practice requires further study. More work should also be done to assess long‐term retention of skills30 and to determine the utility and benefit of simulation‐based training in procedural certification and credentialing.

This study had several limitations. It was conducted in 1 training program at a single medical center. The sample size (n = 40) was relatively small. The thoracentesis model was used for both education and testing, potentially confounding the events. However, these limitations do not diminish the pronounced impact that the simulation‐based training had on the skills and knowledge of our residents.

In conclusion, this study has demonstrated the ability of deliberate practice using a thoracentesis model to produce high‐level performance of simulated thoracenteses. The project received high ratings from learners and provides reliable assessments of procedural competence. Although internists are performing fewer invasive procedures now than in years past, procedural training is still an important component of internal medicine training.29, 33 Attainment of high procedural skill levels may be especially important for residents who plan to practice hospital medicine. We believe that simulation‐based training using deliberate practice should be a key contributor to future internal medicine residency education, certification, and accreditation.

Acknowledgements

The authors thank Charles Watts, MD, and J. Larry Jameson, MD, PhD, for their support of this work. We recognize and appreciate the Northwestern University internal medicine residents for their dedication to patient care and education.

References
  1. Dressler DD,Pistoria MJ,Budnitz TL,McKean SC,Amin AN.Core competencies in hospital medicine: development and methodology.J Hosp Med.2006;1:48–56.
  2. Huang GC,Smith CC,Gordon CE, et al.Beyond the comfort zone: residents assess their comfort performing inpatient medical procedures.Am J Med.2006;119:71.e17–71.e24.
  3. Sharp LK,Wang R,Lipsky MS.Perception of competency to perform procedures and future practice intent: a national survey of family practice residents.Acad Med.2003;78:926–932.
  4. Bartter T,Mayo PD,Pratter MR,Santarelli RJ,Leeds WM,Akers SM.Lower risk and higher yield for thoracentesis when performed by experienced operators.Chest.1993;103:1873–1876.
  5. Issenberg SB,McGaghie WC,Hart IR, et al.Simulation technology for health care professional skills training and assessment.JAMA.1999;282:861–866.
  6. Boulet JR,Murray D,Kras J, et al.Reliability and validity of a simulation‐based acute care skills assessment for medical students and residents.Anesthesiology.2003;99:1270–1280.
  7. Patel AD,Gallagher AG,Nicholson WJ,Cates CU.Learning curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiography.J Am Coll Cardiol.2006;47:1796–1802.
  8. Mayo PH,Hackney JE,Mueck T,Ribaudo V,Schneider RF.Achieving house staff competence in emergency airway management: results of a teaching program using a computerized patient simulator.Crit Care Med.2004;32:2422–2427.
  9. Blum MG,Powers TW,Sundaresan S.Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy.Ann Thorac Surg.2004;78:287–291.
  10. Wayne DB,Butter J,Siddall VJ, et al.Simulation‐based training of internal medicine residents in advanced cardiac life support protocols: a randomized trial.Teach Learn Med.2005;17:210–216.
  11. Wayne DB,Butter J,Siddall VJ, et al.Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice.J Gen Intern Med.2006;21:251–256.
  12. Block JH, ed.Mastery Learning: Theory and Practice.New York:Holt, Rinehart and Winston;1971.
  13. McGaghie WC,Miller GE,Sajid A,Telder TV.Competency‐Based Curriculum Development in Medical Education. Public Health Paper No. 68.Geneva, Switzerland:World Health Organization;1978.
  14. Wayne DB,Fudala MJ,Butter J, et al.Comparison of two standard‐setting methods for advanced cardiac life support training.Acad Med.2005;80(10 Suppl):S63–S66.
  15. Shadish WR,Cook TD,Campbell DT.Experimental and Quasi‐Experimental Designs for Generalized Causal Inference.Boston:Houghton Mifflin;2002.
  16. Ericsson KA.Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains.Acad Med.2004;79(10 Suppl):S70–S81.
  17. Sokolowski JW,Burgher LW,Jones FL,Patterson JR,Selecky PA.Guidelines for thoracentesis and needle biopsy of the pleura. This position paper of the American Thoracic Society was adopted by the ATS Board of Directors June 1988.Am Rev Resp Dis.1989;140:257–258.
  18. Light RW.Clinical practice. Pleural effusion.N Engl J Med2002;346:1971–1977.
  19. Stufflebeam DL. The Checklists Development Checklist. Western Michigan University Evaluation Center, July 2000. Available at: http://www.wmich.edu/evalctr/checklists/cdc.htm. Accessed December 15,2005.
  20. Downing SM,Tekian A,Yudkowsky R.Procedures for establishing defensible absolute passing scores on performance examinations in health professions education.Teach Learn Med2006;18:50–57.
  21. Linn RL,Gronlund NE.Measurement and Assessment in Teaching.8th ed.Upper Saddle River, NJ:Prentice Hall;2000.
  22. Light RW.Pleural Diseases.4th ed.Philadelphia, PA:Lippincott Williams 2000:821–829.
  23. Downing SM.Reliability: on the reproducibility of assessment data.Med Educ.2004;38:1006–1012.
  24. Fleiss JL,Levin B,Paik MC.Statistical Methods for Rates and Proportions.3rd ed.New York:John Wiley 2003.
  25. Brennan RL,Prediger DJ.Coefficient kappa: some uses, misuses, and alternatives.Educ Psychol Meas.1981;41:687–699.
  26. Eisen LA,Berger JS,Hegde A,Schneider RF.Competency in chest radiography: a comparison of medical students, residents and fellows.J Gen Intern Med.2006;21:460–465.
  27. Mangione S,Nieman LZ.Pulmonary auscultatory skills during training in internal medicine and family practice.Am J Resp Crit Care Med.1999;159:1119–1124.
  28. Duffy FD,Holmboe ES.What procedures should internists do?Ann Intern Med.2007;146:392–3.
  29. Wayne DB,Siddall VJ,Butter J, et al.A longitudinal study of internal medicine residents' retention of advanced cardiac life support (ACLS) skills.Acad Med.2006;81(10 Suppl):S9–S12.
  30. Wayne DB,Didwania A,Feinglass J,Barsuk J,Fudala M,McGaghie WC.Simulation‐based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case‐control study.Chest.2008;[Epub ahead of print].
  31. Seymour NE,Gallagher AG,Roman SA, et al.Virtual reality training improves operating room performance: results of a randomized, double‐blinded study.Ann Surg.2002;236:458–464.
  32. Wigton RS,Alguire P.The declining number and variety of procedures done by general internists: a resurvey of members of the American College of Physicians.Ann Intern Med.2007;146:355–360.
References
  1. Dressler DD,Pistoria MJ,Budnitz TL,McKean SC,Amin AN.Core competencies in hospital medicine: development and methodology.J Hosp Med.2006;1:48–56.
  2. Huang GC,Smith CC,Gordon CE, et al.Beyond the comfort zone: residents assess their comfort performing inpatient medical procedures.Am J Med.2006;119:71.e17–71.e24.
  3. Sharp LK,Wang R,Lipsky MS.Perception of competency to perform procedures and future practice intent: a national survey of family practice residents.Acad Med.2003;78:926–932.
  4. Bartter T,Mayo PD,Pratter MR,Santarelli RJ,Leeds WM,Akers SM.Lower risk and higher yield for thoracentesis when performed by experienced operators.Chest.1993;103:1873–1876.
  5. Issenberg SB,McGaghie WC,Hart IR, et al.Simulation technology for health care professional skills training and assessment.JAMA.1999;282:861–866.
  6. Boulet JR,Murray D,Kras J, et al.Reliability and validity of a simulation‐based acute care skills assessment for medical students and residents.Anesthesiology.2003;99:1270–1280.
  7. Patel AD,Gallagher AG,Nicholson WJ,Cates CU.Learning curves and reliability measures for virtual reality simulation in the performance assessment of carotid angiography.J Am Coll Cardiol.2006;47:1796–1802.
  8. Mayo PH,Hackney JE,Mueck T,Ribaudo V,Schneider RF.Achieving house staff competence in emergency airway management: results of a teaching program using a computerized patient simulator.Crit Care Med.2004;32:2422–2427.
  9. Blum MG,Powers TW,Sundaresan S.Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy.Ann Thorac Surg.2004;78:287–291.
  10. Wayne DB,Butter J,Siddall VJ, et al.Simulation‐based training of internal medicine residents in advanced cardiac life support protocols: a randomized trial.Teach Learn Med.2005;17:210–216.
  11. Wayne DB,Butter J,Siddall VJ, et al.Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice.J Gen Intern Med.2006;21:251–256.
  12. Block JH, ed.Mastery Learning: Theory and Practice.New York:Holt, Rinehart and Winston;1971.
  13. McGaghie WC,Miller GE,Sajid A,Telder TV.Competency‐Based Curriculum Development in Medical Education. Public Health Paper No. 68.Geneva, Switzerland:World Health Organization;1978.
  14. Wayne DB,Fudala MJ,Butter J, et al.Comparison of two standard‐setting methods for advanced cardiac life support training.Acad Med.2005;80(10 Suppl):S63–S66.
  15. Shadish WR,Cook TD,Campbell DT.Experimental and Quasi‐Experimental Designs for Generalized Causal Inference.Boston:Houghton Mifflin;2002.
  16. Ericsson KA.Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains.Acad Med.2004;79(10 Suppl):S70–S81.
  17. Sokolowski JW,Burgher LW,Jones FL,Patterson JR,Selecky PA.Guidelines for thoracentesis and needle biopsy of the pleura. This position paper of the American Thoracic Society was adopted by the ATS Board of Directors June 1988.Am Rev Resp Dis.1989;140:257–258.
  18. Light RW.Clinical practice. Pleural effusion.N Engl J Med2002;346:1971–1977.
  19. Stufflebeam DL. The Checklists Development Checklist. Western Michigan University Evaluation Center, July 2000. Available at: http://www.wmich.edu/evalctr/checklists/cdc.htm. Accessed December 15,2005.
  20. Downing SM,Tekian A,Yudkowsky R.Procedures for establishing defensible absolute passing scores on performance examinations in health professions education.Teach Learn Med2006;18:50–57.
  21. Linn RL,Gronlund NE.Measurement and Assessment in Teaching.8th ed.Upper Saddle River, NJ:Prentice Hall;2000.
  22. Light RW.Pleural Diseases.4th ed.Philadelphia, PA:Lippincott Williams 2000:821–829.
  23. Downing SM.Reliability: on the reproducibility of assessment data.Med Educ.2004;38:1006–1012.
  24. Fleiss JL,Levin B,Paik MC.Statistical Methods for Rates and Proportions.3rd ed.New York:John Wiley 2003.
  25. Brennan RL,Prediger DJ.Coefficient kappa: some uses, misuses, and alternatives.Educ Psychol Meas.1981;41:687–699.
  26. Eisen LA,Berger JS,Hegde A,Schneider RF.Competency in chest radiography: a comparison of medical students, residents and fellows.J Gen Intern Med.2006;21:460–465.
  27. Mangione S,Nieman LZ.Pulmonary auscultatory skills during training in internal medicine and family practice.Am J Resp Crit Care Med.1999;159:1119–1124.
  28. Duffy FD,Holmboe ES.What procedures should internists do?Ann Intern Med.2007;146:392–3.
  29. Wayne DB,Siddall VJ,Butter J, et al.A longitudinal study of internal medicine residents' retention of advanced cardiac life support (ACLS) skills.Acad Med.2006;81(10 Suppl):S9–S12.
  30. Wayne DB,Didwania A,Feinglass J,Barsuk J,Fudala M,McGaghie WC.Simulation‐based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case‐control study.Chest.2008;[Epub ahead of print].
  31. Seymour NE,Gallagher AG,Roman SA, et al.Virtual reality training improves operating room performance: results of a randomized, double‐blinded study.Ann Surg.2002;236:458–464.
  32. Wigton RS,Alguire P.The declining number and variety of procedures done by general internists: a resurvey of members of the American College of Physicians.Ann Intern Med.2007;146:355–360.
Issue
Journal of Hospital Medicine - 3(1)
Issue
Journal of Hospital Medicine - 3(1)
Page Number
48-54
Page Number
48-54
Publications
Journal of Hospital Medicine
Publications
Journal of Hospital Medicine
Article Type
Article
Display Headline
Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice
Display Headline
Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice
Legacy Keywords
thoracentesis, residency education, mastery learning, simulation‐based education
Legacy Keywords
thoracentesis, residency education, mastery learning, simulation‐based education
Sections
Original Research
Article Source

Copyright © 2008 Society of Hospital Medicine

Disallow All Ads
Corresponding Author
Diane B. Wayne, MD
Correspondence Location
Department of Medicine, Northwestern University Feinberg School of Medicine, 251 E. Huron St., Galter 3‐150, Chicago, IL 60611; Fax: (312) 926‐6905
Content Gating
No Gating (article Unlocked/Free)
Alternative CME
Teaser Media
Article PDF Media