Study identifies gaps in surgical trainees’ readiness

 

PHILADELPHIA – The question of how prepared general surgery residents are to operate independently after their training is longstanding, but clear definitions of competency and readiness have been elusive. A consortium of general surgery residencies has developed a metric for assessing surgeon readiness, but what the metric revealed may be a cause for concern for the surgical profession.

Brian C. George, MD, of the University of Michigan, Ann Arbor, reported at the annual meeting of the American Surgical Association on results of a study designed to measure the autonomy and readiness for independent practice of residents at 14 general surgery programs.

“There’s increasing concern that U.S. general surgery residents are not ready to operate independently by the time they graduate,” Dr. George said. “The true scope of this problem is unknown.”

The study found that in the final 6 months of training, 96% of residents were rated competent by their observers to perform a straightforward appendectomy on their own, but only 71% were rated the same for partial colectomy, Dr. George said.

The participating general surgery attendings rated residents according to three scales (J Surg Educ. 2016;73:e118-130):
 

•“Performance” scale to measure readiness for independent practice, with competence defined as practice-ready and exceptional performance.

• “Zwisch” scale, named after Jay Zwischenberger, MD, FACS, of the University of Kentucky, to assess the amount of autonomy granted to a resident by the supervising surgical attending.

• “Complexity” scale to measure the patient-related complexity of the case at hand.

The study used a smartphone-based app, SIMPL, to collect data from September 2015 through December 2016. In evaluating performance, 437 observers provided 8,526 different ratings of 522 residents.

In a subset analysis, the study authors focused on the 132 operations the Surgical Council on Resident Education considers the core procedures of general surgery and found that 77% of fifth-year residents were rated as competent. Further restricting the analysis to residents in their final 6 months of training performing the five most commonly rated core procedures – appendectomy, cholecystectomy, ventral hernia repair, inguinal/femoral hernia repair, and partial colectomy – the researchers found that competency ranged from a high of 96% for appendectomy to a low of 71% for partial colectomy.

“If you combine all five procedures into one category, on average the residents are rated competent 84% of the time during the last 6 months of training,” Dr. George said. “But what’s really interesting – and I think this is probably the most important result for this study – is that looking at just the less-frequently rated core procedures and excluding the top five, residents are deemed competent in the last 6 months of training for 74% of those observed procedures.”

According to the Zwisch” scale to measure autonomy, residents in their last 6 months of training displayed what Dr. George termed “meaningful autonomy” in 77% of their observed operations for the core procedures. “Interestingly,” he added, “they were observed to have maximum autonomy, which is the supervision-only level, for 33% of all observed procedures.”

The researchers did an adjusted analysis to account for confounding factors such as the stringency of individual raters and patient complexity. “Using this type of analysis, the likelihood that a typical trainee rated by a typical attendee would be deemed competent by the end of training for a relatively straightforward laparoscopy appendectomy is 97%,” Dr. George said. “For a difficult laparoscopic appendectomy, by the end of training, they’re likely to be deemed competent 92% of time.” In the adjusted analysis, for a straightforward partial colectomy the raters predicted trainees to be competent 91.8% of time, but only 81.8% of the time for a complicated partial colectomy. “For less-frequently performed core procedures, there are many for which residents are likely to be deemed competent at a much lower level,” Dr. George added.

In discussing the study, Ara Darzi, MD, of St. Mary’s Hospital, London, called the methodology “unique and commendable,” and asked “Would you make the case to say we should increase the number of years of trainees from PG5 to PG6 or at least make your fellowships compulsory?”

Dr. George answered that 80% of general surgery residents already go into fellowships. “Whether it’s required or not is above my pay grade,” he said. “I hope not. Five years is already a big ask for a lot of medical students who are considering this profession. If we increase the training requirements, we’re going to have a supply problem that we will then need to address. We will be trading one problem for another.” He later added “The 20,000 hours of surgical residency should be enough to train a general surgeon to competence – its up to us to figure out how.”

The following organizations supported the research: American Board of Surgery, Association for Surgical Education, Association for Program Directors in Surgery, Massachusetts General Hospital, Northwestern University, Indiana University, and the members of the Procedural Learning and Safety Collaborative.

Neither Dr. George nor Dr. Darzi had any relevant financial disclosures.

The complete manuscript of this study and its presentation at the American Surgical Association’s 137th Annual Meeting, April 2017, in Philadelphia, Pennsylvania, is to be published in Annals of Surgery pending editorial review.

 

PHILADELPHIA – The question of how prepared general surgery residents are to operate independently after their training is longstanding, but clear definitions of competency and readiness have been elusive. A consortium of general surgery residencies has developed a metric for assessing surgeon readiness, but what the metric revealed may be a cause for concern for the surgical profession.

Brian C. George, MD, of the University of Michigan, Ann Arbor, reported at the annual meeting of the American Surgical Association on results of a study designed to measure the autonomy and readiness for independent practice of residents at 14 general surgery programs.

“There’s increasing concern that U.S. general surgery residents are not ready to operate independently by the time they graduate,” Dr. George said. “The true scope of this problem is unknown.”

The study found that in the final 6 months of training, 96% of residents were rated competent by their observers to perform a straightforward appendectomy on their own, but only 71% were rated the same for partial colectomy, Dr. George said.

The participating general surgery attendings rated residents according to three scales (J Surg Educ. 2016;73:e118-130):
 

•“Performance” scale to measure readiness for independent practice, with competence defined as practice-ready and exceptional performance.

• “Zwisch” scale, named after Jay Zwischenberger, MD, FACS, of the University of Kentucky, to assess the amount of autonomy granted to a resident by the supervising surgical attending.

• “Complexity” scale to measure the patient-related complexity of the case at hand.

The study used a smartphone-based app, SIMPL, to collect data from September 2015 through December 2016. In evaluating performance, 437 observers provided 8,526 different ratings of 522 residents.

In a subset analysis, the study authors focused on the 132 operations the Surgical Council on Resident Education considers the core procedures of general surgery and found that 77% of fifth-year residents were rated as competent. Further restricting the analysis to residents in their final 6 months of training performing the five most commonly rated core procedures – appendectomy, cholecystectomy, ventral hernia repair, inguinal/femoral hernia repair, and partial colectomy – the researchers found that competency ranged from a high of 96% for appendectomy to a low of 71% for partial colectomy.

“If you combine all five procedures into one category, on average the residents are rated competent 84% of the time during the last 6 months of training,” Dr. George said. “But what’s really interesting – and I think this is probably the most important result for this study – is that looking at just the less-frequently rated core procedures and excluding the top five, residents are deemed competent in the last 6 months of training for 74% of those observed procedures.”

According to the Zwisch” scale to measure autonomy, residents in their last 6 months of training displayed what Dr. George termed “meaningful autonomy” in 77% of their observed operations for the core procedures. “Interestingly,” he added, “they were observed to have maximum autonomy, which is the supervision-only level, for 33% of all observed procedures.”

The researchers did an adjusted analysis to account for confounding factors such as the stringency of individual raters and patient complexity. “Using this type of analysis, the likelihood that a typical trainee rated by a typical attendee would be deemed competent by the end of training for a relatively straightforward laparoscopy appendectomy is 97%,” Dr. George said. “For a difficult laparoscopic appendectomy, by the end of training, they’re likely to be deemed competent 92% of time.” In the adjusted analysis, for a straightforward partial colectomy the raters predicted trainees to be competent 91.8% of time, but only 81.8% of the time for a complicated partial colectomy. “For less-frequently performed core procedures, there are many for which residents are likely to be deemed competent at a much lower level,” Dr. George added.

In discussing the study, Ara Darzi, MD, of St. Mary’s Hospital, London, called the methodology “unique and commendable,” and asked “Would you make the case to say we should increase the number of years of trainees from PG5 to PG6 or at least make your fellowships compulsory?”

Dr. George answered that 80% of general surgery residents already go into fellowships. “Whether it’s required or not is above my pay grade,” he said. “I hope not. Five years is already a big ask for a lot of medical students who are considering this profession. If we increase the training requirements, we’re going to have a supply problem that we will then need to address. We will be trading one problem for another.” He later added “The 20,000 hours of surgical residency should be enough to train a general surgeon to competence – its up to us to figure out how.”

The following organizations supported the research: American Board of Surgery, Association for Surgical Education, Association for Program Directors in Surgery, Massachusetts General Hospital, Northwestern University, Indiana University, and the members of the Procedural Learning and Safety Collaborative.

Neither Dr. George nor Dr. Darzi had any relevant financial disclosures.

The complete manuscript of this study and its presentation at the American Surgical Association’s 137th Annual Meeting, April 2017, in Philadelphia, Pennsylvania, is to be published in Annals of Surgery pending editorial review.

 

PHILADELPHIA – The question of how prepared general surgery residents are to operate independently after their training is longstanding, but clear definitions of competency and readiness have been elusive. A consortium of general surgery residencies has developed a metric for assessing surgeon readiness, but what the metric revealed may be a cause for concern for the surgical profession.

Brian C. George, MD, of the University of Michigan, Ann Arbor, reported at the annual meeting of the American Surgical Association on results of a study designed to measure the autonomy and readiness for independent practice of residents at 14 general surgery programs.

“There’s increasing concern that U.S. general surgery residents are not ready to operate independently by the time they graduate,” Dr. George said. “The true scope of this problem is unknown.”

The study found that in the final 6 months of training, 96% of residents were rated competent by their observers to perform a straightforward appendectomy on their own, but only 71% were rated the same for partial colectomy, Dr. George said.

The participating general surgery attendings rated residents according to three scales (J Surg Educ. 2016;73:e118-130):
 

•“Performance” scale to measure readiness for independent practice, with competence defined as practice-ready and exceptional performance.

• “Zwisch” scale, named after Jay Zwischenberger, MD, FACS, of the University of Kentucky, to assess the amount of autonomy granted to a resident by the supervising surgical attending.

• “Complexity” scale to measure the patient-related complexity of the case at hand.

The study used a smartphone-based app, SIMPL, to collect data from September 2015 through December 2016. In evaluating performance, 437 observers provided 8,526 different ratings of 522 residents.

In a subset analysis, the study authors focused on the 132 operations the Surgical Council on Resident Education considers the core procedures of general surgery and found that 77% of fifth-year residents were rated as competent. Further restricting the analysis to residents in their final 6 months of training performing the five most commonly rated core procedures – appendectomy, cholecystectomy, ventral hernia repair, inguinal/femoral hernia repair, and partial colectomy – the researchers found that competency ranged from a high of 96% for appendectomy to a low of 71% for partial colectomy.

“If you combine all five procedures into one category, on average the residents are rated competent 84% of the time during the last 6 months of training,” Dr. George said. “But what’s really interesting – and I think this is probably the most important result for this study – is that looking at just the less-frequently rated core procedures and excluding the top five, residents are deemed competent in the last 6 months of training for 74% of those observed procedures.”

According to the Zwisch” scale to measure autonomy, residents in their last 6 months of training displayed what Dr. George termed “meaningful autonomy” in 77% of their observed operations for the core procedures. “Interestingly,” he added, “they were observed to have maximum autonomy, which is the supervision-only level, for 33% of all observed procedures.”

The researchers did an adjusted analysis to account for confounding factors such as the stringency of individual raters and patient complexity. “Using this type of analysis, the likelihood that a typical trainee rated by a typical attendee would be deemed competent by the end of training for a relatively straightforward laparoscopy appendectomy is 97%,” Dr. George said. “For a difficult laparoscopic appendectomy, by the end of training, they’re likely to be deemed competent 92% of time.” In the adjusted analysis, for a straightforward partial colectomy the raters predicted trainees to be competent 91.8% of time, but only 81.8% of the time for a complicated partial colectomy. “For less-frequently performed core procedures, there are many for which residents are likely to be deemed competent at a much lower level,” Dr. George added.

In discussing the study, Ara Darzi, MD, of St. Mary’s Hospital, London, called the methodology “unique and commendable,” and asked “Would you make the case to say we should increase the number of years of trainees from PG5 to PG6 or at least make your fellowships compulsory?”

Dr. George answered that 80% of general surgery residents already go into fellowships. “Whether it’s required or not is above my pay grade,” he said. “I hope not. Five years is already a big ask for a lot of medical students who are considering this profession. If we increase the training requirements, we’re going to have a supply problem that we will then need to address. We will be trading one problem for another.” He later added “The 20,000 hours of surgical residency should be enough to train a general surgeon to competence – its up to us to figure out how.”

The following organizations supported the research: American Board of Surgery, Association for Surgical Education, Association for Program Directors in Surgery, Massachusetts General Hospital, Northwestern University, Indiana University, and the members of the Procedural Learning and Safety Collaborative.

Neither Dr. George nor Dr. Darzi had any relevant financial disclosures.

The complete manuscript of this study and its presentation at the American Surgical Association’s 137th Annual Meeting, April 2017, in Philadelphia, Pennsylvania, is to be published in Annals of Surgery pending editorial review.