USMLE Step 1 Changes: Dermatology Program Director Perspectives and Implications

To the Editor:

With a trend toward increasing pass/fail medical school curricula, residency program directors (PDs) have relied on the US Medical Licensing Examination (USMLE) Step 1 as an objective measurement of applicant achievement, which is particularly true in competitive subspecialties such as dermatology, plastic surgery, orthopedic surgery, ophthalmology, and neurosurgery, in which reported Step 1 scores are consistently the highest among matched applicants.¹ Program directors in dermatology have indicated that Step 1 scores are a priority when considering an applicant.² However, among PDs, the general perception of plans to change Step 1 scores to pass/fail has largely been negative.³ Although the impact of this change on the dermatology residency selection process remains unknown, we undertook a study to determine dermatology PDs’ perspectives on the scoring change and discuss its potential implications among all competitive specialties.

A 19-question survey was designed that assessed PD demographics and opinions of the changes and potential implications of the Step 1 scoring change (eTable). A list of current US dermatology PDs at osteopathic and allopathic programs was obtained through the 2019-2020 Accreditation Council for Graduate Medical Education list of accredited programs. Surveys were piloted at our institution to assess for internal validity and misleading questions, and then were distributed electronically through REDCap software (https://www.project-redcap.org/). All responses were kept anonymous. Institutional review board approval was obtained. Variables were assessed with means, proportions, and CIs. Results were deemed statistically significant with nonoverlapping 99% CIs (P<.01).

Of 139 surveys, 57 (41.0%) were completed. Most PDs (54.4% [31/57]) were women. The average years of service as a PD was 8.5 years. Most PDs (61.4% [35/57]) disagreed with the scoring change; 77.2% (44/57) of PDs noted that it would make it difficult to objectively assess candidates. Program directors indicated that this change would increase the emphasis they place on USMLE Step 2 Clinical Knowledge (CK) scores (86.0% [49/57]); 78.2% (43/55) reported that they would start requiring Step 2 CK results with submitted applications.

Meanwhile, 73.7% (42/57) of PDs disagreed that Step 2 CK should be changed to pass/fail. Most PDs (50.9% [29/57]) thought that binary Step 1 scoring would increase the importance of medical school reputation in application decisions. The percentage of PDs who were neutral (eTable) on whether pass/fail scoring would place international graduates at a disadvantage was 52.6% (30/57), decrease socioeconomic disparities in the application process was 46.4% (26/56), and improve student well-being was 38.2% (21/55).

Results of our survey indicate generally negative perceptions by dermatology PDs to pass/fail scoring of the USMLE Step 1. A primary goal of introducing binary scoring in both medical school grading and the USMLE was to improve student well-being, as traditional grading systems have been associated with a higher rate of medical student burnout.^4-6 However, PDs were equivocal about such an impact on student well-being. Furthermore, PDs indicated that the importance of objective measures would merely shift to the USMLE Step 2 CK, which will still be graded with a 3-digit numeric score. Therefore, Step 2 likely will become the source of anxiety for medical students that was once synonymous with Step 1.

Another goal of the scoring change was to encourage a more holistic approach to applicant review, rather than focusing on numerical metrics. However, with most curricula adopting pass/fail models, there is already a lack of objective measures. Although removal of USMLE Step 1 scores could increase the focus on subjective measures, such as letters of recommendation and rank in medical school class (as indicated by our survey), these are susceptible to bias and may not be the best indicators of applicant suitability. This finding also is concerning for maintaining an equitable application process: PDs indicated that the USMLE Step 1 scoring change would not decrease socioeconomic disparities within the selection process.

In dermatology and other competitive specialties, in which USMLE Step 1 scores have become an important consideration, PDs and residency programs will need to identify additional metrics to compare applicants. Examples include research productivity, grades on relevant rotations, and shelf examination scores. Although more reliable subjective measures, such as interviews and performance on away rotations, are already important, they may become of greater significance.

The findings of our survey suggest that PDs are skeptical about changes to Step 1 and more diligence is necessary to maintain a fair and impartial selection process. Increased emphasis on other objective measurements, such as shelf examination scores, graded curricular components, and research productivity, could help maintain an unbiased approach. With changes to USMLE Step 1 expected to be implemented in the 2022 application cycle, programs may need to explore additional options to maintain reliable and transparent applicant review practices.

To the Editor:

With a trend toward increasing pass/fail medical school curricula, residency program directors (PDs) have relied on the US Medical Licensing Examination (USMLE) Step 1 as an objective measurement of applicant achievement, which is particularly true in competitive subspecialties such as dermatology, plastic surgery, orthopedic surgery, ophthalmology, and neurosurgery, in which reported Step 1 scores are consistently the highest among matched applicants.¹ Program directors in dermatology have indicated that Step 1 scores are a priority when considering an applicant.² However, among PDs, the general perception of plans to change Step 1 scores to pass/fail has largely been negative.³ Although the impact of this change on the dermatology residency selection process remains unknown, we undertook a study to determine dermatology PDs’ perspectives on the scoring change and discuss its potential implications among all competitive specialties.

A 19-question survey was designed that assessed PD demographics and opinions of the changes and potential implications of the Step 1 scoring change (eTable). A list of current US dermatology PDs at osteopathic and allopathic programs was obtained through the 2019-2020 Accreditation Council for Graduate Medical Education list of accredited programs. Surveys were piloted at our institution to assess for internal validity and misleading questions, and then were distributed electronically through REDCap software (https://www.project-redcap.org/). All responses were kept anonymous. Institutional review board approval was obtained. Variables were assessed with means, proportions, and CIs. Results were deemed statistically significant with nonoverlapping 99% CIs (P<.01).

Of 139 surveys, 57 (41.0%) were completed. Most PDs (54.4% [31/57]) were women. The average years of service as a PD was 8.5 years. Most PDs (61.4% [35/57]) disagreed with the scoring change; 77.2% (44/57) of PDs noted that it would make it difficult to objectively assess candidates. Program directors indicated that this change would increase the emphasis they place on USMLE Step 2 Clinical Knowledge (CK) scores (86.0% [49/57]); 78.2% (43/55) reported that they would start requiring Step 2 CK results with submitted applications.

Meanwhile, 73.7% (42/57) of PDs disagreed that Step 2 CK should be changed to pass/fail. Most PDs (50.9% [29/57]) thought that binary Step 1 scoring would increase the importance of medical school reputation in application decisions. The percentage of PDs who were neutral (eTable) on whether pass/fail scoring would place international graduates at a disadvantage was 52.6% (30/57), decrease socioeconomic disparities in the application process was 46.4% (26/56), and improve student well-being was 38.2% (21/55).

Results of our survey indicate generally negative perceptions by dermatology PDs to pass/fail scoring of the USMLE Step 1. A primary goal of introducing binary scoring in both medical school grading and the USMLE was to improve student well-being, as traditional grading systems have been associated with a higher rate of medical student burnout.^4-6 However, PDs were equivocal about such an impact on student well-being. Furthermore, PDs indicated that the importance of objective measures would merely shift to the USMLE Step 2 CK, which will still be graded with a 3-digit numeric score. Therefore, Step 2 likely will become the source of anxiety for medical students that was once synonymous with Step 1.

Another goal of the scoring change was to encourage a more holistic approach to applicant review, rather than focusing on numerical metrics. However, with most curricula adopting pass/fail models, there is already a lack of objective measures. Although removal of USMLE Step 1 scores could increase the focus on subjective measures, such as letters of recommendation and rank in medical school class (as indicated by our survey), these are susceptible to bias and may not be the best indicators of applicant suitability. This finding also is concerning for maintaining an equitable application process: PDs indicated that the USMLE Step 1 scoring change would not decrease socioeconomic disparities within the selection process.

In dermatology and other competitive specialties, in which USMLE Step 1 scores have become an important consideration, PDs and residency programs will need to identify additional metrics to compare applicants. Examples include research productivity, grades on relevant rotations, and shelf examination scores. Although more reliable subjective measures, such as interviews and performance on away rotations, are already important, they may become of greater significance.

The findings of our survey suggest that PDs are skeptical about changes to Step 1 and more diligence is necessary to maintain a fair and impartial selection process. Increased emphasis on other objective measurements, such as shelf examination scores, graded curricular components, and research productivity, could help maintain an unbiased approach. With changes to USMLE Step 1 expected to be implemented in the 2022 application cycle, programs may need to explore additional options to maintain reliable and transparent applicant review practices.

To the Editor:

With a trend toward increasing pass/fail medical school curricula, residency program directors (PDs) have relied on the US Medical Licensing Examination (USMLE) Step 1 as an objective measurement of applicant achievement, which is particularly true in competitive subspecialties such as dermatology, plastic surgery, orthopedic surgery, ophthalmology, and neurosurgery, in which reported Step 1 scores are consistently the highest among matched applicants.¹ Program directors in dermatology have indicated that Step 1 scores are a priority when considering an applicant.² However, among PDs, the general perception of plans to change Step 1 scores to pass/fail has largely been negative.³ Although the impact of this change on the dermatology residency selection process remains unknown, we undertook a study to determine dermatology PDs’ perspectives on the scoring change and discuss its potential implications among all competitive specialties.

A 19-question survey was designed that assessed PD demographics and opinions of the changes and potential implications of the Step 1 scoring change (eTable). A list of current US dermatology PDs at osteopathic and allopathic programs was obtained through the 2019-2020 Accreditation Council for Graduate Medical Education list of accredited programs. Surveys were piloted at our institution to assess for internal validity and misleading questions, and then were distributed electronically through REDCap software (https://www.project-redcap.org/). All responses were kept anonymous. Institutional review board approval was obtained. Variables were assessed with means, proportions, and CIs. Results were deemed statistically significant with nonoverlapping 99% CIs (P<.01).

Of 139 surveys, 57 (41.0%) were completed. Most PDs (54.4% [31/57]) were women. The average years of service as a PD was 8.5 years. Most PDs (61.4% [35/57]) disagreed with the scoring change; 77.2% (44/57) of PDs noted that it would make it difficult to objectively assess candidates. Program directors indicated that this change would increase the emphasis they place on USMLE Step 2 Clinical Knowledge (CK) scores (86.0% [49/57]); 78.2% (43/55) reported that they would start requiring Step 2 CK results with submitted applications.

Meanwhile, 73.7% (42/57) of PDs disagreed that Step 2 CK should be changed to pass/fail. Most PDs (50.9% [29/57]) thought that binary Step 1 scoring would increase the importance of medical school reputation in application decisions. The percentage of PDs who were neutral (eTable) on whether pass/fail scoring would place international graduates at a disadvantage was 52.6% (30/57), decrease socioeconomic disparities in the application process was 46.4% (26/56), and improve student well-being was 38.2% (21/55).

Results of our survey indicate generally negative perceptions by dermatology PDs to pass/fail scoring of the USMLE Step 1. A primary goal of introducing binary scoring in both medical school grading and the USMLE was to improve student well-being, as traditional grading systems have been associated with a higher rate of medical student burnout.^4-6 However, PDs were equivocal about such an impact on student well-being. Furthermore, PDs indicated that the importance of objective measures would merely shift to the USMLE Step 2 CK, which will still be graded with a 3-digit numeric score. Therefore, Step 2 likely will become the source of anxiety for medical students that was once synonymous with Step 1.

Another goal of the scoring change was to encourage a more holistic approach to applicant review, rather than focusing on numerical metrics. However, with most curricula adopting pass/fail models, there is already a lack of objective measures. Although removal of USMLE Step 1 scores could increase the focus on subjective measures, such as letters of recommendation and rank in medical school class (as indicated by our survey), these are susceptible to bias and may not be the best indicators of applicant suitability. This finding also is concerning for maintaining an equitable application process: PDs indicated that the USMLE Step 1 scoring change would not decrease socioeconomic disparities within the selection process.

In dermatology and other competitive specialties, in which USMLE Step 1 scores have become an important consideration, PDs and residency programs will need to identify additional metrics to compare applicants. Examples include research productivity, grades on relevant rotations, and shelf examination scores. Although more reliable subjective measures, such as interviews and performance on away rotations, are already important, they may become of greater significance.

The findings of our survey suggest that PDs are skeptical about changes to Step 1 and more diligence is necessary to maintain a fair and impartial selection process. Increased emphasis on other objective measurements, such as shelf examination scores, graded curricular components, and research productivity, could help maintain an unbiased approach. With changes to USMLE Step 1 expected to be implemented in the 2022 application cycle, programs may need to explore additional options to maintain reliable and transparent applicant review practices.