Effect of Educational Intervention on Intern Confidence

Cross‐cover is defined as an on‐call physician managing acute problems such as chest pain, dyspnea, and hypoxemia for patients primarily cared for by another physician. Cross‐cover problems are commonly encountered with hospitalized patients, and inappropriate evaluation and management can result in misdiagnosis. Residents in many internal medicine residency programs receive only informal instruction about how to manage cross‐cover problems, usually from senior medical residents. Unfortunately, instruction is often provided while a patient is experiencing a problem, a frequent occurrence in the chaotic circumstances of a stressful learning environment. Furthermore, the knowledge base, experience, and teaching skills of senior residents vary substantially, and typically senior residents receive no formal instruction to guide them in how or what to teach more junior residents. If formal instruction is provided to residents, it is typically through often poorly attended didactic lectures that have been shown to be an ineffective forum for acquiring skills or changing physician behavior.15

Although previous studies did find that educational interventions can improve confidence and increase knowledge about various aspects of residency training, many of these studies were not randomized,68 or they involved complex interventions requiring a significant amount of resident and teaching staff time.911 The few randomized studies that used simple educational interventions focused on outpatient education, but most of a resident's time is spent in an inpatient setting.1213

Therefore, we designed a simple, randomized educational intervention consisting of 2 formal small‐group, case‐based discussion sessions addressing 1 cross‐cover situation: a hospitalized patient with acute dyspnea. We hypothesized that the addition of small‐group sessions would improve intern knowledge about and confidence in managing acute dyspnea above that gained from a combination of informal education and formal but lecture‐based education.

METHODS

Thirty‐eight internal medicine residents in their first year of postgraduate training (interns) at the University of Michigan were approached to participate in the study. Twenty‐six interns signed informed consent forms and were randomized using a random number generator to receive either the standard education (the control group) or the standard education plus the educational intervention (the intervention group). The standard education was informal teaching by senior medical residents on the wards and a 1‐hour lecture on Approach to the Patient with Acute Dyspnea, taught by an attending physician from the Department of Pulmonary and Critical Care Medicine. The educational intervention included the standard education as well as 2 small‐group, case‐based interactive sessions on acute dyspnea management. Both sessions were developed and taught by the first author (T.M.R.), a third‐year resident in internal medicine. A senior resident taught the sessions to try to make the information more relevant and practical and to make asking questions less intimidating. The first session, which lasted 50 minutes, discussed cases of bronchospasm, pulmonary edema, and pulmonary embolism as causes of acute dyspnea. It addressed several concepts: knowing when and how quickly to evaluate a dyspneic patient, formulating a differential diagnosis, appropriately evaluating acute dyspnea, providing empiric therapy, and recognizing indications for intubation. The second small‐group session occurred approximately 1 month after the first session and lasted 30 minutes. In this session key concepts learned during the first session were reviewed, and a case of ventricular tachycardia presenting as acute dyspnea was discussed. In an effort to increase attendance, free food and drink were provided at each session, and participants were sent reminders via e‐mail and the paging system prior to each session.

All study participants completed pre‐ and postintervention surveys that assessed their knowledge of acute dyspnea management and their confidence in managing patients with this condition. The pretests were conducted just before the first small‐group session was held. The post‐tests were conducted 4 months later. Knowledge was assessed by the score on the 45‐point test, which contained both open‐ and closed‐ended questions derived from 10 case‐based items. The number of points that a question was worth varied depending on how many elements made up a correct answer. For example, one question asked, What tests (if any) do you plan to order immediately after you examine the patient? As 3 tests should have been obtained (EKG, CXR, and ABG), this item had a maximum score of 3 points. Confidence was assessed by averaging 17 items scored on a 5‐point Likert scale (from strongly agree to strongly disagree). The items measured the physician's confidence in managing various aspects of the dyspneic patient (eg, confidence in knowing when to intubate a patient, when to obtain an ABG/CXR/EKG, and when to transfer a patient to the ICU). Data were analyzed using repeated‐measures analysis of variance. Primary analysis was based on the intention‐to‐treat principle, with alpha set to .05 (2‐sided). A secondary, per‐protocol analysis was also performed. In this analysis, study participants who attended both small‐group sessions (ie, completed the entire intervention) were compared with the control group. The protocol was approved by the institutional review board at the University of Michigan Health System.

RESULTS

All participants completed the study. Overall, only 3 of the 26 interns attended the lecture on Approach to the Patient with Acute Dyspnea. Fourteen of the 16 interns assigned to the intervention group attended 1 of the 2 small‐group sessions (11 attended the first session, and 10 attended the second session). Seven interns attended both sessions. The study period was 4 months. Both the intervention and control groups reported managing a similar number of patients with acute dyspnea, both prior to the study (mean of 5.9 in the intervention group and 7.4 in the control group, P = .51) and at the end of study (mean 10.6 in the intervention group and 10.2 in the control group, P = .91). There was no significant difference in the total number of completed inpatient months (mean of 4.9 in the intervention group and 4.7 in the control group, P =. 32) or in the number of inpatient months completed prior to the start of the study (mean of 2 in the intervention group and 2.4 in the control group, P = .15).

Confidence

Subjects in both the intervention and control groups showed increased confidence over time. The mean score of the intervention group increased from 3.77 to 4.57 (a 21.2% increase) and that of the control group increased from 3.74 to 4.28 (a 14.4% increase). Although the trend over time was highly significant for both groups (P < .001), the effect of the intervention was not significant (P = .19). However, the power to detect a difference between the groups was low (0.25). In the per‐protocol analysis, there was no significant difference between the groups (P = .26; see Fig. 1).

Figure 1

Knowledge

In the primary analysis, results for knowledge were similar to those obtained for the confidence outcome. In the intervention group, the mean score increased from 35.6 to 38.3 (a 7.6% increase); in the control group, the mean increased from 36.2 to 38.2 (a 5.5% increase). Scores ranged from 31 to 42. Again, the trend for both groups was significant (P < .01), but the effect of the intervention was not significant (P = .65). The power to detect a difference between groups was again low (0.07). In the per‐protocol analysis a trend toward significance was seen, with mean scores increasing from 34.6 to 40.0, a 15.6% increase (P = .067; see Fig. 2).

Figure 2

DISCUSSION

Our randomized controlled trial found that intern confidence and knowledge about acute dyspnea management both increased significantly over time; however, no significant differences between the intervention and control groups were observed. The complete intervention was not administered to the vast majority of those in the intervention group, however, likely skewing results toward the null. As suggested by the per‐protocol analysis, there was a trend toward a significant increase in the knowledge of the interns who had received the entire intervention. This is similar to results found in a randomized study by Schroy et al., which demonstrated a significant increase in resident knowledge of colorectal cancer screening after an educational intervention that used an interactive, case‐based seminar.13

Our study had several strengths. First, we employed the most robust design to detect efficacy, a randomized controlled study design. Second, we had complete follow‐up because all participants finished the study. Finally, our intervention is easily reproducible.

Our findings should also be considered within the context of several limitations. Despite the use of a random number generator, the control and intervention groups were unequal in number, which may have affected the results, particularly with such a small sample size.

Second, the intervention did not occur until 3 months after the start of each participant's internship. The intention was to implement the intervention at the start of internship, but institutional review board approval did not occur for an additional 3 months. This late timing might have been unfortunate because interns may already have had an established management plan for acute dyspnea, making their behavior more difficult to alter, even with additional education.

Third, because we were unaware of available test instruments to assess resident knowledge of acute dyspnea in the hospitalized patient, we needed to create our own. Unfortunately, the instrument yielded only a small variance in test scores, which may have made it difficult to detect an effect on scores if present.

Fourth, attendance at each session was suboptimal, and thus the complete intervention was not administered to the vast majority of those in the intervention group. Because the first small‐group session was the main teaching session, interns who only attended the second session were exposed to just one case discussion and only a review, rather than a full formal discussion, of the material presented during the first session. Therefore, it is not known if the intervention really had no effect or if no differences were detected simply because the complete intervention was not received. The trend toward significance observed in the per‐protocol analysis suggests that compliance with the intervention may be the key to improving knowledge.

Given the small differences observed in this study, future interventions ideally should use a more sensitive testing instrument, a larger sample, and a more powerful intervention that occurs early in training. Future efforts should also be designed to improve attendance at educational interventions. In the setting of reduced resident work hours and increased demands on resident time, this will prove to be a true challenge for all educators and residency programs.

Cross‐cover is defined as an on‐call physician managing acute problems such as chest pain, dyspnea, and hypoxemia for patients primarily cared for by another physician. Cross‐cover problems are commonly encountered with hospitalized patients, and inappropriate evaluation and management can result in misdiagnosis. Residents in many internal medicine residency programs receive only informal instruction about how to manage cross‐cover problems, usually from senior medical residents. Unfortunately, instruction is often provided while a patient is experiencing a problem, a frequent occurrence in the chaotic circumstances of a stressful learning environment. Furthermore, the knowledge base, experience, and teaching skills of senior residents vary substantially, and typically senior residents receive no formal instruction to guide them in how or what to teach more junior residents. If formal instruction is provided to residents, it is typically through often poorly attended didactic lectures that have been shown to be an ineffective forum for acquiring skills or changing physician behavior.15

Although previous studies did find that educational interventions can improve confidence and increase knowledge about various aspects of residency training, many of these studies were not randomized,68 or they involved complex interventions requiring a significant amount of resident and teaching staff time.911 The few randomized studies that used simple educational interventions focused on outpatient education, but most of a resident's time is spent in an inpatient setting.1213

Therefore, we designed a simple, randomized educational intervention consisting of 2 formal small‐group, case‐based discussion sessions addressing 1 cross‐cover situation: a hospitalized patient with acute dyspnea. We hypothesized that the addition of small‐group sessions would improve intern knowledge about and confidence in managing acute dyspnea above that gained from a combination of informal education and formal but lecture‐based education.

METHODS

Thirty‐eight internal medicine residents in their first year of postgraduate training (interns) at the University of Michigan were approached to participate in the study. Twenty‐six interns signed informed consent forms and were randomized using a random number generator to receive either the standard education (the control group) or the standard education plus the educational intervention (the intervention group). The standard education was informal teaching by senior medical residents on the wards and a 1‐hour lecture on Approach to the Patient with Acute Dyspnea, taught by an attending physician from the Department of Pulmonary and Critical Care Medicine. The educational intervention included the standard education as well as 2 small‐group, case‐based interactive sessions on acute dyspnea management. Both sessions were developed and taught by the first author (T.M.R.), a third‐year resident in internal medicine. A senior resident taught the sessions to try to make the information more relevant and practical and to make asking questions less intimidating. The first session, which lasted 50 minutes, discussed cases of bronchospasm, pulmonary edema, and pulmonary embolism as causes of acute dyspnea. It addressed several concepts: knowing when and how quickly to evaluate a dyspneic patient, formulating a differential diagnosis, appropriately evaluating acute dyspnea, providing empiric therapy, and recognizing indications for intubation. The second small‐group session occurred approximately 1 month after the first session and lasted 30 minutes. In this session key concepts learned during the first session were reviewed, and a case of ventricular tachycardia presenting as acute dyspnea was discussed. In an effort to increase attendance, free food and drink were provided at each session, and participants were sent reminders via e‐mail and the paging system prior to each session.

All study participants completed pre‐ and postintervention surveys that assessed their knowledge of acute dyspnea management and their confidence in managing patients with this condition. The pretests were conducted just before the first small‐group session was held. The post‐tests were conducted 4 months later. Knowledge was assessed by the score on the 45‐point test, which contained both open‐ and closed‐ended questions derived from 10 case‐based items. The number of points that a question was worth varied depending on how many elements made up a correct answer. For example, one question asked, What tests (if any) do you plan to order immediately after you examine the patient? As 3 tests should have been obtained (EKG, CXR, and ABG), this item had a maximum score of 3 points. Confidence was assessed by averaging 17 items scored on a 5‐point Likert scale (from strongly agree to strongly disagree). The items measured the physician's confidence in managing various aspects of the dyspneic patient (eg, confidence in knowing when to intubate a patient, when to obtain an ABG/CXR/EKG, and when to transfer a patient to the ICU). Data were analyzed using repeated‐measures analysis of variance. Primary analysis was based on the intention‐to‐treat principle, with alpha set to .05 (2‐sided). A secondary, per‐protocol analysis was also performed. In this analysis, study participants who attended both small‐group sessions (ie, completed the entire intervention) were compared with the control group. The protocol was approved by the institutional review board at the University of Michigan Health System.

RESULTS

All participants completed the study. Overall, only 3 of the 26 interns attended the lecture on Approach to the Patient with Acute Dyspnea. Fourteen of the 16 interns assigned to the intervention group attended 1 of the 2 small‐group sessions (11 attended the first session, and 10 attended the second session). Seven interns attended both sessions. The study period was 4 months. Both the intervention and control groups reported managing a similar number of patients with acute dyspnea, both prior to the study (mean of 5.9 in the intervention group and 7.4 in the control group, P = .51) and at the end of study (mean 10.6 in the intervention group and 10.2 in the control group, P = .91). There was no significant difference in the total number of completed inpatient months (mean of 4.9 in the intervention group and 4.7 in the control group, P =. 32) or in the number of inpatient months completed prior to the start of the study (mean of 2 in the intervention group and 2.4 in the control group, P = .15).

Confidence

Subjects in both the intervention and control groups showed increased confidence over time. The mean score of the intervention group increased from 3.77 to 4.57 (a 21.2% increase) and that of the control group increased from 3.74 to 4.28 (a 14.4% increase). Although the trend over time was highly significant for both groups (P < .001), the effect of the intervention was not significant (P = .19). However, the power to detect a difference between the groups was low (0.25). In the per‐protocol analysis, there was no significant difference between the groups (P = .26; see Fig. 1).

Figure 1

Knowledge

In the primary analysis, results for knowledge were similar to those obtained for the confidence outcome. In the intervention group, the mean score increased from 35.6 to 38.3 (a 7.6% increase); in the control group, the mean increased from 36.2 to 38.2 (a 5.5% increase). Scores ranged from 31 to 42. Again, the trend for both groups was significant (P < .01), but the effect of the intervention was not significant (P = .65). The power to detect a difference between groups was again low (0.07). In the per‐protocol analysis a trend toward significance was seen, with mean scores increasing from 34.6 to 40.0, a 15.6% increase (P = .067; see Fig. 2).

Figure 2

DISCUSSION

Our randomized controlled trial found that intern confidence and knowledge about acute dyspnea management both increased significantly over time; however, no significant differences between the intervention and control groups were observed. The complete intervention was not administered to the vast majority of those in the intervention group, however, likely skewing results toward the null. As suggested by the per‐protocol analysis, there was a trend toward a significant increase in the knowledge of the interns who had received the entire intervention. This is similar to results found in a randomized study by Schroy et al., which demonstrated a significant increase in resident knowledge of colorectal cancer screening after an educational intervention that used an interactive, case‐based seminar.13

Our study had several strengths. First, we employed the most robust design to detect efficacy, a randomized controlled study design. Second, we had complete follow‐up because all participants finished the study. Finally, our intervention is easily reproducible.

Our findings should also be considered within the context of several limitations. Despite the use of a random number generator, the control and intervention groups were unequal in number, which may have affected the results, particularly with such a small sample size.

Second, the intervention did not occur until 3 months after the start of each participant's internship. The intention was to implement the intervention at the start of internship, but institutional review board approval did not occur for an additional 3 months. This late timing might have been unfortunate because interns may already have had an established management plan for acute dyspnea, making their behavior more difficult to alter, even with additional education.

Third, because we were unaware of available test instruments to assess resident knowledge of acute dyspnea in the hospitalized patient, we needed to create our own. Unfortunately, the instrument yielded only a small variance in test scores, which may have made it difficult to detect an effect on scores if present.

Fourth, attendance at each session was suboptimal, and thus the complete intervention was not administered to the vast majority of those in the intervention group. Because the first small‐group session was the main teaching session, interns who only attended the second session were exposed to just one case discussion and only a review, rather than a full formal discussion, of the material presented during the first session. Therefore, it is not known if the intervention really had no effect or if no differences were detected simply because the complete intervention was not received. The trend toward significance observed in the per‐protocol analysis suggests that compliance with the intervention may be the key to improving knowledge.

Given the small differences observed in this study, future interventions ideally should use a more sensitive testing instrument, a larger sample, and a more powerful intervention that occurs early in training. Future efforts should also be designed to improve attendance at educational interventions. In the setting of reduced resident work hours and increased demands on resident time, this will prove to be a true challenge for all educators and residency programs.

Cross‐cover is defined as an on‐call physician managing acute problems such as chest pain, dyspnea, and hypoxemia for patients primarily cared for by another physician. Cross‐cover problems are commonly encountered with hospitalized patients, and inappropriate evaluation and management can result in misdiagnosis. Residents in many internal medicine residency programs receive only informal instruction about how to manage cross‐cover problems, usually from senior medical residents. Unfortunately, instruction is often provided while a patient is experiencing a problem, a frequent occurrence in the chaotic circumstances of a stressful learning environment. Furthermore, the knowledge base, experience, and teaching skills of senior residents vary substantially, and typically senior residents receive no formal instruction to guide them in how or what to teach more junior residents. If formal instruction is provided to residents, it is typically through often poorly attended didactic lectures that have been shown to be an ineffective forum for acquiring skills or changing physician behavior.15

Although previous studies did find that educational interventions can improve confidence and increase knowledge about various aspects of residency training, many of these studies were not randomized,68 or they involved complex interventions requiring a significant amount of resident and teaching staff time.911 The few randomized studies that used simple educational interventions focused on outpatient education, but most of a resident's time is spent in an inpatient setting.1213

Therefore, we designed a simple, randomized educational intervention consisting of 2 formal small‐group, case‐based discussion sessions addressing 1 cross‐cover situation: a hospitalized patient with acute dyspnea. We hypothesized that the addition of small‐group sessions would improve intern knowledge about and confidence in managing acute dyspnea above that gained from a combination of informal education and formal but lecture‐based education.

METHODS

Thirty‐eight internal medicine residents in their first year of postgraduate training (interns) at the University of Michigan were approached to participate in the study. Twenty‐six interns signed informed consent forms and were randomized using a random number generator to receive either the standard education (the control group) or the standard education plus the educational intervention (the intervention group). The standard education was informal teaching by senior medical residents on the wards and a 1‐hour lecture on Approach to the Patient with Acute Dyspnea, taught by an attending physician from the Department of Pulmonary and Critical Care Medicine. The educational intervention included the standard education as well as 2 small‐group, case‐based interactive sessions on acute dyspnea management. Both sessions were developed and taught by the first author (T.M.R.), a third‐year resident in internal medicine. A senior resident taught the sessions to try to make the information more relevant and practical and to make asking questions less intimidating. The first session, which lasted 50 minutes, discussed cases of bronchospasm, pulmonary edema, and pulmonary embolism as causes of acute dyspnea. It addressed several concepts: knowing when and how quickly to evaluate a dyspneic patient, formulating a differential diagnosis, appropriately evaluating acute dyspnea, providing empiric therapy, and recognizing indications for intubation. The second small‐group session occurred approximately 1 month after the first session and lasted 30 minutes. In this session key concepts learned during the first session were reviewed, and a case of ventricular tachycardia presenting as acute dyspnea was discussed. In an effort to increase attendance, free food and drink were provided at each session, and participants were sent reminders via e‐mail and the paging system prior to each session.

All study participants completed pre‐ and postintervention surveys that assessed their knowledge of acute dyspnea management and their confidence in managing patients with this condition. The pretests were conducted just before the first small‐group session was held. The post‐tests were conducted 4 months later. Knowledge was assessed by the score on the 45‐point test, which contained both open‐ and closed‐ended questions derived from 10 case‐based items. The number of points that a question was worth varied depending on how many elements made up a correct answer. For example, one question asked, What tests (if any) do you plan to order immediately after you examine the patient? As 3 tests should have been obtained (EKG, CXR, and ABG), this item had a maximum score of 3 points. Confidence was assessed by averaging 17 items scored on a 5‐point Likert scale (from strongly agree to strongly disagree). The items measured the physician's confidence in managing various aspects of the dyspneic patient (eg, confidence in knowing when to intubate a patient, when to obtain an ABG/CXR/EKG, and when to transfer a patient to the ICU). Data were analyzed using repeated‐measures analysis of variance. Primary analysis was based on the intention‐to‐treat principle, with alpha set to .05 (2‐sided). A secondary, per‐protocol analysis was also performed. In this analysis, study participants who attended both small‐group sessions (ie, completed the entire intervention) were compared with the control group. The protocol was approved by the institutional review board at the University of Michigan Health System.

RESULTS

All participants completed the study. Overall, only 3 of the 26 interns attended the lecture on Approach to the Patient with Acute Dyspnea. Fourteen of the 16 interns assigned to the intervention group attended 1 of the 2 small‐group sessions (11 attended the first session, and 10 attended the second session). Seven interns attended both sessions. The study period was 4 months. Both the intervention and control groups reported managing a similar number of patients with acute dyspnea, both prior to the study (mean of 5.9 in the intervention group and 7.4 in the control group, P = .51) and at the end of study (mean 10.6 in the intervention group and 10.2 in the control group, P = .91). There was no significant difference in the total number of completed inpatient months (mean of 4.9 in the intervention group and 4.7 in the control group, P =. 32) or in the number of inpatient months completed prior to the start of the study (mean of 2 in the intervention group and 2.4 in the control group, P = .15).

Confidence

Subjects in both the intervention and control groups showed increased confidence over time. The mean score of the intervention group increased from 3.77 to 4.57 (a 21.2% increase) and that of the control group increased from 3.74 to 4.28 (a 14.4% increase). Although the trend over time was highly significant for both groups (P < .001), the effect of the intervention was not significant (P = .19). However, the power to detect a difference between the groups was low (0.25). In the per‐protocol analysis, there was no significant difference between the groups (P = .26; see Fig. 1).

Figure 1

Knowledge

In the primary analysis, results for knowledge were similar to those obtained for the confidence outcome. In the intervention group, the mean score increased from 35.6 to 38.3 (a 7.6% increase); in the control group, the mean increased from 36.2 to 38.2 (a 5.5% increase). Scores ranged from 31 to 42. Again, the trend for both groups was significant (P < .01), but the effect of the intervention was not significant (P = .65). The power to detect a difference between groups was again low (0.07). In the per‐protocol analysis a trend toward significance was seen, with mean scores increasing from 34.6 to 40.0, a 15.6% increase (P = .067; see Fig. 2).

Figure 2

DISCUSSION

Our randomized controlled trial found that intern confidence and knowledge about acute dyspnea management both increased significantly over time; however, no significant differences between the intervention and control groups were observed. The complete intervention was not administered to the vast majority of those in the intervention group, however, likely skewing results toward the null. As suggested by the per‐protocol analysis, there was a trend toward a significant increase in the knowledge of the interns who had received the entire intervention. This is similar to results found in a randomized study by Schroy et al., which demonstrated a significant increase in resident knowledge of colorectal cancer screening after an educational intervention that used an interactive, case‐based seminar.13

Our study had several strengths. First, we employed the most robust design to detect efficacy, a randomized controlled study design. Second, we had complete follow‐up because all participants finished the study. Finally, our intervention is easily reproducible.

Our findings should also be considered within the context of several limitations. Despite the use of a random number generator, the control and intervention groups were unequal in number, which may have affected the results, particularly with such a small sample size.

Second, the intervention did not occur until 3 months after the start of each participant's internship. The intention was to implement the intervention at the start of internship, but institutional review board approval did not occur for an additional 3 months. This late timing might have been unfortunate because interns may already have had an established management plan for acute dyspnea, making their behavior more difficult to alter, even with additional education.

Third, because we were unaware of available test instruments to assess resident knowledge of acute dyspnea in the hospitalized patient, we needed to create our own. Unfortunately, the instrument yielded only a small variance in test scores, which may have made it difficult to detect an effect on scores if present.

Fourth, attendance at each session was suboptimal, and thus the complete intervention was not administered to the vast majority of those in the intervention group. Because the first small‐group session was the main teaching session, interns who only attended the second session were exposed to just one case discussion and only a review, rather than a full formal discussion, of the material presented during the first session. Therefore, it is not known if the intervention really had no effect or if no differences were detected simply because the complete intervention was not received. The trend toward significance observed in the per‐protocol analysis suggests that compliance with the intervention may be the key to improving knowledge.

Given the small differences observed in this study, future interventions ideally should use a more sensitive testing instrument, a larger sample, and a more powerful intervention that occurs early in training. Future efforts should also be designed to improve attendance at educational interventions. In the setting of reduced resident work hours and increased demands on resident time, this will prove to be a true challenge for all educators and residency programs.