Harry Steinberg, MD

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.

METHODS

The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.

RESULTS

A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

DISCUSSION

Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.

METHODS

The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.

RESULTS

A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

DISCUSSION

Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.

METHODS

The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.

RESULTS

A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

DISCUSSION

Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.

Given recent changes in the goals and objectives of residency training as well as changes in the functioning of teaching hospitals, traditional educational formats may need to be supplemented or replaced.1 The Accreditation Council for Graduate Medical Education (ACGME) is promoting changes in resident education with the goal of not only enhancing trainee competency using innovative methods but also of demonstrating that these educational innovations result in enhanced quality of patient care and improved patient safety.1 A challenging aspect of these initiatives is that programs are working to implement them at a time when there are greater nonteaching demands on faculty time, mandated resident work‐hour limitations have been instituted, in some states by law, and resident patient care and educational activities are prone to disruptions inherent in caring for patients in a complex health system. Various solutions have been proposed including increased incorporation of self‐directed learning as a means of meeting modern resident educational challenges, yet the ideal tools with which to accomplish this are unknown.

Computer‐based instruction in medicine has been available since the 1960s, and although its use had initially been more widespread in medical student, nursing, allied health professional, and patient education,2, 3 it is being increasingly incorporated into resident education as well. Some studies have shown that for medical students, computer‐based teaching is at least as effective in improving knowledge as conventional lectures4 and that learners' satisfaction with computer‐based formats appears comparable with that of traditional didactic lectures.5 In recent years computer‐based teaching has been applied to resident education in various fields including surgery and surgical subspecialties, pediatrics, and obstetrics and gynecology.612 Little is known, however, about how computer‐based educational methods affect resident knowledge and especially how these methods might affect clinical practice.

Venous thromboembolism (VTE) is a common and hazardous complication of acute inpatient hospitalization.13 Recognizing that errors in proper prescribing and monitoring of anticoagulants are a major cause of acute inpatient morbidity and mortality,14 we began an initiative to educate our residents and improve their patient care practices regarding the proper use of anticoagulants. To accomplish this, we developed a CD‐ROM‐based learning module with the aim of increasing resident knowledge of anticoagulation as well as compliance with national standards for VTE prevention. In this study we assessed the impact of the CD‐ROM intervention on resident knowledge and their appropriate use of VTE prophylaxis.

METHODS

The study was approved by the institutional review board. With the participation of faculty educators in the departments of medicine, surgery, and neurology, one of the authors (H.K.) coordinated the development of a CD‐ROM containing concise modules on core topics in anticoagulation (Table 1). The presenters for these topics included the director of clinical hematology, 2 cardiologists including the director of the coronary care unit, the director of the medical intensive care unit, the director of cerebrovascular diseases, and 2 vascular surgeons, one of whom serves as vice chair of surgery. These modules, each lasting about 1 hour, had audio and slide components detailing the proper indications, monitoring, and efficacy of anticoagulants in atrial fibrillation, acute ischemic stroke, acute coronary syndromes, and VTE prevention in acutely ill hospitalized patients. The guidelines presented were based on the sixth (2000) ACCP guidelines for antithrombotic therapy for the prevention and treatment of thrombosis.15 The content of the CD‐ROM was reviewed for accuracy by the authors, though none of them were speakers. We asked that before all current residents in the departments of cardiothoracic surgery, emergency medicine, otolaryngology, internal medicine, neurosurgery, dental medicine, neurology, obstetrics and gynecology, orthopedics, surgery, and urology viewed the CD‐ROM, they complete a pretest to determine their baseline knowledge of this subject. After completing the pretest, the residents were required to view the CD‐ROM and retake the same test. We then compared pre‐ and posttest scores.

To determine whether an increase in knowledge was secondary to the CD‐ROM intervention or simply a consequence of acquired clinical experience during training, we compared test scores of residents who did and did not receive the CD‐ROM intervention. In the academic year following our initial testing, we asked the incoming categorical medical PGY‐1 classes at our hospital and at a comparable local tertiary‐care hospital in our health system 2 miles away to take an anticoagulation pretest (different from the examination given for the initial testing) during their PGY‐1 orientations. The 2 institutions are comparable in many ways including in patient demographics and size and most residents come from the same medical schools, have a similar rotation structure, and use a comparable curriculum under a unified graduate medical education office. The CD‐ROMs were only given to categorical PGY‐1 residents at our institution. Both groups then retook the same test (posttest) 3 months into their clinical training, a time chosen because it is when all PGY‐1s would be expected to have gained significant clinical experience on the medical wards and or in the intensive care units. The exam questions were generated by one of the authors (B.M.) and covered all the topics in the CD‐ROM.

An Anticoagulation Steering Committee was formed to assess whether the CD‐ROM intervention affected our residents' patient care practices. None of the members of this committee were authors of this work. Members of this committee reviewed inpatient charts and documented resident compliance with VTE prevention standards during periods before and after they had viewed the CD‐ROMs. We chose this particular portion of the CD‐ROM because at both the test and control hospitals, initiatives were underway using order sets to improve anticoagulation in cardiac, neurological, and surgical patients but not in VTE prophylaxis. Charts from the same 2 nursing units on the medical service were reviewed in each period and included patients with a discharge diagnosis of congestive heart failure, any oncologic diagnosis, or sepsis. The chart review tool was developed by the anticoagulation committee and included a thrombosis risk factor assessment section as well as a list of contraindications to anticoagulation to determine if anticoagulation was appropriately implemented. Charts were reviewed for compliance with VTE prophylaxis after the CD‐ROM intervention (given in July 2004) in August 2004. To have a comparable pre‐CD‐ROM comparison, charts of patients with the diagnoses stated above were reviewed from August of the preceding year. The same month was chosen in the previous year to minimize any impact of resident experience, which would likely be a confounding factor if charts from May or June of the academic year were used as a control, for example. To determine whether an improvement in adherence to VTE prophylaxis standards was sustained, an additional chart review was carried out 7 months after the initial CD‐ROM viewing. The same group of observers, none of whom were authors, did all the chart assessments.

RESULTS

Assessment of Independent Effect of CD‐ROM Intervention

To determine what independent effect the CD‐ROM intervention might have, given that scores may improve with the acquisition of clinical experience alone, in July 2004 we tested internal medicine categorical PGY‐1s at our institution and at another tertiary‐care hospital, as described in the Methods section. The results of testing both groups are shown in Figure 1. Nineteen medical PGY‐1s at our hospital (hospital A) completed the anticoagulation pretest, and 16 completed the posttest. Twenty‐two medical PGY‐1s completed the pretest, and 17 completed the posttest at our neighboring hospital (hospital B). Although posttest scores were higher at both institutions, the increase in scores at our institution, which received the CD‐ROM intervention, was statistically significant, whereas the increase for the group not receiving the intervention was not significant. These findings suggest that the CD‐ROM intervention may have had an independent effect on resident knowledge of anticoagulation.

Figure 1

Effect of CD‐ROM Intervention on Resident Use of VTE Prophylaxis

Appropriate use of VTE prophylaxis by residents was assessed at 3 points, as detailed in the Methods section: 1 year before the CD‐ROM intervention (baseline), immediately after the CD‐ROM intervention, and 7 months after the CD‐ROM intervention. VTE prophylaxis, one element of the CD‐ROM, was chosen as a surrogate marker for the impact of the CD‐ROM initiative. A review of 40 charts of patients with the specified diagnoses (100% of the patients with the specified diagnoses, which represented about one third of admissions to the unit) before the CD‐ROM intervention revealed that 30 patients (75%) received appropriate VTE prophylaxis. A review of 38 charts after the CD‐ROM intervention showed that 36 patients (95%) received appropriate prophylaxis; similar findings were obtained 7 months after the CD‐ROM intervention (33 of 35 patients, 94%, P = .0107). These findings, which are shown in Figure 2, suggest that the CD‐ROM intervention enhanced resident compliance with VTE prophylaxis guidelines and that this effect was sustained for at least 7 months.

Figure 2

DISCUSSION

Residency training is facing challenges on several fronts. In addition to substantially changed educational requirements, strict limits on the amount of time that trainees can spend in the hospital have resulted from ACGME requirements and several state laws. Residents who are on night rotations or were on call the night before often miss educational conferences or must choose between attending patient carerelated activities and educational sessions. Time constraints on faculty have compounded this problem, and for residents to effectively learn, the focus of graduate medical education may need to shift somewhat from teaching medical information toward teaching the practice of self‐directed learning, with CD‐ROMs one such mode by which this can take place. Accomplishing this will require novel teaching approaches, and residency programs will need to document their effectiveness.

In this study we demonstrated that our residents increased their knowledge and improved their patient care practices using a CD‐ROM‐based educational tool. Residents frequently make use of computer‐based educational resources in the form of journals, textbooks, informational databases such as comprehensive drug listings, and personal digital assistantbased tools. Advantages of the computer‐based learning format include increased accessibility and flexibility in viewing the material. Residents have the option of repeated screening as desired and of viewing the CD‐ROM in segments if necessary. Although residents often must choose to attend a scheduled traditional lecture or engage in a patient carerelated activity, the CD‐ROM format allows the resident to choose the ideal time and setting to engage in structured educational activities. Other advantages of the CD‐ROM format would be ease of monitoring for accuracy, applicability, and comprehensiveness as well as more flexibility in faculty time commitments. It should be noted that we have no information about how much time residents devoted to the CD‐ROM program and how often they may have returned to the module for review. It should also be noted that although there have been some reports suggesting that CD‐ROM‐based education may play a useful role in student and perhaps resident education,1618 there is no evidence to date demonstrating that widespread use of CD‐ROMs in residency training can differentially affect resident behavior compared with the use of traditional methods.

A number of variables could have affected our results. For overall test scores, the response rate was less than 50%, with variability between departments suggesting that perhaps it was more motivated residents who participated and were therefore more likely to demonstrate improvement. Although our data comparing institutions with and without the CD‐ROM intervention suggested that the CD‐ROM intervention had a discernable effect on resident knowledge, we must also consider the possibility that the 2 groups might not have been comparable, as attitudes, expectations, and other variables might have differed. All the residents were categorical trainees, and given the similarities in many aspects of the training programs in these 2 tertiary‐care hospitals, as described in the Methods section, it is hoped that any such differences were minor. Nevertheless, this must be considered a limitation of our study. Also of note, the number of trainees was small, as was the patient population studied with VTE prophylaxis; hence, we recognize that our work can best be regarded as a pilot study using an alternative learning method. We also realize that giving a group of residents a test followed by distribution of a CD‐ROM might have suggested that we were directing them toward a goal, and this may have affected the results. Heightened awareness of the importance of anticoagulation from the introduction of new guidelines and other variables also could have affected our findings. The taking of an examination itself might also have had an impact on knowledge that could affect subsequent test scores. An additional point to consider is that if knowledge and patient care did improve, we do not know whether this affects residents acquiring other knowledge or whether this will translate into improved patient care in other areas.

Although CD‐ROM‐based learning could serve a useful function in the increasingly complex environment of residency training, this learning method also has disadvantages, including not providing personal contact or having the capability of question‐and‐answer sessions between teacher and resident. This could be overcome by providing time for faculty‐precepted question‐and‐answer sessions or perhaps creating a Web‐based venue for questions to be submitted and answered. In addition, the CD‐ROMs themselves can be designed in an interactive format in which residents can provide answers to clinical questions with feedback based on their selections provided as part of the CD‐ROM program.

In summary, the CD‐ROM‐based program in this study appears to have had an effect on not only knowledge but also patient care practice and suggests that this type of format could serve a useful role in residency training. Studies of additional interventions such as this one might allow for more extensive evaluation of the utility of CD‐ROM‐based learning as a residency training tool.

Given recent changes in the goals and objectives of residency training as well as changes in the functioning of teaching hospitals, traditional educational formats may need to be supplemented or replaced.1 The Accreditation Council for Graduate Medical Education (ACGME) is promoting changes in resident education with the goal of not only enhancing trainee competency using innovative methods but also of demonstrating that these educational innovations result in enhanced quality of patient care and improved patient safety.1 A challenging aspect of these initiatives is that programs are working to implement them at a time when there are greater nonteaching demands on faculty time, mandated resident work‐hour limitations have been instituted, in some states by law, and resident patient care and educational activities are prone to disruptions inherent in caring for patients in a complex health system. Various solutions have been proposed including increased incorporation of self‐directed learning as a means of meeting modern resident educational challenges, yet the ideal tools with which to accomplish this are unknown.

Computer‐based instruction in medicine has been available since the 1960s, and although its use had initially been more widespread in medical student, nursing, allied health professional, and patient education,2, 3 it is being increasingly incorporated into resident education as well. Some studies have shown that for medical students, computer‐based teaching is at least as effective in improving knowledge as conventional lectures4 and that learners' satisfaction with computer‐based formats appears comparable with that of traditional didactic lectures.5 In recent years computer‐based teaching has been applied to resident education in various fields including surgery and surgical subspecialties, pediatrics, and obstetrics and gynecology.612 Little is known, however, about how computer‐based educational methods affect resident knowledge and especially how these methods might affect clinical practice.

Venous thromboembolism (VTE) is a common and hazardous complication of acute inpatient hospitalization.13 Recognizing that errors in proper prescribing and monitoring of anticoagulants are a major cause of acute inpatient morbidity and mortality,14 we began an initiative to educate our residents and improve their patient care practices regarding the proper use of anticoagulants. To accomplish this, we developed a CD‐ROM‐based learning module with the aim of increasing resident knowledge of anticoagulation as well as compliance with national standards for VTE prevention. In this study we assessed the impact of the CD‐ROM intervention on resident knowledge and their appropriate use of VTE prophylaxis.

METHODS

The study was approved by the institutional review board. With the participation of faculty educators in the departments of medicine, surgery, and neurology, one of the authors (H.K.) coordinated the development of a CD‐ROM containing concise modules on core topics in anticoagulation (Table 1). The presenters for these topics included the director of clinical hematology, 2 cardiologists including the director of the coronary care unit, the director of the medical intensive care unit, the director of cerebrovascular diseases, and 2 vascular surgeons, one of whom serves as vice chair of surgery. These modules, each lasting about 1 hour, had audio and slide components detailing the proper indications, monitoring, and efficacy of anticoagulants in atrial fibrillation, acute ischemic stroke, acute coronary syndromes, and VTE prevention in acutely ill hospitalized patients. The guidelines presented were based on the sixth (2000) ACCP guidelines for antithrombotic therapy for the prevention and treatment of thrombosis.15 The content of the CD‐ROM was reviewed for accuracy by the authors, though none of them were speakers. We asked that before all current residents in the departments of cardiothoracic surgery, emergency medicine, otolaryngology, internal medicine, neurosurgery, dental medicine, neurology, obstetrics and gynecology, orthopedics, surgery, and urology viewed the CD‐ROM, they complete a pretest to determine their baseline knowledge of this subject. After completing the pretest, the residents were required to view the CD‐ROM and retake the same test. We then compared pre‐ and posttest scores.

To determine whether an increase in knowledge was secondary to the CD‐ROM intervention or simply a consequence of acquired clinical experience during training, we compared test scores of residents who did and did not receive the CD‐ROM intervention. In the academic year following our initial testing, we asked the incoming categorical medical PGY‐1 classes at our hospital and at a comparable local tertiary‐care hospital in our health system 2 miles away to take an anticoagulation pretest (different from the examination given for the initial testing) during their PGY‐1 orientations. The 2 institutions are comparable in many ways including in patient demographics and size and most residents come from the same medical schools, have a similar rotation structure, and use a comparable curriculum under a unified graduate medical education office. The CD‐ROMs were only given to categorical PGY‐1 residents at our institution. Both groups then retook the same test (posttest) 3 months into their clinical training, a time chosen because it is when all PGY‐1s would be expected to have gained significant clinical experience on the medical wards and or in the intensive care units. The exam questions were generated by one of the authors (B.M.) and covered all the topics in the CD‐ROM.

An Anticoagulation Steering Committee was formed to assess whether the CD‐ROM intervention affected our residents' patient care practices. None of the members of this committee were authors of this work. Members of this committee reviewed inpatient charts and documented resident compliance with VTE prevention standards during periods before and after they had viewed the CD‐ROMs. We chose this particular portion of the CD‐ROM because at both the test and control hospitals, initiatives were underway using order sets to improve anticoagulation in cardiac, neurological, and surgical patients but not in VTE prophylaxis. Charts from the same 2 nursing units on the medical service were reviewed in each period and included patients with a discharge diagnosis of congestive heart failure, any oncologic diagnosis, or sepsis. The chart review tool was developed by the anticoagulation committee and included a thrombosis risk factor assessment section as well as a list of contraindications to anticoagulation to determine if anticoagulation was appropriately implemented. Charts were reviewed for compliance with VTE prophylaxis after the CD‐ROM intervention (given in July 2004) in August 2004. To have a comparable pre‐CD‐ROM comparison, charts of patients with the diagnoses stated above were reviewed from August of the preceding year. The same month was chosen in the previous year to minimize any impact of resident experience, which would likely be a confounding factor if charts from May or June of the academic year were used as a control, for example. To determine whether an improvement in adherence to VTE prophylaxis standards was sustained, an additional chart review was carried out 7 months after the initial CD‐ROM viewing. The same group of observers, none of whom were authors, did all the chart assessments.

RESULTS

Assessment of Independent Effect of CD‐ROM Intervention

To determine what independent effect the CD‐ROM intervention might have, given that scores may improve with the acquisition of clinical experience alone, in July 2004 we tested internal medicine categorical PGY‐1s at our institution and at another tertiary‐care hospital, as described in the Methods section. The results of testing both groups are shown in Figure 1. Nineteen medical PGY‐1s at our hospital (hospital A) completed the anticoagulation pretest, and 16 completed the posttest. Twenty‐two medical PGY‐1s completed the pretest, and 17 completed the posttest at our neighboring hospital (hospital B). Although posttest scores were higher at both institutions, the increase in scores at our institution, which received the CD‐ROM intervention, was statistically significant, whereas the increase for the group not receiving the intervention was not significant. These findings suggest that the CD‐ROM intervention may have had an independent effect on resident knowledge of anticoagulation.

Figure 1

Effect of CD‐ROM Intervention on Resident Use of VTE Prophylaxis

Appropriate use of VTE prophylaxis by residents was assessed at 3 points, as detailed in the Methods section: 1 year before the CD‐ROM intervention (baseline), immediately after the CD‐ROM intervention, and 7 months after the CD‐ROM intervention. VTE prophylaxis, one element of the CD‐ROM, was chosen as a surrogate marker for the impact of the CD‐ROM initiative. A review of 40 charts of patients with the specified diagnoses (100% of the patients with the specified diagnoses, which represented about one third of admissions to the unit) before the CD‐ROM intervention revealed that 30 patients (75%) received appropriate VTE prophylaxis. A review of 38 charts after the CD‐ROM intervention showed that 36 patients (95%) received appropriate prophylaxis; similar findings were obtained 7 months after the CD‐ROM intervention (33 of 35 patients, 94%, P = .0107). These findings, which are shown in Figure 2, suggest that the CD‐ROM intervention enhanced resident compliance with VTE prophylaxis guidelines and that this effect was sustained for at least 7 months.

Figure 2

DISCUSSION

Residency training is facing challenges on several fronts. In addition to substantially changed educational requirements, strict limits on the amount of time that trainees can spend in the hospital have resulted from ACGME requirements and several state laws. Residents who are on night rotations or were on call the night before often miss educational conferences or must choose between attending patient carerelated activities and educational sessions. Time constraints on faculty have compounded this problem, and for residents to effectively learn, the focus of graduate medical education may need to shift somewhat from teaching medical information toward teaching the practice of self‐directed learning, with CD‐ROMs one such mode by which this can take place. Accomplishing this will require novel teaching approaches, and residency programs will need to document their effectiveness.

In this study we demonstrated that our residents increased their knowledge and improved their patient care practices using a CD‐ROM‐based educational tool. Residents frequently make use of computer‐based educational resources in the form of journals, textbooks, informational databases such as comprehensive drug listings, and personal digital assistantbased tools. Advantages of the computer‐based learning format include increased accessibility and flexibility in viewing the material. Residents have the option of repeated screening as desired and of viewing the CD‐ROM in segments if necessary. Although residents often must choose to attend a scheduled traditional lecture or engage in a patient carerelated activity, the CD‐ROM format allows the resident to choose the ideal time and setting to engage in structured educational activities. Other advantages of the CD‐ROM format would be ease of monitoring for accuracy, applicability, and comprehensiveness as well as more flexibility in faculty time commitments. It should be noted that we have no information about how much time residents devoted to the CD‐ROM program and how often they may have returned to the module for review. It should also be noted that although there have been some reports suggesting that CD‐ROM‐based education may play a useful role in student and perhaps resident education,1618 there is no evidence to date demonstrating that widespread use of CD‐ROMs in residency training can differentially affect resident behavior compared with the use of traditional methods.

A number of variables could have affected our results. For overall test scores, the response rate was less than 50%, with variability between departments suggesting that perhaps it was more motivated residents who participated and were therefore more likely to demonstrate improvement. Although our data comparing institutions with and without the CD‐ROM intervention suggested that the CD‐ROM intervention had a discernable effect on resident knowledge, we must also consider the possibility that the 2 groups might not have been comparable, as attitudes, expectations, and other variables might have differed. All the residents were categorical trainees, and given the similarities in many aspects of the training programs in these 2 tertiary‐care hospitals, as described in the Methods section, it is hoped that any such differences were minor. Nevertheless, this must be considered a limitation of our study. Also of note, the number of trainees was small, as was the patient population studied with VTE prophylaxis; hence, we recognize that our work can best be regarded as a pilot study using an alternative learning method. We also realize that giving a group of residents a test followed by distribution of a CD‐ROM might have suggested that we were directing them toward a goal, and this may have affected the results. Heightened awareness of the importance of anticoagulation from the introduction of new guidelines and other variables also could have affected our findings. The taking of an examination itself might also have had an impact on knowledge that could affect subsequent test scores. An additional point to consider is that if knowledge and patient care did improve, we do not know whether this affects residents acquiring other knowledge or whether this will translate into improved patient care in other areas.

Although CD‐ROM‐based learning could serve a useful function in the increasingly complex environment of residency training, this learning method also has disadvantages, including not providing personal contact or having the capability of question‐and‐answer sessions between teacher and resident. This could be overcome by providing time for faculty‐precepted question‐and‐answer sessions or perhaps creating a Web‐based venue for questions to be submitted and answered. In addition, the CD‐ROMs themselves can be designed in an interactive format in which residents can provide answers to clinical questions with feedback based on their selections provided as part of the CD‐ROM program.

In summary, the CD‐ROM‐based program in this study appears to have had an effect on not only knowledge but also patient care practice and suggests that this type of format could serve a useful role in residency training. Studies of additional interventions such as this one might allow for more extensive evaluation of the utility of CD‐ROM‐based learning as a residency training tool.

Given recent changes in the goals and objectives of residency training as well as changes in the functioning of teaching hospitals, traditional educational formats may need to be supplemented or replaced.1 The Accreditation Council for Graduate Medical Education (ACGME) is promoting changes in resident education with the goal of not only enhancing trainee competency using innovative methods but also of demonstrating that these educational innovations result in enhanced quality of patient care and improved patient safety.1 A challenging aspect of these initiatives is that programs are working to implement them at a time when there are greater nonteaching demands on faculty time, mandated resident work‐hour limitations have been instituted, in some states by law, and resident patient care and educational activities are prone to disruptions inherent in caring for patients in a complex health system. Various solutions have been proposed including increased incorporation of self‐directed learning as a means of meeting modern resident educational challenges, yet the ideal tools with which to accomplish this are unknown.

Computer‐based instruction in medicine has been available since the 1960s, and although its use had initially been more widespread in medical student, nursing, allied health professional, and patient education,2, 3 it is being increasingly incorporated into resident education as well. Some studies have shown that for medical students, computer‐based teaching is at least as effective in improving knowledge as conventional lectures4 and that learners' satisfaction with computer‐based formats appears comparable with that of traditional didactic lectures.5 In recent years computer‐based teaching has been applied to resident education in various fields including surgery and surgical subspecialties, pediatrics, and obstetrics and gynecology.612 Little is known, however, about how computer‐based educational methods affect resident knowledge and especially how these methods might affect clinical practice.

Venous thromboembolism (VTE) is a common and hazardous complication of acute inpatient hospitalization.13 Recognizing that errors in proper prescribing and monitoring of anticoagulants are a major cause of acute inpatient morbidity and mortality,14 we began an initiative to educate our residents and improve their patient care practices regarding the proper use of anticoagulants. To accomplish this, we developed a CD‐ROM‐based learning module with the aim of increasing resident knowledge of anticoagulation as well as compliance with national standards for VTE prevention. In this study we assessed the impact of the CD‐ROM intervention on resident knowledge and their appropriate use of VTE prophylaxis.

METHODS

The study was approved by the institutional review board. With the participation of faculty educators in the departments of medicine, surgery, and neurology, one of the authors (H.K.) coordinated the development of a CD‐ROM containing concise modules on core topics in anticoagulation (Table 1). The presenters for these topics included the director of clinical hematology, 2 cardiologists including the director of the coronary care unit, the director of the medical intensive care unit, the director of cerebrovascular diseases, and 2 vascular surgeons, one of whom serves as vice chair of surgery. These modules, each lasting about 1 hour, had audio and slide components detailing the proper indications, monitoring, and efficacy of anticoagulants in atrial fibrillation, acute ischemic stroke, acute coronary syndromes, and VTE prevention in acutely ill hospitalized patients. The guidelines presented were based on the sixth (2000) ACCP guidelines for antithrombotic therapy for the prevention and treatment of thrombosis.15 The content of the CD‐ROM was reviewed for accuracy by the authors, though none of them were speakers. We asked that before all current residents in the departments of cardiothoracic surgery, emergency medicine, otolaryngology, internal medicine, neurosurgery, dental medicine, neurology, obstetrics and gynecology, orthopedics, surgery, and urology viewed the CD‐ROM, they complete a pretest to determine their baseline knowledge of this subject. After completing the pretest, the residents were required to view the CD‐ROM and retake the same test. We then compared pre‐ and posttest scores.

To determine whether an increase in knowledge was secondary to the CD‐ROM intervention or simply a consequence of acquired clinical experience during training, we compared test scores of residents who did and did not receive the CD‐ROM intervention. In the academic year following our initial testing, we asked the incoming categorical medical PGY‐1 classes at our hospital and at a comparable local tertiary‐care hospital in our health system 2 miles away to take an anticoagulation pretest (different from the examination given for the initial testing) during their PGY‐1 orientations. The 2 institutions are comparable in many ways including in patient demographics and size and most residents come from the same medical schools, have a similar rotation structure, and use a comparable curriculum under a unified graduate medical education office. The CD‐ROMs were only given to categorical PGY‐1 residents at our institution. Both groups then retook the same test (posttest) 3 months into their clinical training, a time chosen because it is when all PGY‐1s would be expected to have gained significant clinical experience on the medical wards and or in the intensive care units. The exam questions were generated by one of the authors (B.M.) and covered all the topics in the CD‐ROM.

An Anticoagulation Steering Committee was formed to assess whether the CD‐ROM intervention affected our residents' patient care practices. None of the members of this committee were authors of this work. Members of this committee reviewed inpatient charts and documented resident compliance with VTE prevention standards during periods before and after they had viewed the CD‐ROMs. We chose this particular portion of the CD‐ROM because at both the test and control hospitals, initiatives were underway using order sets to improve anticoagulation in cardiac, neurological, and surgical patients but not in VTE prophylaxis. Charts from the same 2 nursing units on the medical service were reviewed in each period and included patients with a discharge diagnosis of congestive heart failure, any oncologic diagnosis, or sepsis. The chart review tool was developed by the anticoagulation committee and included a thrombosis risk factor assessment section as well as a list of contraindications to anticoagulation to determine if anticoagulation was appropriately implemented. Charts were reviewed for compliance with VTE prophylaxis after the CD‐ROM intervention (given in July 2004) in August 2004. To have a comparable pre‐CD‐ROM comparison, charts of patients with the diagnoses stated above were reviewed from August of the preceding year. The same month was chosen in the previous year to minimize any impact of resident experience, which would likely be a confounding factor if charts from May or June of the academic year were used as a control, for example. To determine whether an improvement in adherence to VTE prophylaxis standards was sustained, an additional chart review was carried out 7 months after the initial CD‐ROM viewing. The same group of observers, none of whom were authors, did all the chart assessments.

RESULTS

Assessment of Independent Effect of CD‐ROM Intervention

To determine what independent effect the CD‐ROM intervention might have, given that scores may improve with the acquisition of clinical experience alone, in July 2004 we tested internal medicine categorical PGY‐1s at our institution and at another tertiary‐care hospital, as described in the Methods section. The results of testing both groups are shown in Figure 1. Nineteen medical PGY‐1s at our hospital (hospital A) completed the anticoagulation pretest, and 16 completed the posttest. Twenty‐two medical PGY‐1s completed the pretest, and 17 completed the posttest at our neighboring hospital (hospital B). Although posttest scores were higher at both institutions, the increase in scores at our institution, which received the CD‐ROM intervention, was statistically significant, whereas the increase for the group not receiving the intervention was not significant. These findings suggest that the CD‐ROM intervention may have had an independent effect on resident knowledge of anticoagulation.

Figure 1

Effect of CD‐ROM Intervention on Resident Use of VTE Prophylaxis

Appropriate use of VTE prophylaxis by residents was assessed at 3 points, as detailed in the Methods section: 1 year before the CD‐ROM intervention (baseline), immediately after the CD‐ROM intervention, and 7 months after the CD‐ROM intervention. VTE prophylaxis, one element of the CD‐ROM, was chosen as a surrogate marker for the impact of the CD‐ROM initiative. A review of 40 charts of patients with the specified diagnoses (100% of the patients with the specified diagnoses, which represented about one third of admissions to the unit) before the CD‐ROM intervention revealed that 30 patients (75%) received appropriate VTE prophylaxis. A review of 38 charts after the CD‐ROM intervention showed that 36 patients (95%) received appropriate prophylaxis; similar findings were obtained 7 months after the CD‐ROM intervention (33 of 35 patients, 94%, P = .0107). These findings, which are shown in Figure 2, suggest that the CD‐ROM intervention enhanced resident compliance with VTE prophylaxis guidelines and that this effect was sustained for at least 7 months.

Figure 2

DISCUSSION

Residency training is facing challenges on several fronts. In addition to substantially changed educational requirements, strict limits on the amount of time that trainees can spend in the hospital have resulted from ACGME requirements and several state laws. Residents who are on night rotations or were on call the night before often miss educational conferences or must choose between attending patient carerelated activities and educational sessions. Time constraints on faculty have compounded this problem, and for residents to effectively learn, the focus of graduate medical education may need to shift somewhat from teaching medical information toward teaching the practice of self‐directed learning, with CD‐ROMs one such mode by which this can take place. Accomplishing this will require novel teaching approaches, and residency programs will need to document their effectiveness.

In this study we demonstrated that our residents increased their knowledge and improved their patient care practices using a CD‐ROM‐based educational tool. Residents frequently make use of computer‐based educational resources in the form of journals, textbooks, informational databases such as comprehensive drug listings, and personal digital assistantbased tools. Advantages of the computer‐based learning format include increased accessibility and flexibility in viewing the material. Residents have the option of repeated screening as desired and of viewing the CD‐ROM in segments if necessary. Although residents often must choose to attend a scheduled traditional lecture or engage in a patient carerelated activity, the CD‐ROM format allows the resident to choose the ideal time and setting to engage in structured educational activities. Other advantages of the CD‐ROM format would be ease of monitoring for accuracy, applicability, and comprehensiveness as well as more flexibility in faculty time commitments. It should be noted that we have no information about how much time residents devoted to the CD‐ROM program and how often they may have returned to the module for review. It should also be noted that although there have been some reports suggesting that CD‐ROM‐based education may play a useful role in student and perhaps resident education,1618 there is no evidence to date demonstrating that widespread use of CD‐ROMs in residency training can differentially affect resident behavior compared with the use of traditional methods.

A number of variables could have affected our results. For overall test scores, the response rate was less than 50%, with variability between departments suggesting that perhaps it was more motivated residents who participated and were therefore more likely to demonstrate improvement. Although our data comparing institutions with and without the CD‐ROM intervention suggested that the CD‐ROM intervention had a discernable effect on resident knowledge, we must also consider the possibility that the 2 groups might not have been comparable, as attitudes, expectations, and other variables might have differed. All the residents were categorical trainees, and given the similarities in many aspects of the training programs in these 2 tertiary‐care hospitals, as described in the Methods section, it is hoped that any such differences were minor. Nevertheless, this must be considered a limitation of our study. Also of note, the number of trainees was small, as was the patient population studied with VTE prophylaxis; hence, we recognize that our work can best be regarded as a pilot study using an alternative learning method. We also realize that giving a group of residents a test followed by distribution of a CD‐ROM might have suggested that we were directing them toward a goal, and this may have affected the results. Heightened awareness of the importance of anticoagulation from the introduction of new guidelines and other variables also could have affected our findings. The taking of an examination itself might also have had an impact on knowledge that could affect subsequent test scores. An additional point to consider is that if knowledge and patient care did improve, we do not know whether this affects residents acquiring other knowledge or whether this will translate into improved patient care in other areas.

Although CD‐ROM‐based learning could serve a useful function in the increasingly complex environment of residency training, this learning method also has disadvantages, including not providing personal contact or having the capability of question‐and‐answer sessions between teacher and resident. This could be overcome by providing time for faculty‐precepted question‐and‐answer sessions or perhaps creating a Web‐based venue for questions to be submitted and answered. In addition, the CD‐ROMs themselves can be designed in an interactive format in which residents can provide answers to clinical questions with feedback based on their selections provided as part of the CD‐ROM program.

In summary, the CD‐ROM‐based program in this study appears to have had an effect on not only knowledge but also patient care practice and suggests that this type of format could serve a useful role in residency training. Studies of additional interventions such as this one might allow for more extensive evaluation of the utility of CD‐ROM‐based learning as a residency training tool.