Quantifying Resident Clinical Experience

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.

METHODS

The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.

RESULTS

A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

DISCUSSION

Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.

METHODS

The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.

RESULTS

A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

DISCUSSION

Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.

METHODS

The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.

RESULTS

A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

DISCUSSION

Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.