Affiliations
Section of Geriatrics, Department of Internal Medicine, Yale School of Medicine
Given name(s)
Peter H.
Family name
Ness
Degrees
PhD, MPH

Handoff CEX

Article Type
Changed
Mon, 05/22/2017 - 18:12
Display Headline
Development of a handoff evaluation tool for shift‐to‐shift physician handoffs: The handoff CEX

Transfers among trainee physicians within the hospital typically occur at least twice a day and have been increasing among trainees as work hours have declined.[1] The 2011 Accreditation Council for Graduate Medical Education (ACGME) guidelines,[2] which restrict intern working hours to 16 hours from a previous maximum of 30, have likely increased the frequency of physician trainee handoffs even further. Similarly, transfers among hospitalist attendings occur at least twice a day, given typical shifts of 8 to 12 hours.

Given the frequency of transfers, and the potential for harm generated by failed transitions,[3, 4, 5, 6] the end‐of‐shift written and verbal handoffs have assumed increasingly greater importance in hospital care among both trainees and hospitalist attendings.

The ACGME now requires that programs assess the competency of trainees in handoff communication.[2] Yet, there are few tools for assessing the quality of sign‐out communication. Those that exist primarily focus on the written sign‐out, and are rarely validated.[7, 8, 9, 10, 11, 12] Furthermore, it is uncertain whether such assessments must be done by supervisors or whether peers can participate in the evaluation. In this prospective multi‐institutional study we assess the performance characteristics of a verbal sign‐out evaluation tool for internal medicine housestaff and hospitalist attendings, and examine whether it can be used by peers as well as by external evaluators. This tool has previously been found to effectively discriminate between experienced and inexperienced nurses conducting nursing handoffs.[13]

METHODS

Tool Design and Measures

The Handoff CEX (clinical evaluation exercise) is a structured assessment based on the format of the mini‐CEX, an instrument used to assess the quality of history and physical examination by trainees for which validation studies have previously been conducted.[14, 15, 16, 17] We developed the tool based on themes we identified from our own expertise,[1, 5, 6, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29] the ACGME core competencies for trainees,[2] and the literature to maximize content validity. First, standardization has numerous demonstrable benefits for safety in general and handoffs in particular.[30, 31, 32] Consequently we created a domain for organization in which standardization was a characteristic of high performance.

Second, there is evidence that people engaged in conversation routinely overestimate peer comprehension,[27] and that explicit strategies to combat this overestimation, such as confirming understanding, explicitly assigning tasks rather than using open‐ended language, and using concrete language, are effective.[33] Accordingly we created a domain for communication skills, which is also an ACGME competency.

Third, although there were no formal guidelines for sign‐out content when we developed this tool, our own research had demonstrated that the content elements most often missing and felt to be important by stakeholders were related to clinical condition and explicating thinking processes,[5, 6] so we created a domain for content that highlighted these areas and met the ACGME competency of medical knowledge. In accordance with standards for evaluation of learners, we incorporated a domain for judgment to identify where trainees were in the RIME spectrum of reporter, interpreter, master, and educator.

Next, we added a section for professionalism in accordance with the ACGME core competencies of professionalism and patient care.[34] To avoid the disinclination of peers to label each other unprofessional, we labeled the professionalism domain as patient‐focused on the tool.

Finally, we included a domain for setting because of an extensive literature demonstrating increased handoff failures in noisy or interruptive settings.[35, 36, 37] We then revised the tool slightly based on our experiences among nurses and students.[13, 38] The final tool included the 6 domains described above and an assessment of overall competency. Each domain was scored on a 9‐point scale and included descriptive anchors at high and low ends of performance. We further divided the scale into 3 main sections: unsatisfactory (score 13), satisfactory (46), and superior (79). We designed 2 tools, 1 to assess the person providing the handoff and 1 to assess the handoff recipient, each with its own descriptive anchors. The recipient tool did not include a content domain (see Supporting Information, Appendix 1, in the online version of this article).

Setting and Subjects

We tested the tool in 2 different urban academic medical centers: the University of Chicago Medicine (UCM) and Yale‐New Haven Hospital (Yale). At UCM, we tested the tool among hospitalists, nurse practitioners, and physician assistants during the Monday and Tuesday morning and Friday evening sign‐out sessions. At Yale, we tested the tool among housestaff during the evening sign‐out session from the primary team to the on‐call covering team.

The UCM is a 550‐bed urban academic medical center in which the nonteaching hospitalist service cares for patients with liver disease, or end‐stage renal or lung disease awaiting transplant, and a small fraction of general medicine and oncology patients when the housestaff service exceeds its cap. No formal training on sign‐out is provided to attending or midlevel providers. The nonteaching hospitalist service operates as a separate service from the housestaff service and consists of 38 hospitalist clinicians (hospitalist attendings, nurse practitioners, and physicians assistants). There are 2 handoffs each day. In the morning the departing night hospitalist hands off to the incoming daytime hospitalist or midlevel provider. These handoffs occur at 7:30 am in a dedicated room. In the evening the daytime hospitalist or midlevel provider hands off to an incoming night hospitalist. This handoff occurs at 5:30 pm or 7:30 pm in a dedicated location. The written sign‐out is maintained on a Microsoft Word (Microsoft Corp., Redmond, WA) document on a password‐protected server and updated daily.

Yale is a 946‐bed urban academic medical center with a large internal medicine training program. Formal sign‐out education that covers the main domains of the tool is provided to new interns during the first 3 months of the year,[19] and a templated electronic medical record‐based electronic written handoff report is produced by the housestaff for all patients.[22] Approximately half of inpatient medicine patients are cared for by housestaff teams, which are entirely separate from the hospitalist service. Housestaff sign‐out occurs between 4 pm and 7 pm every night. At a minimum, the departing intern signs out to the incoming intern; this handoff is typically supervised by at least 1 second‐ or third‐year resident. All patients are signed out verbally; in addition, the written handoff report is provided to the incoming team. Most handoffs occur in a quiet charting room.

Data Collection

Data collection at UCM occurred between March and December 2010 on 3 days of each week: Mondays, Tuesdays, and Fridays. On Mondays and Tuesdays the morning handoffs were observed; on Fridays the evening handoffs were observed. Data collection at Yale occurred between March and May 2011. Only evening handoffs from the primary team to the overnight coverage were observed. At both sites, participants provided verbal informed consent prior to data collection. At the time of an eligible sign‐out session, a research assistant (D.R. at Yale, P.S. at UCM) provided the evaluation tools to all members of the incoming and outgoing teams, and observed the sign‐out session himself. Each person providing a handoff was asked to evaluate the recipient of the handoff; each person receiving a handoff was asked to evaluate the provider of the handoff. In addition, the trained third‐party observer (D.R., P.S.) evaluated both the provider and recipient of the handoff. The external evaluators were trained in principles of effective communication and the use of the tool, with specific review of anchors at each end of each domain. One evaluator had a DO degree and was completing an MPH degree. The second evaluator was an experienced clinical research assistant whose training consisted of supervised observation of 10 handoffs by a physician investigator. At Yale, if a resident was present, she or he was also asked to evaluate both the provider and recipient of the handoff. Consequently, every sign‐out session included at least 2 evaluations of each participant, 1 by a peer evaluator and 1 by a consistent external evaluator who did not know the patients. At Yale, many sign‐outs also included a third evaluation by a resident supervisor.

The study was approved by the institutional review boards at both UCM and Yale.

Statistical Analysis

We obtained mean, median, and interquartile range of scores for each subdomain of the tool as well as the overall assessment of handoff quality. We assessed convergent construct validity by assessing performance of the tool in different contexts. To do so, we determined whether scores differed by type of participant (provider or recipient), by site, by training level of evaluatee, or by type of evaluator (external, resident supervisor, or peer) by using Wilcoxon rank sum tests and Kruskal‐Wallis tests. For the assessment of differences in ratings by training level, we used evaluations of sign‐out providers only, because the 2 sites differed in scores for recipients. We also assessed construct validity by using Spearman rank correlation coefficients to describe the internal consistency of the tool in terms of the correlation between domains of the tool, and we conducted an exploratory factor analysis to gain insight into whether the subdomains of the tool were measuring the same construct. In conducting this analysis, we restricted the dataset to evaluations of sign‐out providers only, and used a principal components estimation method, a promax rotation, and squared multiple correlation communality priors. Finally, we conducted some preliminary studies of reliability by testing whether different types of evaluators provided similar assessments. We calculated a weighted kappa using Fleiss‐Cohen weights for external versus peer scores and again for supervising resident versus peer scores (Yale only). We were not able to assess test‐retest reliability by nature of the sign‐out process. Statistical significance was defined by a P value 0.05, and analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).

RESULTS

A total of 149 handoff sessions were observed: 89 at UCM and 60 at Yale. Each site conducted a similar total number of evaluations: 336 at UCM, 337 at Yale. These sessions involved 97 unique individuals, 34 at UCM and 63 at Yale. Overall scores were high at both sites, but a wide range of scores was applied (Table 1).

Median, Mean, and Range of Handoff CEX Scores in Each Domain, Providers, and Recipients
DomainProvider, N=343Recipient, N=330P Value
Median (IQR)Mean (SD)RangeMedian (IQR)Mean (SD)Range
  • NOTE: Abbreviations: IQR, interquartile range; SD, standard deviation.

Setting7 (69)7.0 (1.7)297 (69)7.3 (1.6)290.05
Organization7 (68)7.2 (1.5)298 (69)7.4 (1.4)290.07
Communication7 (69)7.2 (1.6)198 (79)7.4 (1.5)290.22
Content7 (68)7.0 (1.6)29    
Judgment8 (68)7.3 (1.4)398 (79)7.5 (1.4)390.06
Professionalism8 (79)7.4 (1.5)298 (79)7.6 (1.4)390.23
Overall7 (68)7.1 (1.5)297 (68)7.4 (1.4)290.02

Handoff Providers

A total of 343 evaluations of handoff providers were completed regarding 67 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism (median: 8; interquartile range [IQR]: 79). The lowest rated domain was content (median: 7; IQR: 68) (Table 1).

Handoff Recipients

A total of 330 evaluations of handoff recipients were completed regarding 58 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism, with a median of 8 (IQR: 79). The lowest rated domain was setting, with a median score of 7 (IQR: 6‐9) (Table 1).

Validity Testing

Comparing provider scores to recipient scores, recipients received significantly higher scores for overall assessment (Table 1). Scores at UCM and Yale were similar in all domains for providers but were slightly lower at UCM in several domains for recipients (see Supporting Information, Appendix 2, in the online version of this article). Scores did not differ significantly by training level (Table 2). Third‐party external evaluators consistently gave lower marks for the same handoff than peer evaluators did (Table 3).

Handoff CEX Scores by Training Level, Providers Only
DomainMedian (Range)P Value
NP/PA, N=33Subintern or Intern, N=170Resident, N=44Hospitalist, N=95
  • NOTE: Abbreviations: NP/PA: nurse practitioner/physician assistant.

Setting7 (29)7 (39)7 (49)7 (29)0.89
Organization8 (49)7 (29)7 (49)8 (39)0.11
Communication8 (49)7 (29)7 (49)8 (19)0.72
Content7 (39)7 (29)7 (49)7 (29)0.92
Judgment8 (59)7 (39)8 (49)8 (49)0.09
Professionalism8 (49)7 (29)8 (39)8 (49)0.82
Overall7 (39)7 (29)8 (49)7 (29)0.28
Handoff CEX Scores by Peer Versus External Evaluators
 Provider, Median (Range)Recipient, Median (Range)
DomainPeer, N=152Resident, Supervisor, N=43External, N=147P ValuePeer, N=145Resident Supervisor, N=43External, N=142P Value
  • NOTE: Abbreviations: N/A, not applicable.

Setting8 (39)7 (39)7 (29)0.028 (29)7 (39)7 (29)<0.001
Organization8 (39)8 (39)7 (29)0.188 (39)8 (69)7 (29)<0.001
Communication8 (39)8 (39)7 (19)<0.0018 (39)8 (49)7 (29)<0.001
Content8 (39)8 (29)7 (29)<0.001N/AN/AN/AN/A
Judgment8 (49)8 (39)7 (39)<0.0018 (39)8 (49)7 (39)<0.001
Professionalism8 (39)8 (59)7 (29)0.028 (39)8 (69)7 (39)<0.001
Overall8 (39)8 (39)7 (29)0.0018 (29)8 (49)7 (29)<0.001

Spearman rank correlation coefficients among the CEX subdomains for provider scores ranged from 0.71 to 0.86, except for setting (Table 4). Setting was less well correlated with the other subdomains, with correlation coefficients ranging from 0.39 to 0.41. Correlations between individual domains and the overall rating ranged from 0.80 to 0.86, except setting, which had a correlation of 0.55. Every correlation was significant at P<0.001. Correlation coefficients for recipient scores were very similar to those for provider scores (see Supporting Information, Appendix 3, in the online version of this article).

Spearman Correlation Coefficients, Provider Evaluations (N=342)
 Spearman Correlation Coefficients
 SettingOrganizationCommunicationContentJudgmentProfessionalism
  • NOTE: All P values <0.0001.

Setting1.0000.400.400.390.390.41
Organization0.401.000.800.710.770.73
Communication0.400.801.000.790.820.77
Content0.390.710.791.000.800.74
Judgment0.390.770.820.801.000.78
Professionalism0.410.730.770.740.781.00
Overall0.550.800.840.830.860.82

We analyzed 343 provider evaluations in the factor analysis; there were 6 missing values. The scree plot of eigenvalues did not support more than 1 factor; however, the rotated factor pattern for standardized regression coefficients for the first factor and the final communality estimates showed the setting component yielding smaller values than did other scale components (see Supporting Information, Appendix 4, in the online version of this article).

Reliability Testing

Weighted kappa scores for provider evaluations ranged from 0.28 (95% confidence interval [CI]: 0.01, 0.56) for setting to 0.59 (95% CI: 0.38, 0.80) for organization, and were generally higher for resident versus peer comparisons than for external versus peer comparisons. Weighted kappa scores for recipient evaluation were slightly lower for external versus peer evaluations, but agreement was no better than chance for resident versus peer evaluations (Table 5).

Weighted Kappa Scores
DomainProviderRecipient
External vs Peer, N=144 (95% CI)Resident vs Peer, N=42 (95% CI)External vs Peer, N=134 (95% CI)Resident vs Peer, N=43 (95% CI)
  • NOTE: Abbreviations: CI, confidence interval; N/A, not applicable.

Setting0.39 (0.24, 0.54)0.28 (0.01, 0.56)0.34 (0.20, 0.48)0.48 (0.27, 0.69)
Organization0.43 (0.29, 0.58)0.59 (0.39, 0.80)0.39 (0.22, 0.55)0.03 (0.23, 0.29)
Communication0.34 (0.19, 0.49)0.52 (0.37, 0.68)0.36 (0.22, 0.51)0.02 (0.18, 0.23)
Content0.38 (0.25, 0.51)0.53 (0.27, 0.80)N/A (N/A)N/A (N/A)
Judgment0.36 (0.22, 0.49)0.54 (0.25, 0.83)0.28 (0.15, 0.42)0.12 (0.34, 0.09)
Professionalism0.47 (0.32, 0.63)0.47 (0.23, 0.72)0.35 (0.18, 0.51)0.01 (0.29, 0.26)
Overall0.50 (0.36, 0.64)0.45 (0.24, 0.67)0.31 (0.16, 0.48)0.07 (0.20, 0.34)

DISCUSSION

In this study we found that an evaluation tool for direct observation of housestaff and hospitalists generated a range of scores and was well validated in the sense of performing similarly across 2 different institutions and among both trainees and attendings, while having high internal consistency. However, external evaluators gave consistently lower marks than peer evaluators at both sites, resulting in low reliability when comparing these 2 groups of raters.

It has traditionally been difficult to conduct direct evaluations of handoffs, because they may occur at haphazard times, in variable locations, and without very much advance notice. For this reason, several attempts have been made to incorporate peers in evaluations of handoff practices.[5, 39, 40] Using peers to conduct evaluations also has the advantage that peers are more likely to be familiar with the patients being handed off and might recognize handoff flaws that external evaluators would miss. Nonetheless, peer evaluations have some important liabilities. Peers may be unwilling or unable to provide honest critiques of their colleagues given that they must work closely together for years. Trainee peers may also lack sufficient clinical expertise or experience to accurately assess competence. In our study, we found that peers gave consistently higher marks to their colleagues than did external evaluators, suggesting they may have found it difficult to criticize their colleagues. We conclude that peer evaluation alone is likely an insufficient means of evaluating handoff quality.

Supervising residents gave very similar marks as intern peers, suggesting that they also are unwilling to criticize, are insufficiently experienced to evaluate, or alternatively, that the peer evaluations were reasonable. We suspect the latter is unlikely given that external evaluator scores were consistently lower than peers. One would expect the external evaluators to be biased toward higher scores given that they are not familiar with the patients and are not able to comment on inaccuracies or omissions in the sign‐out.

The tool appeared to perform less well in most cases for recipients than for providers, with a narrower range of scores and low‐weighted kappa scores. Although recipients play a key role in ensuring a high‐quality sign‐out by paying close attention, ensuring it is a bidirectional conversation, asking appropriate questions, and reading back key information, it may be that evaluators were unable to place these activities within the same domains that were used for the provider evaluation. An altogether different recipient evaluation approach may be necessary.[41]

In general, scores were clustered at the top of the score range, as is typical for evaluations. One strategy to spread out scores further would be to refine the tool by adding anchors for satisfactory performance not just the extremes. A second approach might be to reduce the grading scale to only 3 points (unsatisfactory, satisfactory, superior) to force more scores to the middle. However, this approach might limit the discrimination ability of the tool.

We have previously studied the use of this tool among nurses. In that study, we also found consistently higher scores by peers than by external evaluators. We did, however, find a positive effect of experience, in which more experienced nurses received higher scores on average. We did not observe a similar training effect in this study. There are several possible explanations for the lack of a training effect. It is possible that the types of handoffs assessed played a role. At UCM, some assessed handoffs were night staff to day staff, which might be lower quality than day staff to night staff handoffs, whereas at Yale, all handoffs were day to night teams. Thus, average scores at UCM (primarily hospitalists) might have been lowered by the type of handoff provided. Given that hospitalist evaluations were conducted exclusively at UCM and housestaff evaluations exclusively at Yale, lack of difference between hospitalists and housestaff may also have been related to differences in evaluation practice or handoff practice at the 2 sites, not necessarily related to training level. Third, in our experience, attending physicians provide briefer less‐comprehensive sign‐outs than trainees, particularly when communicating with equally experienced attendings; these sign‐outs may appropriately be scored lower on the tool. Fourth, the great majority of the hospitalists at UCM were within 5 years of residency and therefore not very much more experienced than the trainees. Finally, it is possible that skills do not improve over time given widespread lack of observation and feedback during training years for this important skill.

The high internal consistency of most of the subdomains and the loading of all subdomains except setting onto 1 factor are evidence of convergent construct validity, but also suggest that evaluators have difficulty distinguishing among components of sign‐out quality. Internal consistency may also reflect a halo effect, in which scores on different domains are all influenced by a common overall judgment.[42] We are currently testing a shorter version of the tool including domains only for content, professionalism, and setting in addition to overall score. The fact that setting did not correlate as well with the other domains suggests that sign‐out practitioners may not have or exercise control over their surroundings. Consequently, it may ultimately be reasonable to drop this domain from the tool, or alternatively, to refocus on the need to ensure a quiet setting during sign‐out skills training.

There are several limitations to this study. External evaluations were conducted by personnel who were not familiar with the patients, and they may therefore have overestimated the quality of sign‐out. Studying different types of physicians at different sites might have limited our ability to identify differences by training level. As is commonly seen in evaluation studies, scores were skewed to the high end, although we did observe some use of the full range of the tool. Finally, we were limited in our ability to test inter‐rater reliability because of the multiple sources of variability in the data (numerous different raters, with different backgrounds at different settings, rating different individuals).

In summary, we developed a handoff evaluation tool that was easily completed by housestaff and attendings without training, that performed similarly in a variety of different settings at 2 institutions, and that can in principle be used either for peer evaluations or for external evaluations, although peer evaluations may be positively biased. Further work will be done to refine and simplify the tool.

ACKNOWLEDGMENTS

Disclosures: Development and evaluation of the sign‐out CEX was supported by a grant from the Agency for Healthcare Research and Quality (1R03HS018278‐01). Dr. Arora is supported by a National Institute on Aging (K23 AG033763). Dr. Horwitz is supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. Dr. Horwitz is also a Pepper Scholar with support from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality, the National Institute on Aging, the National Institutes of Health, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as a poster presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 9, 2012. Dr. Rand is now with the Department of Medicine, University of Vermont College of Medicine, Burlington, Vermont. Mr. Staisiunas is now with the Law School, Marquette University, Milwaukee, Wisconsin. The authors declare they have no conflicts of interest.

Appendix

A

PROVIDER HAND‐OFF CEX TOOL

 

 

RECIPIENT HAND‐OFF CEX TOOL

 

 

Appendix

B

 

Handoff CEX scores by site of evaluation

DomainProviderRecipient
Median (Range)P‐valueMedian (Range)P‐value
 UCYale UCYale 
N=172N=170 N=163N=167 
Setting7 (29)7 (39)0.327 (29)7 (39)0.36
Organization8 (29)7 (39)0.307 (29)8 (59)0.001
Communication7 (19)7 (39)0.677 (29)8 (49)0.03
Content7 (29)7 (29) N/AN/AN/A
Judgment8 (39)7 (39)0.607 (39)8 (49)0.001
Professionalism8 (29)8 (39)0.678 (39)8 (49)0.35
Overall7 (29)7 (39)0.417 (29)8 (49)0.005

 

Appendix

C

Spearman correlation, recipients (N=330)

SpearmanCorrelationCoefficients
 SettingOrganizationCommunicationJudgmentProfessionalism
Setting1.00.460.480.470.40
Organization0.461.000.780.750.75
Communication0.480.781.000.850.77
Judgment0.470.750.851.000.74
Professionalism0.400.750.770.741.00
Overall0.600.770.840.820.77

 

All p values <0.0001

 

Appendix

D

Factor analysis results for provider evaluations

Rotated Factor Pattern (Standardized Regression Coefficients) N=336
 Factor1Factor2
Organization0.640.27
Communication0.790.16
Content0.820.06
Judgment0.860.06
Professionalism0.660.23
Setting0.180.29

 

 

Files
References
  1. Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):11731177.
  2. Accreditation Council for Graduate Medical Education. Common program requirements. 2011; http://www.acgme‐2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed August 23, 2011.
  3. Petersen LA, Brennan TA, O'Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866872.
  4. Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186194.
  5. Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. Communication failures in patient sign‐out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401407.
  6. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. Consequences of inadequate sign‐out for patient care. Arch Intern Med. 2008;168(16):17551760.
  7. Borowitz SM, Waggoner‐Fountain LA, Bass EJ, Sledd RM. Adequacy of information transferred at resident sign‐out (in‐hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):610.
  8. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. What are covering doctors told about their patients? Analysis of sign‐out among internal medicine house staff. Qual Saf Health Care. 2009;18(4):248255.
  9. Gakhar B, Spencer AL. Using direct observation, formal evaluation, and an interactive curriculum to improve the sign‐out practices of internal medicine interns. Acad Med. 2010;85(7):11821188.
  10. Raduma‐Tomas MA, Flin R, Yule S, Williams D. Doctors' handovers in hospitals: a literature review. Qual Saf Health Care. 2011;20(2):128133.
  11. Bump GM, Jovin F, Destefano L, et al. Resident sign‐out and patient hand‐offs: opportunities for improvement. Teach Learn Med. 2011;23(2):105111.
  12. Helms AS, Perez TE, Baltz J, et al. Use of an appreciative inquiry approach to improve resident sign‐out in an era of multiple shift changes. J Gen Intern Med. 2012;27(3):287291.
  13. Horwitz LI, Dombroski J, Murphy TE, Farnan JM, Johnson JK, Arora VM. Validation of a handoff assessment tool: the Handoff CEX [published online ahead of print June 7, 2012]. J Clin Nurs. doi: 10.1111/j.1365–2702.2012.04131.x.
  14. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123(10):795799.
  15. Norcini JJ, Blank LL, Arnold GK, Kimball HR. Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):2733.
  16. Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini‐clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900904.
  17. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826830.
  18. Horwitz LI, Meredith T, Schuur JD, Shah NR, Kulkarni RG, Jenq GY. Dropping the baton: a qualitative analysis of failures during the transition from emergency department to inpatient care. Ann Emerg Med. 2009;53(6):701710.e4.
  19. Horwitz LI, Moin T, Green ML. Development and implementation of an oral sign‐out skills curriculum. J Gen Intern Med. 2007;22(10):14701474.
  20. Horwitz LI, Moin T, Wang L, Bradley EH. Mixed methods evaluation of oral sign‐out practices. J Gen Intern Med. 2007;22(S1):S114.
  21. Horwitz LI, Parwani V, Shah NR, et al. Evaluation of an asynchronous physician voicemail sign‐out for emergency department admissions. Ann Emerg Med. 2009;54(3):368378.
  22. Horwitz LI, Schuster KM, Thung SF, et al. An institution‐wide handoff task force to standardise and improve physician handoffs. BMJ Qual Saf. 2012;21(10):863871.
  23. Arora V, Johnson J. A model for building a standardized hand‐off protocol. Jt Comm J Qual Patient Saf. 2006;32(11):646655.
  24. Arora V, Kao J, Lovinger D, Seiden SC, Meltzer D. Medication discrepancies in resident sign‐outs and their potential to harm. J Gen Intern Med. 2007;22(12):17511755.
  25. Arora VM, Johnson JK, Meltzer DO, Humphrey HJ. A theoretical framework and competency‐based approach to improving handoffs. Qual Saf Health Care. 2008;17(1):1114.
  26. Arora VM, Manjarrez E, Dressler DD, Basaviah P, Halasyamani L, Kripalani S. Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med. 2009;4(7):433440.
  27. Chang VY, Arora VM, Lev‐Ari S, D'Arcy M, Keysar B. Interns overestimate the effectiveness of their hand‐off communication. Pediatrics. 2010;125(3):491496.
  28. Johnson JK, Arora VM. Improving clinical handovers: creating local solutions for a global problem. Qual Saf Health Care. 2009;18(4):244245.
  29. Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign‐out. J Hosp Med. 2006;1(4):257266.
  30. Salerno SM, Arnett MV, Domanski JP. Standardized sign‐out reduces intern perception of medical errors on the general internal medicine ward. Teach Learn Med. 2009;21(2):121126.
  31. Haig KM, Sutton S, Whittington J. SBAR: a shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf. 2006;32(3):167175.
  32. Patterson ES. Structuring flexibility: the potential good, bad and ugly in standardisation of handovers. Qual Saf Health Care. 2008;17(1):45.
  33. Patterson ES, Roth EM, Woods DD, Chow R, Gomes JO. Handoff strategies in settings with high consequences for failure: lessons for health care operations. Int J Qual Health Care. 2004;16(2):125132.
  34. Ratanawongsa N, Bolen S, Howell EE, Kern DE, Sisson SD, Larriviere D. Residents' perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758763.
  35. Coiera E, Tombs V. Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673676.
  36. Coiera EW, Jayasuriya RA, Hardy J, Bannan A, Thorpe ME. Communication loads on clinical staff in the emergency department. Med J Aust. 2002;176(9):415418.
  37. Ong MS, Coiera E. A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274284.
  38. Farnan JM, Paro JA, Rodriguez RM, et al. Hand‐off education and evaluation: piloting the observed simulated hand‐off experience (OSHE). J Gen Intern Med. 2010;25(2):129134.
  39. Kitch BT, Cooper JB, Zapol WM, et al. Handoffs causing patient harm: a survey of medical and surgical house staff. Jt Comm J Qual Patient Saf. 2008;34(10):563570.
  40. Li P, Stelfox HT, Ghali WA. A prospective observational study of physician handoff for intensive‐care‐unit‐to‐ward patient transfers. Am J Med. 2011;124(9):860867.
  41. Greenstein E, Arora V, Banerjee S, Staisiunas P, Farnan J. Characterizing physician listening behavior during hospitalist handoffs using the HEAR checklist (published online ahead of print December 20, 2012]. BMJ Qual Saf. doi:10.1136/bmjqs‐2012‐001138.
  42. Thorndike EL. A constant error in psychological ratings. J Appl Psychol. 1920;4(1):25.
Article PDF
Issue
Journal of Hospital Medicine - 8(4)
Publications
Page Number
191-200
Sections
Files
Files
Article PDF
Article PDF

Transfers among trainee physicians within the hospital typically occur at least twice a day and have been increasing among trainees as work hours have declined.[1] The 2011 Accreditation Council for Graduate Medical Education (ACGME) guidelines,[2] which restrict intern working hours to 16 hours from a previous maximum of 30, have likely increased the frequency of physician trainee handoffs even further. Similarly, transfers among hospitalist attendings occur at least twice a day, given typical shifts of 8 to 12 hours.

Given the frequency of transfers, and the potential for harm generated by failed transitions,[3, 4, 5, 6] the end‐of‐shift written and verbal handoffs have assumed increasingly greater importance in hospital care among both trainees and hospitalist attendings.

The ACGME now requires that programs assess the competency of trainees in handoff communication.[2] Yet, there are few tools for assessing the quality of sign‐out communication. Those that exist primarily focus on the written sign‐out, and are rarely validated.[7, 8, 9, 10, 11, 12] Furthermore, it is uncertain whether such assessments must be done by supervisors or whether peers can participate in the evaluation. In this prospective multi‐institutional study we assess the performance characteristics of a verbal sign‐out evaluation tool for internal medicine housestaff and hospitalist attendings, and examine whether it can be used by peers as well as by external evaluators. This tool has previously been found to effectively discriminate between experienced and inexperienced nurses conducting nursing handoffs.[13]

METHODS

Tool Design and Measures

The Handoff CEX (clinical evaluation exercise) is a structured assessment based on the format of the mini‐CEX, an instrument used to assess the quality of history and physical examination by trainees for which validation studies have previously been conducted.[14, 15, 16, 17] We developed the tool based on themes we identified from our own expertise,[1, 5, 6, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29] the ACGME core competencies for trainees,[2] and the literature to maximize content validity. First, standardization has numerous demonstrable benefits for safety in general and handoffs in particular.[30, 31, 32] Consequently we created a domain for organization in which standardization was a characteristic of high performance.

Second, there is evidence that people engaged in conversation routinely overestimate peer comprehension,[27] and that explicit strategies to combat this overestimation, such as confirming understanding, explicitly assigning tasks rather than using open‐ended language, and using concrete language, are effective.[33] Accordingly we created a domain for communication skills, which is also an ACGME competency.

Third, although there were no formal guidelines for sign‐out content when we developed this tool, our own research had demonstrated that the content elements most often missing and felt to be important by stakeholders were related to clinical condition and explicating thinking processes,[5, 6] so we created a domain for content that highlighted these areas and met the ACGME competency of medical knowledge. In accordance with standards for evaluation of learners, we incorporated a domain for judgment to identify where trainees were in the RIME spectrum of reporter, interpreter, master, and educator.

Next, we added a section for professionalism in accordance with the ACGME core competencies of professionalism and patient care.[34] To avoid the disinclination of peers to label each other unprofessional, we labeled the professionalism domain as patient‐focused on the tool.

Finally, we included a domain for setting because of an extensive literature demonstrating increased handoff failures in noisy or interruptive settings.[35, 36, 37] We then revised the tool slightly based on our experiences among nurses and students.[13, 38] The final tool included the 6 domains described above and an assessment of overall competency. Each domain was scored on a 9‐point scale and included descriptive anchors at high and low ends of performance. We further divided the scale into 3 main sections: unsatisfactory (score 13), satisfactory (46), and superior (79). We designed 2 tools, 1 to assess the person providing the handoff and 1 to assess the handoff recipient, each with its own descriptive anchors. The recipient tool did not include a content domain (see Supporting Information, Appendix 1, in the online version of this article).

Setting and Subjects

We tested the tool in 2 different urban academic medical centers: the University of Chicago Medicine (UCM) and Yale‐New Haven Hospital (Yale). At UCM, we tested the tool among hospitalists, nurse practitioners, and physician assistants during the Monday and Tuesday morning and Friday evening sign‐out sessions. At Yale, we tested the tool among housestaff during the evening sign‐out session from the primary team to the on‐call covering team.

The UCM is a 550‐bed urban academic medical center in which the nonteaching hospitalist service cares for patients with liver disease, or end‐stage renal or lung disease awaiting transplant, and a small fraction of general medicine and oncology patients when the housestaff service exceeds its cap. No formal training on sign‐out is provided to attending or midlevel providers. The nonteaching hospitalist service operates as a separate service from the housestaff service and consists of 38 hospitalist clinicians (hospitalist attendings, nurse practitioners, and physicians assistants). There are 2 handoffs each day. In the morning the departing night hospitalist hands off to the incoming daytime hospitalist or midlevel provider. These handoffs occur at 7:30 am in a dedicated room. In the evening the daytime hospitalist or midlevel provider hands off to an incoming night hospitalist. This handoff occurs at 5:30 pm or 7:30 pm in a dedicated location. The written sign‐out is maintained on a Microsoft Word (Microsoft Corp., Redmond, WA) document on a password‐protected server and updated daily.

Yale is a 946‐bed urban academic medical center with a large internal medicine training program. Formal sign‐out education that covers the main domains of the tool is provided to new interns during the first 3 months of the year,[19] and a templated electronic medical record‐based electronic written handoff report is produced by the housestaff for all patients.[22] Approximately half of inpatient medicine patients are cared for by housestaff teams, which are entirely separate from the hospitalist service. Housestaff sign‐out occurs between 4 pm and 7 pm every night. At a minimum, the departing intern signs out to the incoming intern; this handoff is typically supervised by at least 1 second‐ or third‐year resident. All patients are signed out verbally; in addition, the written handoff report is provided to the incoming team. Most handoffs occur in a quiet charting room.

Data Collection

Data collection at UCM occurred between March and December 2010 on 3 days of each week: Mondays, Tuesdays, and Fridays. On Mondays and Tuesdays the morning handoffs were observed; on Fridays the evening handoffs were observed. Data collection at Yale occurred between March and May 2011. Only evening handoffs from the primary team to the overnight coverage were observed. At both sites, participants provided verbal informed consent prior to data collection. At the time of an eligible sign‐out session, a research assistant (D.R. at Yale, P.S. at UCM) provided the evaluation tools to all members of the incoming and outgoing teams, and observed the sign‐out session himself. Each person providing a handoff was asked to evaluate the recipient of the handoff; each person receiving a handoff was asked to evaluate the provider of the handoff. In addition, the trained third‐party observer (D.R., P.S.) evaluated both the provider and recipient of the handoff. The external evaluators were trained in principles of effective communication and the use of the tool, with specific review of anchors at each end of each domain. One evaluator had a DO degree and was completing an MPH degree. The second evaluator was an experienced clinical research assistant whose training consisted of supervised observation of 10 handoffs by a physician investigator. At Yale, if a resident was present, she or he was also asked to evaluate both the provider and recipient of the handoff. Consequently, every sign‐out session included at least 2 evaluations of each participant, 1 by a peer evaluator and 1 by a consistent external evaluator who did not know the patients. At Yale, many sign‐outs also included a third evaluation by a resident supervisor.

The study was approved by the institutional review boards at both UCM and Yale.

Statistical Analysis

We obtained mean, median, and interquartile range of scores for each subdomain of the tool as well as the overall assessment of handoff quality. We assessed convergent construct validity by assessing performance of the tool in different contexts. To do so, we determined whether scores differed by type of participant (provider or recipient), by site, by training level of evaluatee, or by type of evaluator (external, resident supervisor, or peer) by using Wilcoxon rank sum tests and Kruskal‐Wallis tests. For the assessment of differences in ratings by training level, we used evaluations of sign‐out providers only, because the 2 sites differed in scores for recipients. We also assessed construct validity by using Spearman rank correlation coefficients to describe the internal consistency of the tool in terms of the correlation between domains of the tool, and we conducted an exploratory factor analysis to gain insight into whether the subdomains of the tool were measuring the same construct. In conducting this analysis, we restricted the dataset to evaluations of sign‐out providers only, and used a principal components estimation method, a promax rotation, and squared multiple correlation communality priors. Finally, we conducted some preliminary studies of reliability by testing whether different types of evaluators provided similar assessments. We calculated a weighted kappa using Fleiss‐Cohen weights for external versus peer scores and again for supervising resident versus peer scores (Yale only). We were not able to assess test‐retest reliability by nature of the sign‐out process. Statistical significance was defined by a P value 0.05, and analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).

RESULTS

A total of 149 handoff sessions were observed: 89 at UCM and 60 at Yale. Each site conducted a similar total number of evaluations: 336 at UCM, 337 at Yale. These sessions involved 97 unique individuals, 34 at UCM and 63 at Yale. Overall scores were high at both sites, but a wide range of scores was applied (Table 1).

Median, Mean, and Range of Handoff CEX Scores in Each Domain, Providers, and Recipients
DomainProvider, N=343Recipient, N=330P Value
Median (IQR)Mean (SD)RangeMedian (IQR)Mean (SD)Range
  • NOTE: Abbreviations: IQR, interquartile range; SD, standard deviation.

Setting7 (69)7.0 (1.7)297 (69)7.3 (1.6)290.05
Organization7 (68)7.2 (1.5)298 (69)7.4 (1.4)290.07
Communication7 (69)7.2 (1.6)198 (79)7.4 (1.5)290.22
Content7 (68)7.0 (1.6)29    
Judgment8 (68)7.3 (1.4)398 (79)7.5 (1.4)390.06
Professionalism8 (79)7.4 (1.5)298 (79)7.6 (1.4)390.23
Overall7 (68)7.1 (1.5)297 (68)7.4 (1.4)290.02

Handoff Providers

A total of 343 evaluations of handoff providers were completed regarding 67 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism (median: 8; interquartile range [IQR]: 79). The lowest rated domain was content (median: 7; IQR: 68) (Table 1).

Handoff Recipients

A total of 330 evaluations of handoff recipients were completed regarding 58 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism, with a median of 8 (IQR: 79). The lowest rated domain was setting, with a median score of 7 (IQR: 6‐9) (Table 1).

Validity Testing

Comparing provider scores to recipient scores, recipients received significantly higher scores for overall assessment (Table 1). Scores at UCM and Yale were similar in all domains for providers but were slightly lower at UCM in several domains for recipients (see Supporting Information, Appendix 2, in the online version of this article). Scores did not differ significantly by training level (Table 2). Third‐party external evaluators consistently gave lower marks for the same handoff than peer evaluators did (Table 3).

Handoff CEX Scores by Training Level, Providers Only
DomainMedian (Range)P Value
NP/PA, N=33Subintern or Intern, N=170Resident, N=44Hospitalist, N=95
  • NOTE: Abbreviations: NP/PA: nurse practitioner/physician assistant.

Setting7 (29)7 (39)7 (49)7 (29)0.89
Organization8 (49)7 (29)7 (49)8 (39)0.11
Communication8 (49)7 (29)7 (49)8 (19)0.72
Content7 (39)7 (29)7 (49)7 (29)0.92
Judgment8 (59)7 (39)8 (49)8 (49)0.09
Professionalism8 (49)7 (29)8 (39)8 (49)0.82
Overall7 (39)7 (29)8 (49)7 (29)0.28
Handoff CEX Scores by Peer Versus External Evaluators
 Provider, Median (Range)Recipient, Median (Range)
DomainPeer, N=152Resident, Supervisor, N=43External, N=147P ValuePeer, N=145Resident Supervisor, N=43External, N=142P Value
  • NOTE: Abbreviations: N/A, not applicable.

Setting8 (39)7 (39)7 (29)0.028 (29)7 (39)7 (29)<0.001
Organization8 (39)8 (39)7 (29)0.188 (39)8 (69)7 (29)<0.001
Communication8 (39)8 (39)7 (19)<0.0018 (39)8 (49)7 (29)<0.001
Content8 (39)8 (29)7 (29)<0.001N/AN/AN/AN/A
Judgment8 (49)8 (39)7 (39)<0.0018 (39)8 (49)7 (39)<0.001
Professionalism8 (39)8 (59)7 (29)0.028 (39)8 (69)7 (39)<0.001
Overall8 (39)8 (39)7 (29)0.0018 (29)8 (49)7 (29)<0.001

Spearman rank correlation coefficients among the CEX subdomains for provider scores ranged from 0.71 to 0.86, except for setting (Table 4). Setting was less well correlated with the other subdomains, with correlation coefficients ranging from 0.39 to 0.41. Correlations between individual domains and the overall rating ranged from 0.80 to 0.86, except setting, which had a correlation of 0.55. Every correlation was significant at P<0.001. Correlation coefficients for recipient scores were very similar to those for provider scores (see Supporting Information, Appendix 3, in the online version of this article).

Spearman Correlation Coefficients, Provider Evaluations (N=342)
 Spearman Correlation Coefficients
 SettingOrganizationCommunicationContentJudgmentProfessionalism
  • NOTE: All P values <0.0001.

Setting1.0000.400.400.390.390.41
Organization0.401.000.800.710.770.73
Communication0.400.801.000.790.820.77
Content0.390.710.791.000.800.74
Judgment0.390.770.820.801.000.78
Professionalism0.410.730.770.740.781.00
Overall0.550.800.840.830.860.82

We analyzed 343 provider evaluations in the factor analysis; there were 6 missing values. The scree plot of eigenvalues did not support more than 1 factor; however, the rotated factor pattern for standardized regression coefficients for the first factor and the final communality estimates showed the setting component yielding smaller values than did other scale components (see Supporting Information, Appendix 4, in the online version of this article).

Reliability Testing

Weighted kappa scores for provider evaluations ranged from 0.28 (95% confidence interval [CI]: 0.01, 0.56) for setting to 0.59 (95% CI: 0.38, 0.80) for organization, and were generally higher for resident versus peer comparisons than for external versus peer comparisons. Weighted kappa scores for recipient evaluation were slightly lower for external versus peer evaluations, but agreement was no better than chance for resident versus peer evaluations (Table 5).

Weighted Kappa Scores
DomainProviderRecipient
External vs Peer, N=144 (95% CI)Resident vs Peer, N=42 (95% CI)External vs Peer, N=134 (95% CI)Resident vs Peer, N=43 (95% CI)
  • NOTE: Abbreviations: CI, confidence interval; N/A, not applicable.

Setting0.39 (0.24, 0.54)0.28 (0.01, 0.56)0.34 (0.20, 0.48)0.48 (0.27, 0.69)
Organization0.43 (0.29, 0.58)0.59 (0.39, 0.80)0.39 (0.22, 0.55)0.03 (0.23, 0.29)
Communication0.34 (0.19, 0.49)0.52 (0.37, 0.68)0.36 (0.22, 0.51)0.02 (0.18, 0.23)
Content0.38 (0.25, 0.51)0.53 (0.27, 0.80)N/A (N/A)N/A (N/A)
Judgment0.36 (0.22, 0.49)0.54 (0.25, 0.83)0.28 (0.15, 0.42)0.12 (0.34, 0.09)
Professionalism0.47 (0.32, 0.63)0.47 (0.23, 0.72)0.35 (0.18, 0.51)0.01 (0.29, 0.26)
Overall0.50 (0.36, 0.64)0.45 (0.24, 0.67)0.31 (0.16, 0.48)0.07 (0.20, 0.34)

DISCUSSION

In this study we found that an evaluation tool for direct observation of housestaff and hospitalists generated a range of scores and was well validated in the sense of performing similarly across 2 different institutions and among both trainees and attendings, while having high internal consistency. However, external evaluators gave consistently lower marks than peer evaluators at both sites, resulting in low reliability when comparing these 2 groups of raters.

It has traditionally been difficult to conduct direct evaluations of handoffs, because they may occur at haphazard times, in variable locations, and without very much advance notice. For this reason, several attempts have been made to incorporate peers in evaluations of handoff practices.[5, 39, 40] Using peers to conduct evaluations also has the advantage that peers are more likely to be familiar with the patients being handed off and might recognize handoff flaws that external evaluators would miss. Nonetheless, peer evaluations have some important liabilities. Peers may be unwilling or unable to provide honest critiques of their colleagues given that they must work closely together for years. Trainee peers may also lack sufficient clinical expertise or experience to accurately assess competence. In our study, we found that peers gave consistently higher marks to their colleagues than did external evaluators, suggesting they may have found it difficult to criticize their colleagues. We conclude that peer evaluation alone is likely an insufficient means of evaluating handoff quality.

Supervising residents gave very similar marks as intern peers, suggesting that they also are unwilling to criticize, are insufficiently experienced to evaluate, or alternatively, that the peer evaluations were reasonable. We suspect the latter is unlikely given that external evaluator scores were consistently lower than peers. One would expect the external evaluators to be biased toward higher scores given that they are not familiar with the patients and are not able to comment on inaccuracies or omissions in the sign‐out.

The tool appeared to perform less well in most cases for recipients than for providers, with a narrower range of scores and low‐weighted kappa scores. Although recipients play a key role in ensuring a high‐quality sign‐out by paying close attention, ensuring it is a bidirectional conversation, asking appropriate questions, and reading back key information, it may be that evaluators were unable to place these activities within the same domains that were used for the provider evaluation. An altogether different recipient evaluation approach may be necessary.[41]

In general, scores were clustered at the top of the score range, as is typical for evaluations. One strategy to spread out scores further would be to refine the tool by adding anchors for satisfactory performance not just the extremes. A second approach might be to reduce the grading scale to only 3 points (unsatisfactory, satisfactory, superior) to force more scores to the middle. However, this approach might limit the discrimination ability of the tool.

We have previously studied the use of this tool among nurses. In that study, we also found consistently higher scores by peers than by external evaluators. We did, however, find a positive effect of experience, in which more experienced nurses received higher scores on average. We did not observe a similar training effect in this study. There are several possible explanations for the lack of a training effect. It is possible that the types of handoffs assessed played a role. At UCM, some assessed handoffs were night staff to day staff, which might be lower quality than day staff to night staff handoffs, whereas at Yale, all handoffs were day to night teams. Thus, average scores at UCM (primarily hospitalists) might have been lowered by the type of handoff provided. Given that hospitalist evaluations were conducted exclusively at UCM and housestaff evaluations exclusively at Yale, lack of difference between hospitalists and housestaff may also have been related to differences in evaluation practice or handoff practice at the 2 sites, not necessarily related to training level. Third, in our experience, attending physicians provide briefer less‐comprehensive sign‐outs than trainees, particularly when communicating with equally experienced attendings; these sign‐outs may appropriately be scored lower on the tool. Fourth, the great majority of the hospitalists at UCM were within 5 years of residency and therefore not very much more experienced than the trainees. Finally, it is possible that skills do not improve over time given widespread lack of observation and feedback during training years for this important skill.

The high internal consistency of most of the subdomains and the loading of all subdomains except setting onto 1 factor are evidence of convergent construct validity, but also suggest that evaluators have difficulty distinguishing among components of sign‐out quality. Internal consistency may also reflect a halo effect, in which scores on different domains are all influenced by a common overall judgment.[42] We are currently testing a shorter version of the tool including domains only for content, professionalism, and setting in addition to overall score. The fact that setting did not correlate as well with the other domains suggests that sign‐out practitioners may not have or exercise control over their surroundings. Consequently, it may ultimately be reasonable to drop this domain from the tool, or alternatively, to refocus on the need to ensure a quiet setting during sign‐out skills training.

There are several limitations to this study. External evaluations were conducted by personnel who were not familiar with the patients, and they may therefore have overestimated the quality of sign‐out. Studying different types of physicians at different sites might have limited our ability to identify differences by training level. As is commonly seen in evaluation studies, scores were skewed to the high end, although we did observe some use of the full range of the tool. Finally, we were limited in our ability to test inter‐rater reliability because of the multiple sources of variability in the data (numerous different raters, with different backgrounds at different settings, rating different individuals).

In summary, we developed a handoff evaluation tool that was easily completed by housestaff and attendings without training, that performed similarly in a variety of different settings at 2 institutions, and that can in principle be used either for peer evaluations or for external evaluations, although peer evaluations may be positively biased. Further work will be done to refine and simplify the tool.

ACKNOWLEDGMENTS

Disclosures: Development and evaluation of the sign‐out CEX was supported by a grant from the Agency for Healthcare Research and Quality (1R03HS018278‐01). Dr. Arora is supported by a National Institute on Aging (K23 AG033763). Dr. Horwitz is supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. Dr. Horwitz is also a Pepper Scholar with support from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality, the National Institute on Aging, the National Institutes of Health, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as a poster presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 9, 2012. Dr. Rand is now with the Department of Medicine, University of Vermont College of Medicine, Burlington, Vermont. Mr. Staisiunas is now with the Law School, Marquette University, Milwaukee, Wisconsin. The authors declare they have no conflicts of interest.

Appendix

A

PROVIDER HAND‐OFF CEX TOOL

 

 

RECIPIENT HAND‐OFF CEX TOOL

 

 

Appendix

B

 

Handoff CEX scores by site of evaluation

DomainProviderRecipient
Median (Range)P‐valueMedian (Range)P‐value
 UCYale UCYale 
N=172N=170 N=163N=167 
Setting7 (29)7 (39)0.327 (29)7 (39)0.36
Organization8 (29)7 (39)0.307 (29)8 (59)0.001
Communication7 (19)7 (39)0.677 (29)8 (49)0.03
Content7 (29)7 (29) N/AN/AN/A
Judgment8 (39)7 (39)0.607 (39)8 (49)0.001
Professionalism8 (29)8 (39)0.678 (39)8 (49)0.35
Overall7 (29)7 (39)0.417 (29)8 (49)0.005

 

Appendix

C

Spearman correlation, recipients (N=330)

SpearmanCorrelationCoefficients
 SettingOrganizationCommunicationJudgmentProfessionalism
Setting1.00.460.480.470.40
Organization0.461.000.780.750.75
Communication0.480.781.000.850.77
Judgment0.470.750.851.000.74
Professionalism0.400.750.770.741.00
Overall0.600.770.840.820.77

 

All p values <0.0001

 

Appendix

D

Factor analysis results for provider evaluations

Rotated Factor Pattern (Standardized Regression Coefficients) N=336
 Factor1Factor2
Organization0.640.27
Communication0.790.16
Content0.820.06
Judgment0.860.06
Professionalism0.660.23
Setting0.180.29

 

 

Transfers among trainee physicians within the hospital typically occur at least twice a day and have been increasing among trainees as work hours have declined.[1] The 2011 Accreditation Council for Graduate Medical Education (ACGME) guidelines,[2] which restrict intern working hours to 16 hours from a previous maximum of 30, have likely increased the frequency of physician trainee handoffs even further. Similarly, transfers among hospitalist attendings occur at least twice a day, given typical shifts of 8 to 12 hours.

Given the frequency of transfers, and the potential for harm generated by failed transitions,[3, 4, 5, 6] the end‐of‐shift written and verbal handoffs have assumed increasingly greater importance in hospital care among both trainees and hospitalist attendings.

The ACGME now requires that programs assess the competency of trainees in handoff communication.[2] Yet, there are few tools for assessing the quality of sign‐out communication. Those that exist primarily focus on the written sign‐out, and are rarely validated.[7, 8, 9, 10, 11, 12] Furthermore, it is uncertain whether such assessments must be done by supervisors or whether peers can participate in the evaluation. In this prospective multi‐institutional study we assess the performance characteristics of a verbal sign‐out evaluation tool for internal medicine housestaff and hospitalist attendings, and examine whether it can be used by peers as well as by external evaluators. This tool has previously been found to effectively discriminate between experienced and inexperienced nurses conducting nursing handoffs.[13]

METHODS

Tool Design and Measures

The Handoff CEX (clinical evaluation exercise) is a structured assessment based on the format of the mini‐CEX, an instrument used to assess the quality of history and physical examination by trainees for which validation studies have previously been conducted.[14, 15, 16, 17] We developed the tool based on themes we identified from our own expertise,[1, 5, 6, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29] the ACGME core competencies for trainees,[2] and the literature to maximize content validity. First, standardization has numerous demonstrable benefits for safety in general and handoffs in particular.[30, 31, 32] Consequently we created a domain for organization in which standardization was a characteristic of high performance.

Second, there is evidence that people engaged in conversation routinely overestimate peer comprehension,[27] and that explicit strategies to combat this overestimation, such as confirming understanding, explicitly assigning tasks rather than using open‐ended language, and using concrete language, are effective.[33] Accordingly we created a domain for communication skills, which is also an ACGME competency.

Third, although there were no formal guidelines for sign‐out content when we developed this tool, our own research had demonstrated that the content elements most often missing and felt to be important by stakeholders were related to clinical condition and explicating thinking processes,[5, 6] so we created a domain for content that highlighted these areas and met the ACGME competency of medical knowledge. In accordance with standards for evaluation of learners, we incorporated a domain for judgment to identify where trainees were in the RIME spectrum of reporter, interpreter, master, and educator.

Next, we added a section for professionalism in accordance with the ACGME core competencies of professionalism and patient care.[34] To avoid the disinclination of peers to label each other unprofessional, we labeled the professionalism domain as patient‐focused on the tool.

Finally, we included a domain for setting because of an extensive literature demonstrating increased handoff failures in noisy or interruptive settings.[35, 36, 37] We then revised the tool slightly based on our experiences among nurses and students.[13, 38] The final tool included the 6 domains described above and an assessment of overall competency. Each domain was scored on a 9‐point scale and included descriptive anchors at high and low ends of performance. We further divided the scale into 3 main sections: unsatisfactory (score 13), satisfactory (46), and superior (79). We designed 2 tools, 1 to assess the person providing the handoff and 1 to assess the handoff recipient, each with its own descriptive anchors. The recipient tool did not include a content domain (see Supporting Information, Appendix 1, in the online version of this article).

Setting and Subjects

We tested the tool in 2 different urban academic medical centers: the University of Chicago Medicine (UCM) and Yale‐New Haven Hospital (Yale). At UCM, we tested the tool among hospitalists, nurse practitioners, and physician assistants during the Monday and Tuesday morning and Friday evening sign‐out sessions. At Yale, we tested the tool among housestaff during the evening sign‐out session from the primary team to the on‐call covering team.

The UCM is a 550‐bed urban academic medical center in which the nonteaching hospitalist service cares for patients with liver disease, or end‐stage renal or lung disease awaiting transplant, and a small fraction of general medicine and oncology patients when the housestaff service exceeds its cap. No formal training on sign‐out is provided to attending or midlevel providers. The nonteaching hospitalist service operates as a separate service from the housestaff service and consists of 38 hospitalist clinicians (hospitalist attendings, nurse practitioners, and physicians assistants). There are 2 handoffs each day. In the morning the departing night hospitalist hands off to the incoming daytime hospitalist or midlevel provider. These handoffs occur at 7:30 am in a dedicated room. In the evening the daytime hospitalist or midlevel provider hands off to an incoming night hospitalist. This handoff occurs at 5:30 pm or 7:30 pm in a dedicated location. The written sign‐out is maintained on a Microsoft Word (Microsoft Corp., Redmond, WA) document on a password‐protected server and updated daily.

Yale is a 946‐bed urban academic medical center with a large internal medicine training program. Formal sign‐out education that covers the main domains of the tool is provided to new interns during the first 3 months of the year,[19] and a templated electronic medical record‐based electronic written handoff report is produced by the housestaff for all patients.[22] Approximately half of inpatient medicine patients are cared for by housestaff teams, which are entirely separate from the hospitalist service. Housestaff sign‐out occurs between 4 pm and 7 pm every night. At a minimum, the departing intern signs out to the incoming intern; this handoff is typically supervised by at least 1 second‐ or third‐year resident. All patients are signed out verbally; in addition, the written handoff report is provided to the incoming team. Most handoffs occur in a quiet charting room.

Data Collection

Data collection at UCM occurred between March and December 2010 on 3 days of each week: Mondays, Tuesdays, and Fridays. On Mondays and Tuesdays the morning handoffs were observed; on Fridays the evening handoffs were observed. Data collection at Yale occurred between March and May 2011. Only evening handoffs from the primary team to the overnight coverage were observed. At both sites, participants provided verbal informed consent prior to data collection. At the time of an eligible sign‐out session, a research assistant (D.R. at Yale, P.S. at UCM) provided the evaluation tools to all members of the incoming and outgoing teams, and observed the sign‐out session himself. Each person providing a handoff was asked to evaluate the recipient of the handoff; each person receiving a handoff was asked to evaluate the provider of the handoff. In addition, the trained third‐party observer (D.R., P.S.) evaluated both the provider and recipient of the handoff. The external evaluators were trained in principles of effective communication and the use of the tool, with specific review of anchors at each end of each domain. One evaluator had a DO degree and was completing an MPH degree. The second evaluator was an experienced clinical research assistant whose training consisted of supervised observation of 10 handoffs by a physician investigator. At Yale, if a resident was present, she or he was also asked to evaluate both the provider and recipient of the handoff. Consequently, every sign‐out session included at least 2 evaluations of each participant, 1 by a peer evaluator and 1 by a consistent external evaluator who did not know the patients. At Yale, many sign‐outs also included a third evaluation by a resident supervisor.

The study was approved by the institutional review boards at both UCM and Yale.

Statistical Analysis

We obtained mean, median, and interquartile range of scores for each subdomain of the tool as well as the overall assessment of handoff quality. We assessed convergent construct validity by assessing performance of the tool in different contexts. To do so, we determined whether scores differed by type of participant (provider or recipient), by site, by training level of evaluatee, or by type of evaluator (external, resident supervisor, or peer) by using Wilcoxon rank sum tests and Kruskal‐Wallis tests. For the assessment of differences in ratings by training level, we used evaluations of sign‐out providers only, because the 2 sites differed in scores for recipients. We also assessed construct validity by using Spearman rank correlation coefficients to describe the internal consistency of the tool in terms of the correlation between domains of the tool, and we conducted an exploratory factor analysis to gain insight into whether the subdomains of the tool were measuring the same construct. In conducting this analysis, we restricted the dataset to evaluations of sign‐out providers only, and used a principal components estimation method, a promax rotation, and squared multiple correlation communality priors. Finally, we conducted some preliminary studies of reliability by testing whether different types of evaluators provided similar assessments. We calculated a weighted kappa using Fleiss‐Cohen weights for external versus peer scores and again for supervising resident versus peer scores (Yale only). We were not able to assess test‐retest reliability by nature of the sign‐out process. Statistical significance was defined by a P value 0.05, and analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).

RESULTS

A total of 149 handoff sessions were observed: 89 at UCM and 60 at Yale. Each site conducted a similar total number of evaluations: 336 at UCM, 337 at Yale. These sessions involved 97 unique individuals, 34 at UCM and 63 at Yale. Overall scores were high at both sites, but a wide range of scores was applied (Table 1).

Median, Mean, and Range of Handoff CEX Scores in Each Domain, Providers, and Recipients
DomainProvider, N=343Recipient, N=330P Value
Median (IQR)Mean (SD)RangeMedian (IQR)Mean (SD)Range
  • NOTE: Abbreviations: IQR, interquartile range; SD, standard deviation.

Setting7 (69)7.0 (1.7)297 (69)7.3 (1.6)290.05
Organization7 (68)7.2 (1.5)298 (69)7.4 (1.4)290.07
Communication7 (69)7.2 (1.6)198 (79)7.4 (1.5)290.22
Content7 (68)7.0 (1.6)29    
Judgment8 (68)7.3 (1.4)398 (79)7.5 (1.4)390.06
Professionalism8 (79)7.4 (1.5)298 (79)7.6 (1.4)390.23
Overall7 (68)7.1 (1.5)297 (68)7.4 (1.4)290.02

Handoff Providers

A total of 343 evaluations of handoff providers were completed regarding 67 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism (median: 8; interquartile range [IQR]: 79). The lowest rated domain was content (median: 7; IQR: 68) (Table 1).

Handoff Recipients

A total of 330 evaluations of handoff recipients were completed regarding 58 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism, with a median of 8 (IQR: 79). The lowest rated domain was setting, with a median score of 7 (IQR: 6‐9) (Table 1).

Validity Testing

Comparing provider scores to recipient scores, recipients received significantly higher scores for overall assessment (Table 1). Scores at UCM and Yale were similar in all domains for providers but were slightly lower at UCM in several domains for recipients (see Supporting Information, Appendix 2, in the online version of this article). Scores did not differ significantly by training level (Table 2). Third‐party external evaluators consistently gave lower marks for the same handoff than peer evaluators did (Table 3).

Handoff CEX Scores by Training Level, Providers Only
DomainMedian (Range)P Value
NP/PA, N=33Subintern or Intern, N=170Resident, N=44Hospitalist, N=95
  • NOTE: Abbreviations: NP/PA: nurse practitioner/physician assistant.

Setting7 (29)7 (39)7 (49)7 (29)0.89
Organization8 (49)7 (29)7 (49)8 (39)0.11
Communication8 (49)7 (29)7 (49)8 (19)0.72
Content7 (39)7 (29)7 (49)7 (29)0.92
Judgment8 (59)7 (39)8 (49)8 (49)0.09
Professionalism8 (49)7 (29)8 (39)8 (49)0.82
Overall7 (39)7 (29)8 (49)7 (29)0.28
Handoff CEX Scores by Peer Versus External Evaluators
 Provider, Median (Range)Recipient, Median (Range)
DomainPeer, N=152Resident, Supervisor, N=43External, N=147P ValuePeer, N=145Resident Supervisor, N=43External, N=142P Value
  • NOTE: Abbreviations: N/A, not applicable.

Setting8 (39)7 (39)7 (29)0.028 (29)7 (39)7 (29)<0.001
Organization8 (39)8 (39)7 (29)0.188 (39)8 (69)7 (29)<0.001
Communication8 (39)8 (39)7 (19)<0.0018 (39)8 (49)7 (29)<0.001
Content8 (39)8 (29)7 (29)<0.001N/AN/AN/AN/A
Judgment8 (49)8 (39)7 (39)<0.0018 (39)8 (49)7 (39)<0.001
Professionalism8 (39)8 (59)7 (29)0.028 (39)8 (69)7 (39)<0.001
Overall8 (39)8 (39)7 (29)0.0018 (29)8 (49)7 (29)<0.001

Spearman rank correlation coefficients among the CEX subdomains for provider scores ranged from 0.71 to 0.86, except for setting (Table 4). Setting was less well correlated with the other subdomains, with correlation coefficients ranging from 0.39 to 0.41. Correlations between individual domains and the overall rating ranged from 0.80 to 0.86, except setting, which had a correlation of 0.55. Every correlation was significant at P<0.001. Correlation coefficients for recipient scores were very similar to those for provider scores (see Supporting Information, Appendix 3, in the online version of this article).

Spearman Correlation Coefficients, Provider Evaluations (N=342)
 Spearman Correlation Coefficients
 SettingOrganizationCommunicationContentJudgmentProfessionalism
  • NOTE: All P values <0.0001.

Setting1.0000.400.400.390.390.41
Organization0.401.000.800.710.770.73
Communication0.400.801.000.790.820.77
Content0.390.710.791.000.800.74
Judgment0.390.770.820.801.000.78
Professionalism0.410.730.770.740.781.00
Overall0.550.800.840.830.860.82

We analyzed 343 provider evaluations in the factor analysis; there were 6 missing values. The scree plot of eigenvalues did not support more than 1 factor; however, the rotated factor pattern for standardized regression coefficients for the first factor and the final communality estimates showed the setting component yielding smaller values than did other scale components (see Supporting Information, Appendix 4, in the online version of this article).

Reliability Testing

Weighted kappa scores for provider evaluations ranged from 0.28 (95% confidence interval [CI]: 0.01, 0.56) for setting to 0.59 (95% CI: 0.38, 0.80) for organization, and were generally higher for resident versus peer comparisons than for external versus peer comparisons. Weighted kappa scores for recipient evaluation were slightly lower for external versus peer evaluations, but agreement was no better than chance for resident versus peer evaluations (Table 5).

Weighted Kappa Scores
DomainProviderRecipient
External vs Peer, N=144 (95% CI)Resident vs Peer, N=42 (95% CI)External vs Peer, N=134 (95% CI)Resident vs Peer, N=43 (95% CI)
  • NOTE: Abbreviations: CI, confidence interval; N/A, not applicable.

Setting0.39 (0.24, 0.54)0.28 (0.01, 0.56)0.34 (0.20, 0.48)0.48 (0.27, 0.69)
Organization0.43 (0.29, 0.58)0.59 (0.39, 0.80)0.39 (0.22, 0.55)0.03 (0.23, 0.29)
Communication0.34 (0.19, 0.49)0.52 (0.37, 0.68)0.36 (0.22, 0.51)0.02 (0.18, 0.23)
Content0.38 (0.25, 0.51)0.53 (0.27, 0.80)N/A (N/A)N/A (N/A)
Judgment0.36 (0.22, 0.49)0.54 (0.25, 0.83)0.28 (0.15, 0.42)0.12 (0.34, 0.09)
Professionalism0.47 (0.32, 0.63)0.47 (0.23, 0.72)0.35 (0.18, 0.51)0.01 (0.29, 0.26)
Overall0.50 (0.36, 0.64)0.45 (0.24, 0.67)0.31 (0.16, 0.48)0.07 (0.20, 0.34)

DISCUSSION

In this study we found that an evaluation tool for direct observation of housestaff and hospitalists generated a range of scores and was well validated in the sense of performing similarly across 2 different institutions and among both trainees and attendings, while having high internal consistency. However, external evaluators gave consistently lower marks than peer evaluators at both sites, resulting in low reliability when comparing these 2 groups of raters.

It has traditionally been difficult to conduct direct evaluations of handoffs, because they may occur at haphazard times, in variable locations, and without very much advance notice. For this reason, several attempts have been made to incorporate peers in evaluations of handoff practices.[5, 39, 40] Using peers to conduct evaluations also has the advantage that peers are more likely to be familiar with the patients being handed off and might recognize handoff flaws that external evaluators would miss. Nonetheless, peer evaluations have some important liabilities. Peers may be unwilling or unable to provide honest critiques of their colleagues given that they must work closely together for years. Trainee peers may also lack sufficient clinical expertise or experience to accurately assess competence. In our study, we found that peers gave consistently higher marks to their colleagues than did external evaluators, suggesting they may have found it difficult to criticize their colleagues. We conclude that peer evaluation alone is likely an insufficient means of evaluating handoff quality.

Supervising residents gave very similar marks as intern peers, suggesting that they also are unwilling to criticize, are insufficiently experienced to evaluate, or alternatively, that the peer evaluations were reasonable. We suspect the latter is unlikely given that external evaluator scores were consistently lower than peers. One would expect the external evaluators to be biased toward higher scores given that they are not familiar with the patients and are not able to comment on inaccuracies or omissions in the sign‐out.

The tool appeared to perform less well in most cases for recipients than for providers, with a narrower range of scores and low‐weighted kappa scores. Although recipients play a key role in ensuring a high‐quality sign‐out by paying close attention, ensuring it is a bidirectional conversation, asking appropriate questions, and reading back key information, it may be that evaluators were unable to place these activities within the same domains that were used for the provider evaluation. An altogether different recipient evaluation approach may be necessary.[41]

In general, scores were clustered at the top of the score range, as is typical for evaluations. One strategy to spread out scores further would be to refine the tool by adding anchors for satisfactory performance not just the extremes. A second approach might be to reduce the grading scale to only 3 points (unsatisfactory, satisfactory, superior) to force more scores to the middle. However, this approach might limit the discrimination ability of the tool.

We have previously studied the use of this tool among nurses. In that study, we also found consistently higher scores by peers than by external evaluators. We did, however, find a positive effect of experience, in which more experienced nurses received higher scores on average. We did not observe a similar training effect in this study. There are several possible explanations for the lack of a training effect. It is possible that the types of handoffs assessed played a role. At UCM, some assessed handoffs were night staff to day staff, which might be lower quality than day staff to night staff handoffs, whereas at Yale, all handoffs were day to night teams. Thus, average scores at UCM (primarily hospitalists) might have been lowered by the type of handoff provided. Given that hospitalist evaluations were conducted exclusively at UCM and housestaff evaluations exclusively at Yale, lack of difference between hospitalists and housestaff may also have been related to differences in evaluation practice or handoff practice at the 2 sites, not necessarily related to training level. Third, in our experience, attending physicians provide briefer less‐comprehensive sign‐outs than trainees, particularly when communicating with equally experienced attendings; these sign‐outs may appropriately be scored lower on the tool. Fourth, the great majority of the hospitalists at UCM were within 5 years of residency and therefore not very much more experienced than the trainees. Finally, it is possible that skills do not improve over time given widespread lack of observation and feedback during training years for this important skill.

The high internal consistency of most of the subdomains and the loading of all subdomains except setting onto 1 factor are evidence of convergent construct validity, but also suggest that evaluators have difficulty distinguishing among components of sign‐out quality. Internal consistency may also reflect a halo effect, in which scores on different domains are all influenced by a common overall judgment.[42] We are currently testing a shorter version of the tool including domains only for content, professionalism, and setting in addition to overall score. The fact that setting did not correlate as well with the other domains suggests that sign‐out practitioners may not have or exercise control over their surroundings. Consequently, it may ultimately be reasonable to drop this domain from the tool, or alternatively, to refocus on the need to ensure a quiet setting during sign‐out skills training.

There are several limitations to this study. External evaluations were conducted by personnel who were not familiar with the patients, and they may therefore have overestimated the quality of sign‐out. Studying different types of physicians at different sites might have limited our ability to identify differences by training level. As is commonly seen in evaluation studies, scores were skewed to the high end, although we did observe some use of the full range of the tool. Finally, we were limited in our ability to test inter‐rater reliability because of the multiple sources of variability in the data (numerous different raters, with different backgrounds at different settings, rating different individuals).

In summary, we developed a handoff evaluation tool that was easily completed by housestaff and attendings without training, that performed similarly in a variety of different settings at 2 institutions, and that can in principle be used either for peer evaluations or for external evaluations, although peer evaluations may be positively biased. Further work will be done to refine and simplify the tool.

ACKNOWLEDGMENTS

Disclosures: Development and evaluation of the sign‐out CEX was supported by a grant from the Agency for Healthcare Research and Quality (1R03HS018278‐01). Dr. Arora is supported by a National Institute on Aging (K23 AG033763). Dr. Horwitz is supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. Dr. Horwitz is also a Pepper Scholar with support from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality, the National Institute on Aging, the National Institutes of Health, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as a poster presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 9, 2012. Dr. Rand is now with the Department of Medicine, University of Vermont College of Medicine, Burlington, Vermont. Mr. Staisiunas is now with the Law School, Marquette University, Milwaukee, Wisconsin. The authors declare they have no conflicts of interest.

Appendix

A

PROVIDER HAND‐OFF CEX TOOL

 

 

RECIPIENT HAND‐OFF CEX TOOL

 

 

Appendix

B

 

Handoff CEX scores by site of evaluation

DomainProviderRecipient
Median (Range)P‐valueMedian (Range)P‐value
 UCYale UCYale 
N=172N=170 N=163N=167 
Setting7 (29)7 (39)0.327 (29)7 (39)0.36
Organization8 (29)7 (39)0.307 (29)8 (59)0.001
Communication7 (19)7 (39)0.677 (29)8 (49)0.03
Content7 (29)7 (29) N/AN/AN/A
Judgment8 (39)7 (39)0.607 (39)8 (49)0.001
Professionalism8 (29)8 (39)0.678 (39)8 (49)0.35
Overall7 (29)7 (39)0.417 (29)8 (49)0.005

 

Appendix

C

Spearman correlation, recipients (N=330)

SpearmanCorrelationCoefficients
 SettingOrganizationCommunicationJudgmentProfessionalism
Setting1.00.460.480.470.40
Organization0.461.000.780.750.75
Communication0.480.781.000.850.77
Judgment0.470.750.851.000.74
Professionalism0.400.750.770.741.00
Overall0.600.770.840.820.77

 

All p values <0.0001

 

Appendix

D

Factor analysis results for provider evaluations

Rotated Factor Pattern (Standardized Regression Coefficients) N=336
 Factor1Factor2
Organization0.640.27
Communication0.790.16
Content0.820.06
Judgment0.860.06
Professionalism0.660.23
Setting0.180.29

 

 

References
  1. Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):11731177.
  2. Accreditation Council for Graduate Medical Education. Common program requirements. 2011; http://www.acgme‐2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed August 23, 2011.
  3. Petersen LA, Brennan TA, O'Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866872.
  4. Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186194.
  5. Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. Communication failures in patient sign‐out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401407.
  6. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. Consequences of inadequate sign‐out for patient care. Arch Intern Med. 2008;168(16):17551760.
  7. Borowitz SM, Waggoner‐Fountain LA, Bass EJ, Sledd RM. Adequacy of information transferred at resident sign‐out (in‐hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):610.
  8. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. What are covering doctors told about their patients? Analysis of sign‐out among internal medicine house staff. Qual Saf Health Care. 2009;18(4):248255.
  9. Gakhar B, Spencer AL. Using direct observation, formal evaluation, and an interactive curriculum to improve the sign‐out practices of internal medicine interns. Acad Med. 2010;85(7):11821188.
  10. Raduma‐Tomas MA, Flin R, Yule S, Williams D. Doctors' handovers in hospitals: a literature review. Qual Saf Health Care. 2011;20(2):128133.
  11. Bump GM, Jovin F, Destefano L, et al. Resident sign‐out and patient hand‐offs: opportunities for improvement. Teach Learn Med. 2011;23(2):105111.
  12. Helms AS, Perez TE, Baltz J, et al. Use of an appreciative inquiry approach to improve resident sign‐out in an era of multiple shift changes. J Gen Intern Med. 2012;27(3):287291.
  13. Horwitz LI, Dombroski J, Murphy TE, Farnan JM, Johnson JK, Arora VM. Validation of a handoff assessment tool: the Handoff CEX [published online ahead of print June 7, 2012]. J Clin Nurs. doi: 10.1111/j.1365–2702.2012.04131.x.
  14. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123(10):795799.
  15. Norcini JJ, Blank LL, Arnold GK, Kimball HR. Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):2733.
  16. Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini‐clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900904.
  17. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826830.
  18. Horwitz LI, Meredith T, Schuur JD, Shah NR, Kulkarni RG, Jenq GY. Dropping the baton: a qualitative analysis of failures during the transition from emergency department to inpatient care. Ann Emerg Med. 2009;53(6):701710.e4.
  19. Horwitz LI, Moin T, Green ML. Development and implementation of an oral sign‐out skills curriculum. J Gen Intern Med. 2007;22(10):14701474.
  20. Horwitz LI, Moin T, Wang L, Bradley EH. Mixed methods evaluation of oral sign‐out practices. J Gen Intern Med. 2007;22(S1):S114.
  21. Horwitz LI, Parwani V, Shah NR, et al. Evaluation of an asynchronous physician voicemail sign‐out for emergency department admissions. Ann Emerg Med. 2009;54(3):368378.
  22. Horwitz LI, Schuster KM, Thung SF, et al. An institution‐wide handoff task force to standardise and improve physician handoffs. BMJ Qual Saf. 2012;21(10):863871.
  23. Arora V, Johnson J. A model for building a standardized hand‐off protocol. Jt Comm J Qual Patient Saf. 2006;32(11):646655.
  24. Arora V, Kao J, Lovinger D, Seiden SC, Meltzer D. Medication discrepancies in resident sign‐outs and their potential to harm. J Gen Intern Med. 2007;22(12):17511755.
  25. Arora VM, Johnson JK, Meltzer DO, Humphrey HJ. A theoretical framework and competency‐based approach to improving handoffs. Qual Saf Health Care. 2008;17(1):1114.
  26. Arora VM, Manjarrez E, Dressler DD, Basaviah P, Halasyamani L, Kripalani S. Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med. 2009;4(7):433440.
  27. Chang VY, Arora VM, Lev‐Ari S, D'Arcy M, Keysar B. Interns overestimate the effectiveness of their hand‐off communication. Pediatrics. 2010;125(3):491496.
  28. Johnson JK, Arora VM. Improving clinical handovers: creating local solutions for a global problem. Qual Saf Health Care. 2009;18(4):244245.
  29. Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign‐out. J Hosp Med. 2006;1(4):257266.
  30. Salerno SM, Arnett MV, Domanski JP. Standardized sign‐out reduces intern perception of medical errors on the general internal medicine ward. Teach Learn Med. 2009;21(2):121126.
  31. Haig KM, Sutton S, Whittington J. SBAR: a shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf. 2006;32(3):167175.
  32. Patterson ES. Structuring flexibility: the potential good, bad and ugly in standardisation of handovers. Qual Saf Health Care. 2008;17(1):45.
  33. Patterson ES, Roth EM, Woods DD, Chow R, Gomes JO. Handoff strategies in settings with high consequences for failure: lessons for health care operations. Int J Qual Health Care. 2004;16(2):125132.
  34. Ratanawongsa N, Bolen S, Howell EE, Kern DE, Sisson SD, Larriviere D. Residents' perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758763.
  35. Coiera E, Tombs V. Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673676.
  36. Coiera EW, Jayasuriya RA, Hardy J, Bannan A, Thorpe ME. Communication loads on clinical staff in the emergency department. Med J Aust. 2002;176(9):415418.
  37. Ong MS, Coiera E. A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274284.
  38. Farnan JM, Paro JA, Rodriguez RM, et al. Hand‐off education and evaluation: piloting the observed simulated hand‐off experience (OSHE). J Gen Intern Med. 2010;25(2):129134.
  39. Kitch BT, Cooper JB, Zapol WM, et al. Handoffs causing patient harm: a survey of medical and surgical house staff. Jt Comm J Qual Patient Saf. 2008;34(10):563570.
  40. Li P, Stelfox HT, Ghali WA. A prospective observational study of physician handoff for intensive‐care‐unit‐to‐ward patient transfers. Am J Med. 2011;124(9):860867.
  41. Greenstein E, Arora V, Banerjee S, Staisiunas P, Farnan J. Characterizing physician listening behavior during hospitalist handoffs using the HEAR checklist (published online ahead of print December 20, 2012]. BMJ Qual Saf. doi:10.1136/bmjqs‐2012‐001138.
  42. Thorndike EL. A constant error in psychological ratings. J Appl Psychol. 1920;4(1):25.
References
  1. Horwitz LI, Krumholz HM, Green ML, Huot SJ. Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):11731177.
  2. Accreditation Council for Graduate Medical Education. Common program requirements. 2011; http://www.acgme‐2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed August 23, 2011.
  3. Petersen LA, Brennan TA, O'Neil AC, Cook EF, Lee TH. Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866872.
  4. Sutcliffe KM, Lewton E, Rosenthal MM. Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186194.
  5. Arora V, Johnson J, Lovinger D, Humphrey HJ, Meltzer DO. Communication failures in patient sign‐out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401407.
  6. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. Consequences of inadequate sign‐out for patient care. Arch Intern Med. 2008;168(16):17551760.
  7. Borowitz SM, Waggoner‐Fountain LA, Bass EJ, Sledd RM. Adequacy of information transferred at resident sign‐out (in‐hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):610.
  8. Horwitz LI, Moin T, Krumholz HM, Wang L, Bradley EH. What are covering doctors told about their patients? Analysis of sign‐out among internal medicine house staff. Qual Saf Health Care. 2009;18(4):248255.
  9. Gakhar B, Spencer AL. Using direct observation, formal evaluation, and an interactive curriculum to improve the sign‐out practices of internal medicine interns. Acad Med. 2010;85(7):11821188.
  10. Raduma‐Tomas MA, Flin R, Yule S, Williams D. Doctors' handovers in hospitals: a literature review. Qual Saf Health Care. 2011;20(2):128133.
  11. Bump GM, Jovin F, Destefano L, et al. Resident sign‐out and patient hand‐offs: opportunities for improvement. Teach Learn Med. 2011;23(2):105111.
  12. Helms AS, Perez TE, Baltz J, et al. Use of an appreciative inquiry approach to improve resident sign‐out in an era of multiple shift changes. J Gen Intern Med. 2012;27(3):287291.
  13. Horwitz LI, Dombroski J, Murphy TE, Farnan JM, Johnson JK, Arora VM. Validation of a handoff assessment tool: the Handoff CEX [published online ahead of print June 7, 2012]. J Clin Nurs. doi: 10.1111/j.1365–2702.2012.04131.x.
  14. Norcini JJ, Blank LL, Arnold GK, Kimball HR. The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123(10):795799.
  15. Norcini JJ, Blank LL, Arnold GK, Kimball HR. Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):2733.
  16. Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini‐clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900904.
  17. Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826830.
  18. Horwitz LI, Meredith T, Schuur JD, Shah NR, Kulkarni RG, Jenq GY. Dropping the baton: a qualitative analysis of failures during the transition from emergency department to inpatient care. Ann Emerg Med. 2009;53(6):701710.e4.
  19. Horwitz LI, Moin T, Green ML. Development and implementation of an oral sign‐out skills curriculum. J Gen Intern Med. 2007;22(10):14701474.
  20. Horwitz LI, Moin T, Wang L, Bradley EH. Mixed methods evaluation of oral sign‐out practices. J Gen Intern Med. 2007;22(S1):S114.
  21. Horwitz LI, Parwani V, Shah NR, et al. Evaluation of an asynchronous physician voicemail sign‐out for emergency department admissions. Ann Emerg Med. 2009;54(3):368378.
  22. Horwitz LI, Schuster KM, Thung SF, et al. An institution‐wide handoff task force to standardise and improve physician handoffs. BMJ Qual Saf. 2012;21(10):863871.
  23. Arora V, Johnson J. A model for building a standardized hand‐off protocol. Jt Comm J Qual Patient Saf. 2006;32(11):646655.
  24. Arora V, Kao J, Lovinger D, Seiden SC, Meltzer D. Medication discrepancies in resident sign‐outs and their potential to harm. J Gen Intern Med. 2007;22(12):17511755.
  25. Arora VM, Johnson JK, Meltzer DO, Humphrey HJ. A theoretical framework and competency‐based approach to improving handoffs. Qual Saf Health Care. 2008;17(1):1114.
  26. Arora VM, Manjarrez E, Dressler DD, Basaviah P, Halasyamani L, Kripalani S. Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med. 2009;4(7):433440.
  27. Chang VY, Arora VM, Lev‐Ari S, D'Arcy M, Keysar B. Interns overestimate the effectiveness of their hand‐off communication. Pediatrics. 2010;125(3):491496.
  28. Johnson JK, Arora VM. Improving clinical handovers: creating local solutions for a global problem. Qual Saf Health Care. 2009;18(4):244245.
  29. Vidyarthi AR, Arora V, Schnipper JL, Wall SD, Wachter RM. Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign‐out. J Hosp Med. 2006;1(4):257266.
  30. Salerno SM, Arnett MV, Domanski JP. Standardized sign‐out reduces intern perception of medical errors on the general internal medicine ward. Teach Learn Med. 2009;21(2):121126.
  31. Haig KM, Sutton S, Whittington J. SBAR: a shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf. 2006;32(3):167175.
  32. Patterson ES. Structuring flexibility: the potential good, bad and ugly in standardisation of handovers. Qual Saf Health Care. 2008;17(1):45.
  33. Patterson ES, Roth EM, Woods DD, Chow R, Gomes JO. Handoff strategies in settings with high consequences for failure: lessons for health care operations. Int J Qual Health Care. 2004;16(2):125132.
  34. Ratanawongsa N, Bolen S, Howell EE, Kern DE, Sisson SD, Larriviere D. Residents' perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758763.
  35. Coiera E, Tombs V. Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673676.
  36. Coiera EW, Jayasuriya RA, Hardy J, Bannan A, Thorpe ME. Communication loads on clinical staff in the emergency department. Med J Aust. 2002;176(9):415418.
  37. Ong MS, Coiera E. A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274284.
  38. Farnan JM, Paro JA, Rodriguez RM, et al. Hand‐off education and evaluation: piloting the observed simulated hand‐off experience (OSHE). J Gen Intern Med. 2010;25(2):129134.
  39. Kitch BT, Cooper JB, Zapol WM, et al. Handoffs causing patient harm: a survey of medical and surgical house staff. Jt Comm J Qual Patient Saf. 2008;34(10):563570.
  40. Li P, Stelfox HT, Ghali WA. A prospective observational study of physician handoff for intensive‐care‐unit‐to‐ward patient transfers. Am J Med. 2011;124(9):860867.
  41. Greenstein E, Arora V, Banerjee S, Staisiunas P, Farnan J. Characterizing physician listening behavior during hospitalist handoffs using the HEAR checklist (published online ahead of print December 20, 2012]. BMJ Qual Saf. doi:10.1136/bmjqs‐2012‐001138.
  42. Thorndike EL. A constant error in psychological ratings. J Appl Psychol. 1920;4(1):25.
Issue
Journal of Hospital Medicine - 8(4)
Issue
Journal of Hospital Medicine - 8(4)
Page Number
191-200
Page Number
191-200
Publications
Publications
Article Type
Display Headline
Development of a handoff evaluation tool for shift‐to‐shift physician handoffs: The handoff CEX
Display Headline
Development of a handoff evaluation tool for shift‐to‐shift physician handoffs: The handoff CEX
Sections
Article Source

Copyright © 2013 Society of Hospital Medicine

Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Leora I. Horwitz, MD, Section of General Internal Medicine, Department of Internal Medicine, Yale School of Medicine, P.O. Box 208093, New Haven, CT 06520-8093; Telephone: 203-688‐5678; Fax: 203–737‐3306; E‐mail: leora.horwitz@yale.edu
Content Gating
No Gating (article Unlocked/Free)
Alternative CME
Article PDF Media
Media Files

Discharge Summary Quality

Article Type
Changed
Sun, 05/21/2017 - 18:05
Display Headline
Comprehensive quality of discharge summaries at an academic medical center

Hospitalized patients are often cared for by physicians who do not follow them in the community, creating a discontinuity of care that must be bridged through communication. This communication between inpatient and outpatient physicians occurs, in part via a discharge summary, which is intended to summarize events during hospitalization and prepare the outpatient physician to resume care of the patient. Yet, this form of communication has long been problematic.[1, 2, 3] In a 1960 study, only 30% of discharge letters were received by the primary care physician within 48 hours of discharge.[1]

More recent studies have shown little improvement. Direct communication between hospital and outpatient physicians is rare, and discharge summaries are still largely unavailable at the time of follow‐up.[4] In 1 study, primary care physicians were unaware of 62% of laboratory tests or study results that were pending on discharge,[5] in part because this information is missing from most discharge summaries.[6] Deficits such as these persist despite the fact that the rate of postdischarge completion of recommended tests, referrals, or procedures is significantly increased when the recommendation is included in the discharge summary.[7]

Regulatory mandates for discharge summaries from the Centers for Medicare and Medicaid Services[8] and from The Joint Commission[9] appear to be generally met[10, 11]; however, these mandates have no requirements for timeliness stricter than 30 days, do not require that summaries be transmitted to outpatient physicians, and do not require several content elements that might be useful to outside physicians such as condition of the patient at discharge, cognitive and functional status, goals of care, or pending studies. Expert opinion guidelines have more comprehensive recommendations,[12, 13] but it is uncertain how widely they are followed.

The existence of a discharge summary does not necessarily mean it serves a patient well in the transitional period.[11, 14, 15] Discharge summaries are a complex intervention, and we do not yet understand the best ways discharge summaries may fulfill needs specific to transitional care. Furthermore, it is uncertain what factors improve aspects of discharge summary quality as defined by timeliness, transmission, and content.[6, 16]

The goal of the DIagnosing Systemic failures, Complexities and HARm in GEriatric discharges study (DISCHARGE) was to comprehensively assess the discharge process for older patients discharged to the community. In this article we examine discharge summaries of patients enrolled in the study to determine the timeliness, transmission to outside physicians, and content of the summaries. We further examine the effect of provider training level and timeliness of dictation on discharge summary quality.

METHODS

Study Cohort

The DISCHARGE study was a prospective, observational cohort study of patients 65 years or older discharged to home from YaleNew Haven Hospital (YNHH) who were admitted with acute coronary syndrome (ACS), community‐acquired pneumonia, or heart failure (HF). Patients were screened by physicians for eligibility within 24 hours of admission using specialty society guidelines[17, 18, 19, 20] and were enrolled by telephone within 1 week of discharge. Additional inclusion criteria included speaking English or Spanish, and ability of the patient or caregiver to participate in a telephone interview. Patients enrolled in hospice were excluded, as were patients who failed the Mini‐Cog mental status screen (3‐item recall and a clock draw)[21] while in the hospital or appeared confused or delirious during the telephone interview. Caregivers of cognitively impaired patients were eligible for enrollment instead if the patient provided permission.

Study Setting

YNHH is a 966‐bed urban tertiary care hospital with statistically lower than the national average mortality for acute myocardial infarction, HF, and pneumonia but statistically higher than the national average for 30‐day readmission rates for HF and pneumonia at the time this study was conducted. Advanced practice registered nurses (APRNs) working under the supervision of private or university cardiologists provided care for cardiology service patients. Housestaff under the supervision of university or hospitalist attending physicians, or physician assistants or APRNs under the supervision of hospitalist attending physicians provided care for patients on medical services. Discharge summaries were typically dictated by APRNs for cardiology patients, by 2nd‐ or 3rd‐year residents for housestaff patients, and by hospitalists for hospitalist patients. A dictation guideline was provided to housestaff and hospitalists (see Supporting Information, Appendix 1, in the online version of this article); this guideline suggested including basic demographic information, disposition and diagnoses, the admission history and physical, hospital course, discharge medications, and follow‐up appointments. Additionally, housestaff received a lecture about discharge summaries at the start of their 2nd year. Discharge instructions including medications and follow‐up appointment information were automatically appended to the discharge summaries. Summaries were sent by the medical records department only to physicians in the system who were listed by the dictating physician as needing to receive a copy of the summary; no summary was automatically sent (ie, to the primary care physician) if not requested by the dictating physician.

Data Collection

Experienced registered nurses trained in chart abstraction conducted explicit reviews of medical charts using a standardized review tool. The tool included 24 questions about the discharge summary applicable to all 3 conditions, with 7 additional questions for patients with HF and 1 additional question for patients with ACS. These questions included the 6 elements required by The Joint Commission for all discharge summaries (reason for hospitalization, significant findings, procedures and treatment provided, patient's discharge condition, patient and family instructions, and attending physician's signature)[9] as well as the 7 elements (principal diagnosis and problem list, medication list, transferring physician name and contact information, cognitive status of the patient, test results, and pending test results) recommended by the Transitions of Care Consensus Conference (TOCCC), a recent consensus statement produced by 6 major medical societies.[13] Each content element is shown in (see Supporting Information, Appendix 2, in the online version of this article), which also indicates the elements included in the 2 guidelines.

Main Measures

We assessed quality in 3 main domains: timeliness, transmission, and content. We defined timeliness as days between discharge date and dictation date (not final signature date, which may occur later), and measured both median timeliness and proportion of discharge summaries completed on the day of discharge. We defined transmission as successful fax or mail of the discharge summary to an outside physician as reported by the medical records department, and measured the proportion of discharge summaries sent to any outside physician as well as the median number of physicians per discharge summary who were scheduled to follow‐up with the patient postdischarge but who did not receive a copy of the summary. We defined 21 individual content items and assessed the frequency of each individual content item. We also measured compliance with The Joint Commission mandates and TOCCC recommendations, which included several of the individual content items.

To measure compliance with The Joint Commission requirements, we created a composite score in which 1 point was provided for the presence of each of the 6 required elements (maximum score=6). Every discharge summary received 1 point for attending physician signature, because all discharge summaries were electronically signed. Discharge instructions to family/patients were automatically appended to every discharge summary; however, we gave credit for patient and family instructions only to those that included any information about signs and symptoms to monitor for at home. We defined discharge condition as any information about functional status, cognitive status, physical exam, or laboratory findings at discharge.

To measure compliance with specialty society recommendations for discharge summaries, we created a composite score in which 1 point was provided for the presence of each of the 7 recommended elements (maximum score=7). Every discharge summary received 1 point for discharge medications, because these are automatically appended.

We obtained data on age, race, gender, and length of stay from hospital administrative databases. The study was approved by the Yale Human Investigation Committee, and verbal informed consent was obtained from all study participants.

Statistical Analysis

Characteristics of the sample are described with counts and percentages or means and standard deviations. Medians and interquartile ranges (IQRs) or counts and percentages were calculated for summary measures of timeliness, transmission, and content. We assessed differences in quality measures between APRNs, housestaff, and hospitalists using 2 tests. We conducted multivariable logistic regression analyses for timeliness and for transmission to any outside physician. All discharge summaries included at least 4 of The Joint Commission elements; consequently, we coded this content outcome as an ordinal variable with 3 levels indicating inclusion of 4, 5, or 6 of The Joint Commission elements. We coded the TOCCC content outcome as a 3‐level variable indicating <4, 4, or >4 elements satisfied. Accordingly, proportional odds models were used, in which the reported odds ratios (ORs) can be interpreted as the average effect of the explanatory variable on the odds of having more recommendations, for any dichotomization of the outcome. Residual analysis and goodness‐of‐fit statistics were used to assess model fit; the proportional odds assumption was tested. Statistical analyses were conducted with SAS 9.2 (SAS Institute, Cary, NC). P values <0.05 were interpreted as statistically significant for 2‐sided tests.

RESULTS

Enrollment and Study Sample

A total of 3743 patients over 64 years old were discharged home from the medical service at YNHH during the study period; 3028 patients were screened for eligibility within 24 hours of admission. We identified 635 eligible admissions and enrolled 395 patients (62.2%) in the study. Of these, 377 granted permission for chart review and were included in this analysis (Figure 1).

Figure 1
Flow diagram of enrolled participants.

The study sample had a mean age of 77.1 years (standard deviation: 7.8); 205 (54.4%) were male and 310 (82.5%) were non‐Hispanic white. A total of 195 (51.7%) had ACS, 91 (24.1%) had pneumonia, and 146 (38.7%) had HF; 54 (14.3%) patients had more than 1 qualifying condition. There were similar numbers of patients on the cardiology, medicine housestaff, and medicine hospitalist teams (Table 1).

Study Sample Characteristics (N=377)
CharacteristicN (%) or Mean (SD)
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; N=number of study participants; GED, general educational development; SD=standard deviation.

Condition 
Acute coronary syndrome195 (51.7)
Community‐acquired pneumonia91 (24.1)
Heart failure146 (38.7)
Training level of summary dictator 
APRN140 (37.1)
House staff123 (32.6)
Hospitalist114 (30.2)
Length of stay, mean, d3.5 (2.5)
Total number of medications8.9 (3.3)
Identify a usual source of care360 (96.0)
Age, mean, y77.1 (7.8)
Male205 (54.4)
English‐speaking366 (98.1)
Race/ethnicity 
Non‐Hispanic white310 (82.5)
Non‐Hispanic black44 (11.7)
Hispanic15 (4.0)
Other7 (1.9)
High school graduate or GED Admission source268 (73.4)
Emergency department248 (66.0)
Direct transfer from hospital or nursing facility94 (25.0)
Direct admission from office34 (9.0)

Timeliness

Discharge summaries were completed for 376/377 patients, of which 174 (46.3%) were dictated on the day of discharge. However, 122 (32.4%) summaries were dictated more than 48 hours after discharge, including 93 (24.7%) that were dictated more than 1 week after discharge (see Supporting Information, Appendix 3, in the online version of this article).

Summaries dictated by hospitalists were most likely to be done on the day of discharge (35.3% APRNs, 38.2% housestaff, 68.4% hospitalists, P<0.001). After adjustment for diagnosis and length of stay, hospitalists were still significantly more likely to produce a timely discharge summary than APRNs (OR: 2.82; 95% confidence interval [CI]: 1.56‐5.09), whereas housestaff were no different than APRNs (OR: 0.84; 95% CI: 0.48‐1.46).

Transmission

A total of 144 (38.3%) discharge summaries were not sent to any physician besides the inpatient attending, and 209/374 (55.9%) were not sent to at least 1 physician listed as having a follow‐up appointment planned with the patient. Each discharge summary was sent to a median of 1 physician besides the dictating physician (IQR: 01). However, for each summary, a median of 1 physician (IQR: 01) who had a scheduled follow‐up with the patient did not receive the summary. Summaries dictated by hospitalists were most likely to be sent to at least 1 outside physician (54.7% APRNs, 58.5% housestaff, 73.7% hospitalists, P=0.006). Summaries dictated on the day of discharge were more likely than delayed summaries to be sent to at least 1 outside physician (75.9% vs 49.5%, P<0.001). After adjustment for diagnosis and length of stay, there was no longer a difference in likelihood of transmitting a discharge summary to any outpatient physician according to training level; however, dictations completed on the day of discharge remained significantly more likely to be transmitted to an outside physician (OR: 3.05; 95% CI: 1.88‐4.93) (Table 2).

Logistic Regression Model of Associations With Discharge Summary Transmission (N=376)
Explanatory VariableProportion Transmitted to at Least 1 Outside PhysicianOR for Transmission to Any Outside Physician (95% CI)Adjusted P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; CI, confidence interval; OR, odds ratio.

  • Patients could be categorized as having more than 1 eligible diagnosis.

Training level  0.52
APRN54.7%REF 
Housestaff58.5%1.17 (0.66‐2.06) 
Hospitalist73.7%1.46 (0.76‐2.79) 
Timeliness   
Dictated after discharge49.5%REF<0.001
Dictated day of discharge75.9%3.05 (1.88‐4.93) 
Acute coronary syndrome vs nota52.1 %1.05 (0.49‐2.26)0.89
Pneumonia vs nota69.2 %1.59 (0.66‐3.79)0.30
Heart failure vs nota74.7 %3.32 (1.61‐6.84)0.001
Length of stay, d 0.91 (0.83‐1.00)0.06

Content

Rate of inclusion of each content element is shown in Table 3, overall and by training level. Nearly every discharge summary included information about admitting diagnosis, hospital course, and procedures or tests performed during the hospitalization. However, few summaries included information about the patient's condition at discharge. Less than half included discharge laboratory results; less than one‐third included functional capacity, cognitive capacity, or discharge physical exam. Only 4.1% overall of discharge summaries for patients with HF included the patient's weight at discharge; best were hospitalists who still included this information in only 7.7% of summaries. Information about postdischarge care, including home social support, pending tests, or recommended follow‐up tests/procedures was also rarely specified. Last, only 6.2% of discharge summaries included the name and contact number of the inpatient physician; this information was least likely to be provided by housestaff (1.6%) and most likely to be provided by hospitalists (15.2%) (P<0.001).

Content of Discharge SummariesOverall and by Training Level
Discharge Summary ComponentOverall, n=377, n (%)APRN, n=140, n (%)Housestaff, n=123, n (%)Hospitalist, n=114, n (%)P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; GFR, glomerular filtration rate.

  • Included in Joint Commission composite.

  • Included in Transitions of Care Consensus Conference composite.

  • Patients with heart failure only (n=146).

  • Patients with stents placed only (n=109).

Diagnosisab368 (97.9)136 (97.8)120 (97.6)112 (98.3)1.00
Discharge second diagnosisb289 (76.9)100 (71.9)89 (72.4)100 (87.7)<0.001
Hospital coursea375 (100.0)138 (100)123 (100)114 (100)N/A
Procedures/tests performed during admissionab374 (99.7)138 (99.3)123 (100)113 (100)N/A
Patient and family instructionsa371 (98.4)136 (97.1)122 (99.2)113 (99.1).43
Social support or living situation of patient148 (39.5)18 (12.9)62 (50.4)68 (60.2)<0.001
Functional capacity at dischargea99 (26.4)37 (26.6)32 (26.0)30 (26.6)0.99
Cognitive capacity at dischargeab30 (8.0)6 (4.4)11 (8.9)13 (11.5)0.10
Physical exam at dischargea62 (16.7)19 (13.8)16 (13.1)27 (24.1)0.04
Laboratory results at time of dischargea164 (43.9)63 (45.3)50 (40.7)51 (45.5)0.68
Back to baseline or other nonspecific remark about discharge statusa71 (19.0)30 (21.6)18 (14.8)23 (20.4)0.34
Any test or result still pending or specific comment that nothing is pendingb46 (12.2)9 (6.4)20 (16.3)17 (14.9)0.03
Recommendation for follow‐up tests/procedures157 (41.9)43 (30.9)54 (43.9)60 (53.1)0.002
Call‐back number of responsible in‐house physicianb23 (6.2)4 (2.9)2 (1.6)17 (15.2)<0.001
Resuscitation status27 (7.7)2 (1.5)18 (15.4)7 (6.7)<0.001
Etiology of heart failurec120 (82.8)44 (81.5)34 (87.2)42 (80.8)0.69
Reason/trigger for exacerbationc86 (58.9)30 (55.6)27 (67.5)29 (55.8)0.43
Ejection fractionc107 (73.3)40 (74.1)32 (80.0)35 (67.3)0.39
Discharge weightc6 (4.1)1 (1.9)1 (2.5)4 (7.7)0.33
Target weight rangec5 (3.4)0 (0)2 (5.0)3 (5.8)0.22
Discharge creatinine or GFRc34 (23.3)14 (25.9)10 (25.0)10 (19.2)0.69
If stent placed, whether drug‐eluting or notd89 (81.7)58 (87.9)27 (81.8)4 (40.0)0.001

On average, summaries included 5.6 of the 6 Joint Commission elements and 4.0 of the 7 TOCCC elements. A total of 63.0% of discharge summaries included all 6 elements required by The Joint Commission, whereas no discharge summary included all 7 TOCCC elements.

APRNs, housestaff and hospitalists included the same average number of The Joint Commission elements (5.6 each), but hospitalists on average included slightly more TOCCC elements (4.3) than did housestaff (4.0) or APRNs (3.8) (P<0.001). Summaries dictated on the day of discharge included an average of 4.2 TOCCC elements, compared to 3.9 TOCCC elements in delayed discharge. In multivariable analyses adjusted for diagnosis and length of stay, there was still no difference by training level in presence of The Joint Commission elements, but hospitalists were significantly more likely to include more TOCCC elements than APRNs (OR: 2.70; 95% CI: 1.49‐4.90) (Table 4). Summaries dictated on the day of discharge were significantly more likely to include more TOCCC elements (OR: 1.92; 95% CI: 1.23‐2.99).

Proportional Odds Model of Associations With Including More Elements Recommended by Specialty Societies (N=376)
Explanatory VariableAverage Number of TOCCC Elements IncludedOR (95% CI)Adjusted P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; CI, confidence interval; OR, odds ratio; TOCCC, Transitions of Care Consensus Conference (defined by Snow et al.[13]).

  • Patients could be categorized as having more than 1 eligible diagnosis.

Training level  0.004
APRN3.8REF 
Housestaff4.01.54 (0.90‐2.62) 
Hospitalist4.32.70 (1.49‐4.90) 
Timeliness   
Dictated after discharge3.9REF 
Dictated day of discharge4.21.92 (1.23‐2.99)0.004
Acute coronary syndrome vs nota3.90.72 (0.37‐1.39)0.33
Pneumonia vs nota4.21.02 (0.49‐2.14)0.95
Heart failure vs nota4.11.49 (0.80‐2.78)0.21
Length of stay, d 0.99 (0.90‐1.07)0.73

No discharge summary included all 7 TOCCC‐endorsed content elements, was dictated on the day of discharge, and was sent to an outside physician.

DISCUSSION

In this prospective single‐site study of medical patients with 3 common conditions, we found that discharge summaries were completed relatively promptly, but were often not sent to the appropriate outpatient physicians. We also found that summaries were uniformly excellent at providing details of the hospitalization, but less reliable at providing details relevant to transitional care such as the patient's condition on discharge or existence of pending tests. On average, summaries included 57% of the elements included in consensus guidelines by 6 major medical societies. The content of discharge summaries dictated by hospitalists was slightly more comprehensive than that of APRNs and trainees, but no group exhibited high performance. In fact, not one discharge summary fully met all 3 quality criteria of timeliness, transmission, and content.

Our study, unlike most in the field, focused on multiple dimensions of discharge summary quality simultaneously. For instance, previous studies have found that timely receipt of a discharge summary does not reduce readmission rates.[11, 14, 15] Yet, if the content of the discharge summary is inadequate for postdischarge care, the summary may not be useful even if it is received by the follow‐up visit. Conversely, high‐quality content is ineffective if the summary is not sent to the outpatient physician.

This study suggests several avenues for improving summary quality. Timely discharge summaries in this study were more likely to include key content and to be transmitted to the appropriate physician. Strategies to improve discharge summary quality should therefore prioritize timely summaries, which can be expected to have downstream benefits for other aspects of quality. Some studies have found that templates improve discharge summary content.[22] In our institution, a template exists, but it favors a hospitalization‐focused rather than transition‐focused approach to the discharge summary. For instance, it includes instructions to dictate the admission exam, but not the discharge exam. Thus, designing templates specifically for transitional care is key. Maximizing capabilities of electronic records may help; many content elements that were commonly missing (e.g., pending results, discharge vitals, discharge weight) could be automatically inserted from electronic records. Likewise, automatic transmission of the summary to care providers listed in the electronic record might ameliorate many transmission failures. Some efforts have been made to convert existing electronic data into discharge summaries.[23, 24, 25] However, these activities are very preliminary, and some studies have found the quality of electronic summaries to be lower than dictated or handwritten summaries.[26] As with all automated or electronic applications, it will be essential to consider workflow, readability, and ability to synthesize information prior to adoption.

Hospitalists consistently produced highest‐quality summaries, even though they did not receive explicit training, suggesting experience may be beneficial,[27, 28, 29] or that the hospitalist community focus on transitional care has been effective. In addition, hospitalists at our institution explicitly prioritize timely and comprehensive discharge dictations, because their business relies on maintaining good relationships with outpatient physicians who contract for their services. Housestaff and APRNs have no such incentives or policies; rather, they typically consider discharge summaries to be a useful source of patient history at the time of an admission or readmission. Other academic centers have found similar results.[6, 16] Nonetheless, even though hospitalists had slightly better performance in our study, large gaps in the quality of summaries remained for all groups including hospitalists.

This study has several limitations. First, as a single‐site study at an academic hospital, it may not be generalizable to other hospitals or other settings. It is noteworthy, however, that the average time to dictation in this study was much lower than that of other studies,[4, 14, 30, 31, 32] suggesting that practices at this institution are at least no worse and possibly better than elsewhere. Second, although there are some mandates and expert opinion‐based guidelines for discharge summary content, there is no validated evidence base to confirm what content ought to be present in discharge summaries to improve patient outcomes. Third, we had too few readmissions in the dataset to have enough power to determine whether discharge summary content, timeliness, or transmission predicts readmission. Fourth, we did not determine whether the information in discharge summaries was accurate or complete; we merely assessed whether it was present. For example, we gave every discharge summary full credit for including discharge medications because they are automatically appended. Yet medication reconciliation errors at discharge are common.[33, 34] In fact, in the DISCHARGE study cohort, more than a quarter of discharge medication lists contained a suspected error.[35]

In summary, this study demonstrated the inadequacy of the contemporary discharge summary for conveying information that is critical to the transition from hospital to home. It may be that hospital culture treats hospitalizations as discrete and self‐contained events rather than as components of a larger episode of care. As interest in reducing readmissions rises, reframing the discharge summary to serve as a transitional tool and targeting it for quality assessment will likely be necessary.

Acknowledgments

The authors would like to acknowledge Amy Browning and the staff of the Center for Outcomes Research and Evaluation Follow‐Up Center for conducting patient interviews, Mark Abroms and Katherine Herman for patient recruitment and screening, and Peter Charpentier for Web site development.

Disclosures

At the time this study was conducted, Dr. Horwitz was supported by the CTSA Grant UL1 RR024139 and KL2 RR024138 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH), and NIH roadmap for Medical Research, and was a Centers of Excellence Scholar in Geriatric Medicine by the John A. Hartford Foundation and the American Federation for Aging Research. Dr. Horwitz is now supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. This work was also supported by a grant from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). Dr. Krumholz is supported by grant U01 HL105270‐01 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute. No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging, the National Center for Advancing Translational Sciences, the National Institutes of Health, The John A. Hartford Foundation, the National Heart, Lung, and Blood Institute, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as an oral presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 12, 2012. Dr. Krumholz chairs a cardiac scientific advisory board for UnitedHealth. Dr. Krumholz receives support from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures that are used for public reporting, including readmission measures.

APPENDIX

A

Dictation guidelines provided to house staff and hospitalists

DICTATION GUIDELINES

FORMAT OF DISCHARGE SUMMARY

 

  • Your name(spell it out), andPatient name(spell it out as well)
  • Medical record number, date of admission, date of discharge
  • Attending physician
  • Disposition
  • Principal and other diagnoses, Principal and other operations/procedures
  • Copies to be sent to other physicians
  • Begin narrative: CC, HPI, PMHx, Medications on admit, Social, Family Hx, Physical exam on admission, Data (labs on admission, plus labs relevant to workup, significant changes at discharge, admission EKG, radiologic and other data),Hospital course by problem, discharge meds, follow‐up appointments

 

APPENDIX

B

 

Content Items Abstracted
Diagnosis
Discharge Second Diagnosis
Hospital course
Procedures/tests performed during admission
Patient and Family Instructions
Social support or living situation of patient
Functional capacity at discharge
Cognitive capacity at discharge
Physical exam at discharge
Laboratory results at time of discharge
Back to baseline or other nonspecific remark about discharge status
Any test or result still pending
Specific comment that nothing is pending
Recommendation for follow up tests/procedures
Call back number of responsible in‐house physician
Resuscitation status
Etiology of heart failure
Reason/trigger for exacerbation
Ejection fraction
Discharge weight
Target weight range
Discharge creatinine or GFR
If stent placed, whether drug‐eluting or not
Joint Commission Composite Elements
Composite elementData elements abstracted that qualify as meeting measure
Reason for hospitalizationDiagnosis
Significant findingsHospital course
Procedures and treatment providedProcedures/tests performed during admission
Patient's discharge conditionFunctional capacity at discharge, Cognitive capacity at discharge, Physical exam at discharge, Laboratory results at time of discharge, Back to baseline or other nonspecific remark about discharge status
Patient and family instructionsSigns and symptoms to monitor at home
Attending physician's signatureAttending signature
Transitions of Care Consensus Conference Composite Elements
Composite elementData elements abstracted that qualify as meeting measure
Principal diagnosisDiagnosis
Problem listDischarge second diagnosis
Medication list[Automatically appended; full credit to every summary]
Transferring physician name and contact informationCall back number of responsible in‐house physician
Cognitive status of the patientCognitive capacity at discharge
Test resultsProcedures/tests performed during admission
Pending test resultsAny test or result still pending or specific comment that nothing is pending

APPENDIX

C

Histogram of days between discharge and dictation

 

 

 

Files
References
  1. Alarcon R, Glanville H, Hodson JM. Value of the specialist's report. Br Med J. 1960;2(5213):16631664.
  2. Long A, Atkins JB. Communications between general practitioners and consultants. Br Med J. 1974;4(5942):456459.
  3. Swender PT, Schneider AJ, Oski FA. A functional hospital discharge summary. J Pediatr. 1975;86(1):9798.
  4. Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital‐based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007;297(8):831841.
  5. Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143(2):121128.
  6. Were MC, Li X, Kesterson J, et al. Adequacy of hospital discharge summaries in documenting tests with pending results and outpatient follow‐up providers. J Gen Intern Med. 2009;24(9):10021006.
  7. Moore C, McGinn T, Halm E. Tying up loose ends: discharging patients with unresolved medical issues. Arch Intern Med. 2007;167(12):13051311.
  8. Centers for Medicare and Medicaid Services. Condition of participation: medical record services. 42. Vol 482.C.F.R. § 482.24 (2012).
  9. Joint Commission on Accreditation of Healthcare Organizations. Hospital Accreditation Standards. Standard IM 6.10 EP 7–9. Oakbrook Terrace, IL: The Joint Commission; 2008.
  10. Kind AJH, Smith MA. Documentation of mandated discharge summary components in transitions from acute to subacute care. In: Agency for Healthcare Research and Quality, ed. Advances in Patient Safety: New Directions and Alternative Approaches. Vol 2: Culture and Redesign. AHRQ Publication No. 08-0034‐2. Rockville, MD: Agency for Healthcare Research and Quality; 2008:179–188.
  11. Hansen LO, Strater A, Smith L, et al. Hospital discharge documentation and risk of rehospitalisation. BMJ Qual Saf. 2011;20(9):773778.
  12. Halasyamani L, Kripalani S, Coleman E, et al. Transition of care for hospitalized elderly patients‐development of a discharge checklist for hospitalists. J Hosp Med. 2006;1(6):354360.
  13. Snow V, Beck D, Budnitz T, et al. Transitions of Care Consensus Policy Statement American College of Physicians‐Society of General Internal Medicine‐Society of Hospital Medicine‐American Geriatrics Society‐American College of Emergency Physicians‐Society of Academic Emergency Medicine. J Gen Intern Med. 2009;24(8):971976.
  14. Bell CM, Schnipper JL, Auerbach AD, et al. Association of communication between hospital‐based physicians and primary care providers with patient outcomes. J Gen Intern Med. 2009;24(3):381386.
  15. Walraven C, Seth R, Austin PC, Laupacis A. Effect of discharge summary availability during post‐discharge visits on hospital readmission. J Gen Intern Med. 2002;17(3):186192.
  16. Kind AJ, Thorpe CT, Sattin JA, Walz SE, Smith MA. Provider characteristics, clinical‐work processes and their relationship to discharge summary quality for sub‐acute care patients. J Gen Intern Med. 2012;27(1):7884.
  17. Anderson JL, Adams CD, Antman EM, et al. ACC/AHA 2007 guidelines for the management of patients with unstable angina/non‐ST‐elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines for the Management of Patients With Unstable Angina/Non‐ST‐Elevation Myocardial Infarction) developed in collaboration with the American College of Emergency Physicians, the Society for Cardiovascular Angiography and Interventions, and the Society of Thoracic Surgeons endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation and the Society for Academic Emergency Medicine. J Am Coll Cardiol. 2007;50(7):e1e157.
  18. Thygesen K, Alpert JS, White HD. Universal definition of myocardial infarction. Eur Heart J. 2007;28(20):25252538.
  19. Dickstein K, Cohen‐Solal A, Filippatos G, et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2008: the Task Force for the diagnosis and treatment of acute and chronic heart failure 2008 of the European Society of Cardiology. Developed in collaboration with the Heart Failure Association of the ESC (HFA) and endorsed by the European Society of Intensive Care Medicine (ESICM). Eur J Heart Fail. 2008;10(10):933989.
  20. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community‐acquired pneumonia in adults. Clin Infect Dis. 2007;44(suppl 2):S27S72.
  21. Sunderland T, Hill JL, Mellow AM, et al. Clock drawing in Alzheimer's disease. A novel measure of dementia severity. J Am Geriatr Soc. 1989;37(8):725729.
  22. Rao P, Andrei A, Fried A, Gonzalez D, Shine D. Assessing quality and efficiency of discharge summaries. Am J Med Qual. 2005;20(6):337343.
  23. Maslove DM, Leiter RE, Griesman J, et al. Electronic versus dictated hospital discharge summaries: a randomized controlled trial. J Gen Intern Med. 2009;24(9):9951001.
  24. Walraven C, Laupacis A, Seth R, Wells G. Dictated versus database‐generated discharge summaries: a randomized clinical trial. CMAJ. 1999;160(3):319326.
  25. Llewelyn DE, Ewins DL, Horn J, Evans TG, McGregor AM. Computerised updating of clinical summaries: new opportunities for clinical practice and research? BMJ. 1988;297(6662):15041506.
  26. Callen JL, Alderton M, McIntosh J. Evaluation of electronic discharge summaries: a comparison of documentation in electronic and handwritten discharge summaries. Int J Med Inform. 2008;77(9):613620.
  27. Davis MM, Devoe M, Kansagara D, Nicolaidis C, Englander H. Did I do as best as the system would let me? Healthcare professional views on hospital to home care transitions. J Gen Intern Med. 2012;27(12):16491656.
  28. Greysen SR, Schiliro D, Curry L, Bradley EH, Horwitz LI. Learning by doing—resident perspectives on developing competency in high‐quality discharge care. J Gen Intern Med. 2012;27(9):11881194.
  29. Greysen SR, Schiliro D, Horwitz LI, Curry L, Bradley EH. Out of sight, out of mind: housestaff perceptions of quality‐limiting factors in discharge care at teaching hospitals. J Hosp Med. 2012;7(5):376381.
  30. Walraven C, Seth R, Laupacis A. Dissemination of discharge summaries. Not reaching follow‐up physicians. Can Fam Physician. 2002;48:737742.
  31. Pantilat SZ, Lindenauer PK, Katz PP, Wachter RM. Primary care physician attitudes regarding communication with hospitalists. Am J Med. 2001;111(9B):15S20S.
  32. Wilson S, Ruscoe W, Chapman M, Miller R. General practitioner‐hospital communications: a review of discharge summaries. J Qual Clin Pract. 2001;21(4):104108.
  33. McMillan TE, Allan W, Black PN. Accuracy of information on medicines in hospital discharge summaries. Intern Med J. 2006;36(4):221225.
  34. Callen J, McIntosh J, Li J. Accuracy of medication documentation in hospital discharge summaries: A retrospective analysis of medication transcription errors in manual and electronic discharge summaries. Int J Med Inform. 2010;79(1):5864.
  35. Ziaeian B, Araujo KL, Ness PH, Horwitz LI. Medication reconciliation accuracy and patient understanding of intended medication changes on hospital discharge. J Gen Intern Med. 2012;27(11):15131520.
Article PDF
Issue
Journal of Hospital Medicine - 8(8)
Publications
Page Number
436-443
Sections
Files
Files
Article PDF
Article PDF

Hospitalized patients are often cared for by physicians who do not follow them in the community, creating a discontinuity of care that must be bridged through communication. This communication between inpatient and outpatient physicians occurs, in part via a discharge summary, which is intended to summarize events during hospitalization and prepare the outpatient physician to resume care of the patient. Yet, this form of communication has long been problematic.[1, 2, 3] In a 1960 study, only 30% of discharge letters were received by the primary care physician within 48 hours of discharge.[1]

More recent studies have shown little improvement. Direct communication between hospital and outpatient physicians is rare, and discharge summaries are still largely unavailable at the time of follow‐up.[4] In 1 study, primary care physicians were unaware of 62% of laboratory tests or study results that were pending on discharge,[5] in part because this information is missing from most discharge summaries.[6] Deficits such as these persist despite the fact that the rate of postdischarge completion of recommended tests, referrals, or procedures is significantly increased when the recommendation is included in the discharge summary.[7]

Regulatory mandates for discharge summaries from the Centers for Medicare and Medicaid Services[8] and from The Joint Commission[9] appear to be generally met[10, 11]; however, these mandates have no requirements for timeliness stricter than 30 days, do not require that summaries be transmitted to outpatient physicians, and do not require several content elements that might be useful to outside physicians such as condition of the patient at discharge, cognitive and functional status, goals of care, or pending studies. Expert opinion guidelines have more comprehensive recommendations,[12, 13] but it is uncertain how widely they are followed.

The existence of a discharge summary does not necessarily mean it serves a patient well in the transitional period.[11, 14, 15] Discharge summaries are a complex intervention, and we do not yet understand the best ways discharge summaries may fulfill needs specific to transitional care. Furthermore, it is uncertain what factors improve aspects of discharge summary quality as defined by timeliness, transmission, and content.[6, 16]

The goal of the DIagnosing Systemic failures, Complexities and HARm in GEriatric discharges study (DISCHARGE) was to comprehensively assess the discharge process for older patients discharged to the community. In this article we examine discharge summaries of patients enrolled in the study to determine the timeliness, transmission to outside physicians, and content of the summaries. We further examine the effect of provider training level and timeliness of dictation on discharge summary quality.

METHODS

Study Cohort

The DISCHARGE study was a prospective, observational cohort study of patients 65 years or older discharged to home from YaleNew Haven Hospital (YNHH) who were admitted with acute coronary syndrome (ACS), community‐acquired pneumonia, or heart failure (HF). Patients were screened by physicians for eligibility within 24 hours of admission using specialty society guidelines[17, 18, 19, 20] and were enrolled by telephone within 1 week of discharge. Additional inclusion criteria included speaking English or Spanish, and ability of the patient or caregiver to participate in a telephone interview. Patients enrolled in hospice were excluded, as were patients who failed the Mini‐Cog mental status screen (3‐item recall and a clock draw)[21] while in the hospital or appeared confused or delirious during the telephone interview. Caregivers of cognitively impaired patients were eligible for enrollment instead if the patient provided permission.

Study Setting

YNHH is a 966‐bed urban tertiary care hospital with statistically lower than the national average mortality for acute myocardial infarction, HF, and pneumonia but statistically higher than the national average for 30‐day readmission rates for HF and pneumonia at the time this study was conducted. Advanced practice registered nurses (APRNs) working under the supervision of private or university cardiologists provided care for cardiology service patients. Housestaff under the supervision of university or hospitalist attending physicians, or physician assistants or APRNs under the supervision of hospitalist attending physicians provided care for patients on medical services. Discharge summaries were typically dictated by APRNs for cardiology patients, by 2nd‐ or 3rd‐year residents for housestaff patients, and by hospitalists for hospitalist patients. A dictation guideline was provided to housestaff and hospitalists (see Supporting Information, Appendix 1, in the online version of this article); this guideline suggested including basic demographic information, disposition and diagnoses, the admission history and physical, hospital course, discharge medications, and follow‐up appointments. Additionally, housestaff received a lecture about discharge summaries at the start of their 2nd year. Discharge instructions including medications and follow‐up appointment information were automatically appended to the discharge summaries. Summaries were sent by the medical records department only to physicians in the system who were listed by the dictating physician as needing to receive a copy of the summary; no summary was automatically sent (ie, to the primary care physician) if not requested by the dictating physician.

Data Collection

Experienced registered nurses trained in chart abstraction conducted explicit reviews of medical charts using a standardized review tool. The tool included 24 questions about the discharge summary applicable to all 3 conditions, with 7 additional questions for patients with HF and 1 additional question for patients with ACS. These questions included the 6 elements required by The Joint Commission for all discharge summaries (reason for hospitalization, significant findings, procedures and treatment provided, patient's discharge condition, patient and family instructions, and attending physician's signature)[9] as well as the 7 elements (principal diagnosis and problem list, medication list, transferring physician name and contact information, cognitive status of the patient, test results, and pending test results) recommended by the Transitions of Care Consensus Conference (TOCCC), a recent consensus statement produced by 6 major medical societies.[13] Each content element is shown in (see Supporting Information, Appendix 2, in the online version of this article), which also indicates the elements included in the 2 guidelines.

Main Measures

We assessed quality in 3 main domains: timeliness, transmission, and content. We defined timeliness as days between discharge date and dictation date (not final signature date, which may occur later), and measured both median timeliness and proportion of discharge summaries completed on the day of discharge. We defined transmission as successful fax or mail of the discharge summary to an outside physician as reported by the medical records department, and measured the proportion of discharge summaries sent to any outside physician as well as the median number of physicians per discharge summary who were scheduled to follow‐up with the patient postdischarge but who did not receive a copy of the summary. We defined 21 individual content items and assessed the frequency of each individual content item. We also measured compliance with The Joint Commission mandates and TOCCC recommendations, which included several of the individual content items.

To measure compliance with The Joint Commission requirements, we created a composite score in which 1 point was provided for the presence of each of the 6 required elements (maximum score=6). Every discharge summary received 1 point for attending physician signature, because all discharge summaries were electronically signed. Discharge instructions to family/patients were automatically appended to every discharge summary; however, we gave credit for patient and family instructions only to those that included any information about signs and symptoms to monitor for at home. We defined discharge condition as any information about functional status, cognitive status, physical exam, or laboratory findings at discharge.

To measure compliance with specialty society recommendations for discharge summaries, we created a composite score in which 1 point was provided for the presence of each of the 7 recommended elements (maximum score=7). Every discharge summary received 1 point for discharge medications, because these are automatically appended.

We obtained data on age, race, gender, and length of stay from hospital administrative databases. The study was approved by the Yale Human Investigation Committee, and verbal informed consent was obtained from all study participants.

Statistical Analysis

Characteristics of the sample are described with counts and percentages or means and standard deviations. Medians and interquartile ranges (IQRs) or counts and percentages were calculated for summary measures of timeliness, transmission, and content. We assessed differences in quality measures between APRNs, housestaff, and hospitalists using 2 tests. We conducted multivariable logistic regression analyses for timeliness and for transmission to any outside physician. All discharge summaries included at least 4 of The Joint Commission elements; consequently, we coded this content outcome as an ordinal variable with 3 levels indicating inclusion of 4, 5, or 6 of The Joint Commission elements. We coded the TOCCC content outcome as a 3‐level variable indicating <4, 4, or >4 elements satisfied. Accordingly, proportional odds models were used, in which the reported odds ratios (ORs) can be interpreted as the average effect of the explanatory variable on the odds of having more recommendations, for any dichotomization of the outcome. Residual analysis and goodness‐of‐fit statistics were used to assess model fit; the proportional odds assumption was tested. Statistical analyses were conducted with SAS 9.2 (SAS Institute, Cary, NC). P values <0.05 were interpreted as statistically significant for 2‐sided tests.

RESULTS

Enrollment and Study Sample

A total of 3743 patients over 64 years old were discharged home from the medical service at YNHH during the study period; 3028 patients were screened for eligibility within 24 hours of admission. We identified 635 eligible admissions and enrolled 395 patients (62.2%) in the study. Of these, 377 granted permission for chart review and were included in this analysis (Figure 1).

Figure 1
Flow diagram of enrolled participants.

The study sample had a mean age of 77.1 years (standard deviation: 7.8); 205 (54.4%) were male and 310 (82.5%) were non‐Hispanic white. A total of 195 (51.7%) had ACS, 91 (24.1%) had pneumonia, and 146 (38.7%) had HF; 54 (14.3%) patients had more than 1 qualifying condition. There were similar numbers of patients on the cardiology, medicine housestaff, and medicine hospitalist teams (Table 1).

Study Sample Characteristics (N=377)
CharacteristicN (%) or Mean (SD)
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; N=number of study participants; GED, general educational development; SD=standard deviation.

Condition 
Acute coronary syndrome195 (51.7)
Community‐acquired pneumonia91 (24.1)
Heart failure146 (38.7)
Training level of summary dictator 
APRN140 (37.1)
House staff123 (32.6)
Hospitalist114 (30.2)
Length of stay, mean, d3.5 (2.5)
Total number of medications8.9 (3.3)
Identify a usual source of care360 (96.0)
Age, mean, y77.1 (7.8)
Male205 (54.4)
English‐speaking366 (98.1)
Race/ethnicity 
Non‐Hispanic white310 (82.5)
Non‐Hispanic black44 (11.7)
Hispanic15 (4.0)
Other7 (1.9)
High school graduate or GED Admission source268 (73.4)
Emergency department248 (66.0)
Direct transfer from hospital or nursing facility94 (25.0)
Direct admission from office34 (9.0)

Timeliness

Discharge summaries were completed for 376/377 patients, of which 174 (46.3%) were dictated on the day of discharge. However, 122 (32.4%) summaries were dictated more than 48 hours after discharge, including 93 (24.7%) that were dictated more than 1 week after discharge (see Supporting Information, Appendix 3, in the online version of this article).

Summaries dictated by hospitalists were most likely to be done on the day of discharge (35.3% APRNs, 38.2% housestaff, 68.4% hospitalists, P<0.001). After adjustment for diagnosis and length of stay, hospitalists were still significantly more likely to produce a timely discharge summary than APRNs (OR: 2.82; 95% confidence interval [CI]: 1.56‐5.09), whereas housestaff were no different than APRNs (OR: 0.84; 95% CI: 0.48‐1.46).

Transmission

A total of 144 (38.3%) discharge summaries were not sent to any physician besides the inpatient attending, and 209/374 (55.9%) were not sent to at least 1 physician listed as having a follow‐up appointment planned with the patient. Each discharge summary was sent to a median of 1 physician besides the dictating physician (IQR: 01). However, for each summary, a median of 1 physician (IQR: 01) who had a scheduled follow‐up with the patient did not receive the summary. Summaries dictated by hospitalists were most likely to be sent to at least 1 outside physician (54.7% APRNs, 58.5% housestaff, 73.7% hospitalists, P=0.006). Summaries dictated on the day of discharge were more likely than delayed summaries to be sent to at least 1 outside physician (75.9% vs 49.5%, P<0.001). After adjustment for diagnosis and length of stay, there was no longer a difference in likelihood of transmitting a discharge summary to any outpatient physician according to training level; however, dictations completed on the day of discharge remained significantly more likely to be transmitted to an outside physician (OR: 3.05; 95% CI: 1.88‐4.93) (Table 2).

Logistic Regression Model of Associations With Discharge Summary Transmission (N=376)
Explanatory VariableProportion Transmitted to at Least 1 Outside PhysicianOR for Transmission to Any Outside Physician (95% CI)Adjusted P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; CI, confidence interval; OR, odds ratio.

  • Patients could be categorized as having more than 1 eligible diagnosis.

Training level  0.52
APRN54.7%REF 
Housestaff58.5%1.17 (0.66‐2.06) 
Hospitalist73.7%1.46 (0.76‐2.79) 
Timeliness   
Dictated after discharge49.5%REF<0.001
Dictated day of discharge75.9%3.05 (1.88‐4.93) 
Acute coronary syndrome vs nota52.1 %1.05 (0.49‐2.26)0.89
Pneumonia vs nota69.2 %1.59 (0.66‐3.79)0.30
Heart failure vs nota74.7 %3.32 (1.61‐6.84)0.001
Length of stay, d 0.91 (0.83‐1.00)0.06

Content

Rate of inclusion of each content element is shown in Table 3, overall and by training level. Nearly every discharge summary included information about admitting diagnosis, hospital course, and procedures or tests performed during the hospitalization. However, few summaries included information about the patient's condition at discharge. Less than half included discharge laboratory results; less than one‐third included functional capacity, cognitive capacity, or discharge physical exam. Only 4.1% overall of discharge summaries for patients with HF included the patient's weight at discharge; best were hospitalists who still included this information in only 7.7% of summaries. Information about postdischarge care, including home social support, pending tests, or recommended follow‐up tests/procedures was also rarely specified. Last, only 6.2% of discharge summaries included the name and contact number of the inpatient physician; this information was least likely to be provided by housestaff (1.6%) and most likely to be provided by hospitalists (15.2%) (P<0.001).

Content of Discharge SummariesOverall and by Training Level
Discharge Summary ComponentOverall, n=377, n (%)APRN, n=140, n (%)Housestaff, n=123, n (%)Hospitalist, n=114, n (%)P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; GFR, glomerular filtration rate.

  • Included in Joint Commission composite.

  • Included in Transitions of Care Consensus Conference composite.

  • Patients with heart failure only (n=146).

  • Patients with stents placed only (n=109).

Diagnosisab368 (97.9)136 (97.8)120 (97.6)112 (98.3)1.00
Discharge second diagnosisb289 (76.9)100 (71.9)89 (72.4)100 (87.7)<0.001
Hospital coursea375 (100.0)138 (100)123 (100)114 (100)N/A
Procedures/tests performed during admissionab374 (99.7)138 (99.3)123 (100)113 (100)N/A
Patient and family instructionsa371 (98.4)136 (97.1)122 (99.2)113 (99.1).43
Social support or living situation of patient148 (39.5)18 (12.9)62 (50.4)68 (60.2)<0.001
Functional capacity at dischargea99 (26.4)37 (26.6)32 (26.0)30 (26.6)0.99
Cognitive capacity at dischargeab30 (8.0)6 (4.4)11 (8.9)13 (11.5)0.10
Physical exam at dischargea62 (16.7)19 (13.8)16 (13.1)27 (24.1)0.04
Laboratory results at time of dischargea164 (43.9)63 (45.3)50 (40.7)51 (45.5)0.68
Back to baseline or other nonspecific remark about discharge statusa71 (19.0)30 (21.6)18 (14.8)23 (20.4)0.34
Any test or result still pending or specific comment that nothing is pendingb46 (12.2)9 (6.4)20 (16.3)17 (14.9)0.03
Recommendation for follow‐up tests/procedures157 (41.9)43 (30.9)54 (43.9)60 (53.1)0.002
Call‐back number of responsible in‐house physicianb23 (6.2)4 (2.9)2 (1.6)17 (15.2)<0.001
Resuscitation status27 (7.7)2 (1.5)18 (15.4)7 (6.7)<0.001
Etiology of heart failurec120 (82.8)44 (81.5)34 (87.2)42 (80.8)0.69
Reason/trigger for exacerbationc86 (58.9)30 (55.6)27 (67.5)29 (55.8)0.43
Ejection fractionc107 (73.3)40 (74.1)32 (80.0)35 (67.3)0.39
Discharge weightc6 (4.1)1 (1.9)1 (2.5)4 (7.7)0.33
Target weight rangec5 (3.4)0 (0)2 (5.0)3 (5.8)0.22
Discharge creatinine or GFRc34 (23.3)14 (25.9)10 (25.0)10 (19.2)0.69
If stent placed, whether drug‐eluting or notd89 (81.7)58 (87.9)27 (81.8)4 (40.0)0.001

On average, summaries included 5.6 of the 6 Joint Commission elements and 4.0 of the 7 TOCCC elements. A total of 63.0% of discharge summaries included all 6 elements required by The Joint Commission, whereas no discharge summary included all 7 TOCCC elements.

APRNs, housestaff and hospitalists included the same average number of The Joint Commission elements (5.6 each), but hospitalists on average included slightly more TOCCC elements (4.3) than did housestaff (4.0) or APRNs (3.8) (P<0.001). Summaries dictated on the day of discharge included an average of 4.2 TOCCC elements, compared to 3.9 TOCCC elements in delayed discharge. In multivariable analyses adjusted for diagnosis and length of stay, there was still no difference by training level in presence of The Joint Commission elements, but hospitalists were significantly more likely to include more TOCCC elements than APRNs (OR: 2.70; 95% CI: 1.49‐4.90) (Table 4). Summaries dictated on the day of discharge were significantly more likely to include more TOCCC elements (OR: 1.92; 95% CI: 1.23‐2.99).

Proportional Odds Model of Associations With Including More Elements Recommended by Specialty Societies (N=376)
Explanatory VariableAverage Number of TOCCC Elements IncludedOR (95% CI)Adjusted P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; CI, confidence interval; OR, odds ratio; TOCCC, Transitions of Care Consensus Conference (defined by Snow et al.[13]).

  • Patients could be categorized as having more than 1 eligible diagnosis.

Training level  0.004
APRN3.8REF 
Housestaff4.01.54 (0.90‐2.62) 
Hospitalist4.32.70 (1.49‐4.90) 
Timeliness   
Dictated after discharge3.9REF 
Dictated day of discharge4.21.92 (1.23‐2.99)0.004
Acute coronary syndrome vs nota3.90.72 (0.37‐1.39)0.33
Pneumonia vs nota4.21.02 (0.49‐2.14)0.95
Heart failure vs nota4.11.49 (0.80‐2.78)0.21
Length of stay, d 0.99 (0.90‐1.07)0.73

No discharge summary included all 7 TOCCC‐endorsed content elements, was dictated on the day of discharge, and was sent to an outside physician.

DISCUSSION

In this prospective single‐site study of medical patients with 3 common conditions, we found that discharge summaries were completed relatively promptly, but were often not sent to the appropriate outpatient physicians. We also found that summaries were uniformly excellent at providing details of the hospitalization, but less reliable at providing details relevant to transitional care such as the patient's condition on discharge or existence of pending tests. On average, summaries included 57% of the elements included in consensus guidelines by 6 major medical societies. The content of discharge summaries dictated by hospitalists was slightly more comprehensive than that of APRNs and trainees, but no group exhibited high performance. In fact, not one discharge summary fully met all 3 quality criteria of timeliness, transmission, and content.

Our study, unlike most in the field, focused on multiple dimensions of discharge summary quality simultaneously. For instance, previous studies have found that timely receipt of a discharge summary does not reduce readmission rates.[11, 14, 15] Yet, if the content of the discharge summary is inadequate for postdischarge care, the summary may not be useful even if it is received by the follow‐up visit. Conversely, high‐quality content is ineffective if the summary is not sent to the outpatient physician.

This study suggests several avenues for improving summary quality. Timely discharge summaries in this study were more likely to include key content and to be transmitted to the appropriate physician. Strategies to improve discharge summary quality should therefore prioritize timely summaries, which can be expected to have downstream benefits for other aspects of quality. Some studies have found that templates improve discharge summary content.[22] In our institution, a template exists, but it favors a hospitalization‐focused rather than transition‐focused approach to the discharge summary. For instance, it includes instructions to dictate the admission exam, but not the discharge exam. Thus, designing templates specifically for transitional care is key. Maximizing capabilities of electronic records may help; many content elements that were commonly missing (e.g., pending results, discharge vitals, discharge weight) could be automatically inserted from electronic records. Likewise, automatic transmission of the summary to care providers listed in the electronic record might ameliorate many transmission failures. Some efforts have been made to convert existing electronic data into discharge summaries.[23, 24, 25] However, these activities are very preliminary, and some studies have found the quality of electronic summaries to be lower than dictated or handwritten summaries.[26] As with all automated or electronic applications, it will be essential to consider workflow, readability, and ability to synthesize information prior to adoption.

Hospitalists consistently produced highest‐quality summaries, even though they did not receive explicit training, suggesting experience may be beneficial,[27, 28, 29] or that the hospitalist community focus on transitional care has been effective. In addition, hospitalists at our institution explicitly prioritize timely and comprehensive discharge dictations, because their business relies on maintaining good relationships with outpatient physicians who contract for their services. Housestaff and APRNs have no such incentives or policies; rather, they typically consider discharge summaries to be a useful source of patient history at the time of an admission or readmission. Other academic centers have found similar results.[6, 16] Nonetheless, even though hospitalists had slightly better performance in our study, large gaps in the quality of summaries remained for all groups including hospitalists.

This study has several limitations. First, as a single‐site study at an academic hospital, it may not be generalizable to other hospitals or other settings. It is noteworthy, however, that the average time to dictation in this study was much lower than that of other studies,[4, 14, 30, 31, 32] suggesting that practices at this institution are at least no worse and possibly better than elsewhere. Second, although there are some mandates and expert opinion‐based guidelines for discharge summary content, there is no validated evidence base to confirm what content ought to be present in discharge summaries to improve patient outcomes. Third, we had too few readmissions in the dataset to have enough power to determine whether discharge summary content, timeliness, or transmission predicts readmission. Fourth, we did not determine whether the information in discharge summaries was accurate or complete; we merely assessed whether it was present. For example, we gave every discharge summary full credit for including discharge medications because they are automatically appended. Yet medication reconciliation errors at discharge are common.[33, 34] In fact, in the DISCHARGE study cohort, more than a quarter of discharge medication lists contained a suspected error.[35]

In summary, this study demonstrated the inadequacy of the contemporary discharge summary for conveying information that is critical to the transition from hospital to home. It may be that hospital culture treats hospitalizations as discrete and self‐contained events rather than as components of a larger episode of care. As interest in reducing readmissions rises, reframing the discharge summary to serve as a transitional tool and targeting it for quality assessment will likely be necessary.

Acknowledgments

The authors would like to acknowledge Amy Browning and the staff of the Center for Outcomes Research and Evaluation Follow‐Up Center for conducting patient interviews, Mark Abroms and Katherine Herman for patient recruitment and screening, and Peter Charpentier for Web site development.

Disclosures

At the time this study was conducted, Dr. Horwitz was supported by the CTSA Grant UL1 RR024139 and KL2 RR024138 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH), and NIH roadmap for Medical Research, and was a Centers of Excellence Scholar in Geriatric Medicine by the John A. Hartford Foundation and the American Federation for Aging Research. Dr. Horwitz is now supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. This work was also supported by a grant from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). Dr. Krumholz is supported by grant U01 HL105270‐01 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute. No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging, the National Center for Advancing Translational Sciences, the National Institutes of Health, The John A. Hartford Foundation, the National Heart, Lung, and Blood Institute, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as an oral presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 12, 2012. Dr. Krumholz chairs a cardiac scientific advisory board for UnitedHealth. Dr. Krumholz receives support from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures that are used for public reporting, including readmission measures.

APPENDIX

A

Dictation guidelines provided to house staff and hospitalists

DICTATION GUIDELINES

FORMAT OF DISCHARGE SUMMARY

 

  • Your name(spell it out), andPatient name(spell it out as well)
  • Medical record number, date of admission, date of discharge
  • Attending physician
  • Disposition
  • Principal and other diagnoses, Principal and other operations/procedures
  • Copies to be sent to other physicians
  • Begin narrative: CC, HPI, PMHx, Medications on admit, Social, Family Hx, Physical exam on admission, Data (labs on admission, plus labs relevant to workup, significant changes at discharge, admission EKG, radiologic and other data),Hospital course by problem, discharge meds, follow‐up appointments

 

APPENDIX

B

 

Content Items Abstracted
Diagnosis
Discharge Second Diagnosis
Hospital course
Procedures/tests performed during admission
Patient and Family Instructions
Social support or living situation of patient
Functional capacity at discharge
Cognitive capacity at discharge
Physical exam at discharge
Laboratory results at time of discharge
Back to baseline or other nonspecific remark about discharge status
Any test or result still pending
Specific comment that nothing is pending
Recommendation for follow up tests/procedures
Call back number of responsible in‐house physician
Resuscitation status
Etiology of heart failure
Reason/trigger for exacerbation
Ejection fraction
Discharge weight
Target weight range
Discharge creatinine or GFR
If stent placed, whether drug‐eluting or not
Joint Commission Composite Elements
Composite elementData elements abstracted that qualify as meeting measure
Reason for hospitalizationDiagnosis
Significant findingsHospital course
Procedures and treatment providedProcedures/tests performed during admission
Patient's discharge conditionFunctional capacity at discharge, Cognitive capacity at discharge, Physical exam at discharge, Laboratory results at time of discharge, Back to baseline or other nonspecific remark about discharge status
Patient and family instructionsSigns and symptoms to monitor at home
Attending physician's signatureAttending signature
Transitions of Care Consensus Conference Composite Elements
Composite elementData elements abstracted that qualify as meeting measure
Principal diagnosisDiagnosis
Problem listDischarge second diagnosis
Medication list[Automatically appended; full credit to every summary]
Transferring physician name and contact informationCall back number of responsible in‐house physician
Cognitive status of the patientCognitive capacity at discharge
Test resultsProcedures/tests performed during admission
Pending test resultsAny test or result still pending or specific comment that nothing is pending

APPENDIX

C

Histogram of days between discharge and dictation

 

 

 

Hospitalized patients are often cared for by physicians who do not follow them in the community, creating a discontinuity of care that must be bridged through communication. This communication between inpatient and outpatient physicians occurs, in part via a discharge summary, which is intended to summarize events during hospitalization and prepare the outpatient physician to resume care of the patient. Yet, this form of communication has long been problematic.[1, 2, 3] In a 1960 study, only 30% of discharge letters were received by the primary care physician within 48 hours of discharge.[1]

More recent studies have shown little improvement. Direct communication between hospital and outpatient physicians is rare, and discharge summaries are still largely unavailable at the time of follow‐up.[4] In 1 study, primary care physicians were unaware of 62% of laboratory tests or study results that were pending on discharge,[5] in part because this information is missing from most discharge summaries.[6] Deficits such as these persist despite the fact that the rate of postdischarge completion of recommended tests, referrals, or procedures is significantly increased when the recommendation is included in the discharge summary.[7]

Regulatory mandates for discharge summaries from the Centers for Medicare and Medicaid Services[8] and from The Joint Commission[9] appear to be generally met[10, 11]; however, these mandates have no requirements for timeliness stricter than 30 days, do not require that summaries be transmitted to outpatient physicians, and do not require several content elements that might be useful to outside physicians such as condition of the patient at discharge, cognitive and functional status, goals of care, or pending studies. Expert opinion guidelines have more comprehensive recommendations,[12, 13] but it is uncertain how widely they are followed.

The existence of a discharge summary does not necessarily mean it serves a patient well in the transitional period.[11, 14, 15] Discharge summaries are a complex intervention, and we do not yet understand the best ways discharge summaries may fulfill needs specific to transitional care. Furthermore, it is uncertain what factors improve aspects of discharge summary quality as defined by timeliness, transmission, and content.[6, 16]

The goal of the DIagnosing Systemic failures, Complexities and HARm in GEriatric discharges study (DISCHARGE) was to comprehensively assess the discharge process for older patients discharged to the community. In this article we examine discharge summaries of patients enrolled in the study to determine the timeliness, transmission to outside physicians, and content of the summaries. We further examine the effect of provider training level and timeliness of dictation on discharge summary quality.

METHODS

Study Cohort

The DISCHARGE study was a prospective, observational cohort study of patients 65 years or older discharged to home from YaleNew Haven Hospital (YNHH) who were admitted with acute coronary syndrome (ACS), community‐acquired pneumonia, or heart failure (HF). Patients were screened by physicians for eligibility within 24 hours of admission using specialty society guidelines[17, 18, 19, 20] and were enrolled by telephone within 1 week of discharge. Additional inclusion criteria included speaking English or Spanish, and ability of the patient or caregiver to participate in a telephone interview. Patients enrolled in hospice were excluded, as were patients who failed the Mini‐Cog mental status screen (3‐item recall and a clock draw)[21] while in the hospital or appeared confused or delirious during the telephone interview. Caregivers of cognitively impaired patients were eligible for enrollment instead if the patient provided permission.

Study Setting

YNHH is a 966‐bed urban tertiary care hospital with statistically lower than the national average mortality for acute myocardial infarction, HF, and pneumonia but statistically higher than the national average for 30‐day readmission rates for HF and pneumonia at the time this study was conducted. Advanced practice registered nurses (APRNs) working under the supervision of private or university cardiologists provided care for cardiology service patients. Housestaff under the supervision of university or hospitalist attending physicians, or physician assistants or APRNs under the supervision of hospitalist attending physicians provided care for patients on medical services. Discharge summaries were typically dictated by APRNs for cardiology patients, by 2nd‐ or 3rd‐year residents for housestaff patients, and by hospitalists for hospitalist patients. A dictation guideline was provided to housestaff and hospitalists (see Supporting Information, Appendix 1, in the online version of this article); this guideline suggested including basic demographic information, disposition and diagnoses, the admission history and physical, hospital course, discharge medications, and follow‐up appointments. Additionally, housestaff received a lecture about discharge summaries at the start of their 2nd year. Discharge instructions including medications and follow‐up appointment information were automatically appended to the discharge summaries. Summaries were sent by the medical records department only to physicians in the system who were listed by the dictating physician as needing to receive a copy of the summary; no summary was automatically sent (ie, to the primary care physician) if not requested by the dictating physician.

Data Collection

Experienced registered nurses trained in chart abstraction conducted explicit reviews of medical charts using a standardized review tool. The tool included 24 questions about the discharge summary applicable to all 3 conditions, with 7 additional questions for patients with HF and 1 additional question for patients with ACS. These questions included the 6 elements required by The Joint Commission for all discharge summaries (reason for hospitalization, significant findings, procedures and treatment provided, patient's discharge condition, patient and family instructions, and attending physician's signature)[9] as well as the 7 elements (principal diagnosis and problem list, medication list, transferring physician name and contact information, cognitive status of the patient, test results, and pending test results) recommended by the Transitions of Care Consensus Conference (TOCCC), a recent consensus statement produced by 6 major medical societies.[13] Each content element is shown in (see Supporting Information, Appendix 2, in the online version of this article), which also indicates the elements included in the 2 guidelines.

Main Measures

We assessed quality in 3 main domains: timeliness, transmission, and content. We defined timeliness as days between discharge date and dictation date (not final signature date, which may occur later), and measured both median timeliness and proportion of discharge summaries completed on the day of discharge. We defined transmission as successful fax or mail of the discharge summary to an outside physician as reported by the medical records department, and measured the proportion of discharge summaries sent to any outside physician as well as the median number of physicians per discharge summary who were scheduled to follow‐up with the patient postdischarge but who did not receive a copy of the summary. We defined 21 individual content items and assessed the frequency of each individual content item. We also measured compliance with The Joint Commission mandates and TOCCC recommendations, which included several of the individual content items.

To measure compliance with The Joint Commission requirements, we created a composite score in which 1 point was provided for the presence of each of the 6 required elements (maximum score=6). Every discharge summary received 1 point for attending physician signature, because all discharge summaries were electronically signed. Discharge instructions to family/patients were automatically appended to every discharge summary; however, we gave credit for patient and family instructions only to those that included any information about signs and symptoms to monitor for at home. We defined discharge condition as any information about functional status, cognitive status, physical exam, or laboratory findings at discharge.

To measure compliance with specialty society recommendations for discharge summaries, we created a composite score in which 1 point was provided for the presence of each of the 7 recommended elements (maximum score=7). Every discharge summary received 1 point for discharge medications, because these are automatically appended.

We obtained data on age, race, gender, and length of stay from hospital administrative databases. The study was approved by the Yale Human Investigation Committee, and verbal informed consent was obtained from all study participants.

Statistical Analysis

Characteristics of the sample are described with counts and percentages or means and standard deviations. Medians and interquartile ranges (IQRs) or counts and percentages were calculated for summary measures of timeliness, transmission, and content. We assessed differences in quality measures between APRNs, housestaff, and hospitalists using 2 tests. We conducted multivariable logistic regression analyses for timeliness and for transmission to any outside physician. All discharge summaries included at least 4 of The Joint Commission elements; consequently, we coded this content outcome as an ordinal variable with 3 levels indicating inclusion of 4, 5, or 6 of The Joint Commission elements. We coded the TOCCC content outcome as a 3‐level variable indicating <4, 4, or >4 elements satisfied. Accordingly, proportional odds models were used, in which the reported odds ratios (ORs) can be interpreted as the average effect of the explanatory variable on the odds of having more recommendations, for any dichotomization of the outcome. Residual analysis and goodness‐of‐fit statistics were used to assess model fit; the proportional odds assumption was tested. Statistical analyses were conducted with SAS 9.2 (SAS Institute, Cary, NC). P values <0.05 were interpreted as statistically significant for 2‐sided tests.

RESULTS

Enrollment and Study Sample

A total of 3743 patients over 64 years old were discharged home from the medical service at YNHH during the study period; 3028 patients were screened for eligibility within 24 hours of admission. We identified 635 eligible admissions and enrolled 395 patients (62.2%) in the study. Of these, 377 granted permission for chart review and were included in this analysis (Figure 1).

Figure 1
Flow diagram of enrolled participants.

The study sample had a mean age of 77.1 years (standard deviation: 7.8); 205 (54.4%) were male and 310 (82.5%) were non‐Hispanic white. A total of 195 (51.7%) had ACS, 91 (24.1%) had pneumonia, and 146 (38.7%) had HF; 54 (14.3%) patients had more than 1 qualifying condition. There were similar numbers of patients on the cardiology, medicine housestaff, and medicine hospitalist teams (Table 1).

Study Sample Characteristics (N=377)
CharacteristicN (%) or Mean (SD)
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; N=number of study participants; GED, general educational development; SD=standard deviation.

Condition 
Acute coronary syndrome195 (51.7)
Community‐acquired pneumonia91 (24.1)
Heart failure146 (38.7)
Training level of summary dictator 
APRN140 (37.1)
House staff123 (32.6)
Hospitalist114 (30.2)
Length of stay, mean, d3.5 (2.5)
Total number of medications8.9 (3.3)
Identify a usual source of care360 (96.0)
Age, mean, y77.1 (7.8)
Male205 (54.4)
English‐speaking366 (98.1)
Race/ethnicity 
Non‐Hispanic white310 (82.5)
Non‐Hispanic black44 (11.7)
Hispanic15 (4.0)
Other7 (1.9)
High school graduate or GED Admission source268 (73.4)
Emergency department248 (66.0)
Direct transfer from hospital or nursing facility94 (25.0)
Direct admission from office34 (9.0)

Timeliness

Discharge summaries were completed for 376/377 patients, of which 174 (46.3%) were dictated on the day of discharge. However, 122 (32.4%) summaries were dictated more than 48 hours after discharge, including 93 (24.7%) that were dictated more than 1 week after discharge (see Supporting Information, Appendix 3, in the online version of this article).

Summaries dictated by hospitalists were most likely to be done on the day of discharge (35.3% APRNs, 38.2% housestaff, 68.4% hospitalists, P<0.001). After adjustment for diagnosis and length of stay, hospitalists were still significantly more likely to produce a timely discharge summary than APRNs (OR: 2.82; 95% confidence interval [CI]: 1.56‐5.09), whereas housestaff were no different than APRNs (OR: 0.84; 95% CI: 0.48‐1.46).

Transmission

A total of 144 (38.3%) discharge summaries were not sent to any physician besides the inpatient attending, and 209/374 (55.9%) were not sent to at least 1 physician listed as having a follow‐up appointment planned with the patient. Each discharge summary was sent to a median of 1 physician besides the dictating physician (IQR: 01). However, for each summary, a median of 1 physician (IQR: 01) who had a scheduled follow‐up with the patient did not receive the summary. Summaries dictated by hospitalists were most likely to be sent to at least 1 outside physician (54.7% APRNs, 58.5% housestaff, 73.7% hospitalists, P=0.006). Summaries dictated on the day of discharge were more likely than delayed summaries to be sent to at least 1 outside physician (75.9% vs 49.5%, P<0.001). After adjustment for diagnosis and length of stay, there was no longer a difference in likelihood of transmitting a discharge summary to any outpatient physician according to training level; however, dictations completed on the day of discharge remained significantly more likely to be transmitted to an outside physician (OR: 3.05; 95% CI: 1.88‐4.93) (Table 2).

Logistic Regression Model of Associations With Discharge Summary Transmission (N=376)
Explanatory VariableProportion Transmitted to at Least 1 Outside PhysicianOR for Transmission to Any Outside Physician (95% CI)Adjusted P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; CI, confidence interval; OR, odds ratio.

  • Patients could be categorized as having more than 1 eligible diagnosis.

Training level  0.52
APRN54.7%REF 
Housestaff58.5%1.17 (0.66‐2.06) 
Hospitalist73.7%1.46 (0.76‐2.79) 
Timeliness   
Dictated after discharge49.5%REF<0.001
Dictated day of discharge75.9%3.05 (1.88‐4.93) 
Acute coronary syndrome vs nota52.1 %1.05 (0.49‐2.26)0.89
Pneumonia vs nota69.2 %1.59 (0.66‐3.79)0.30
Heart failure vs nota74.7 %3.32 (1.61‐6.84)0.001
Length of stay, d 0.91 (0.83‐1.00)0.06

Content

Rate of inclusion of each content element is shown in Table 3, overall and by training level. Nearly every discharge summary included information about admitting diagnosis, hospital course, and procedures or tests performed during the hospitalization. However, few summaries included information about the patient's condition at discharge. Less than half included discharge laboratory results; less than one‐third included functional capacity, cognitive capacity, or discharge physical exam. Only 4.1% overall of discharge summaries for patients with HF included the patient's weight at discharge; best were hospitalists who still included this information in only 7.7% of summaries. Information about postdischarge care, including home social support, pending tests, or recommended follow‐up tests/procedures was also rarely specified. Last, only 6.2% of discharge summaries included the name and contact number of the inpatient physician; this information was least likely to be provided by housestaff (1.6%) and most likely to be provided by hospitalists (15.2%) (P<0.001).

Content of Discharge SummariesOverall and by Training Level
Discharge Summary ComponentOverall, n=377, n (%)APRN, n=140, n (%)Housestaff, n=123, n (%)Hospitalist, n=114, n (%)P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; GFR, glomerular filtration rate.

  • Included in Joint Commission composite.

  • Included in Transitions of Care Consensus Conference composite.

  • Patients with heart failure only (n=146).

  • Patients with stents placed only (n=109).

Diagnosisab368 (97.9)136 (97.8)120 (97.6)112 (98.3)1.00
Discharge second diagnosisb289 (76.9)100 (71.9)89 (72.4)100 (87.7)<0.001
Hospital coursea375 (100.0)138 (100)123 (100)114 (100)N/A
Procedures/tests performed during admissionab374 (99.7)138 (99.3)123 (100)113 (100)N/A
Patient and family instructionsa371 (98.4)136 (97.1)122 (99.2)113 (99.1).43
Social support or living situation of patient148 (39.5)18 (12.9)62 (50.4)68 (60.2)<0.001
Functional capacity at dischargea99 (26.4)37 (26.6)32 (26.0)30 (26.6)0.99
Cognitive capacity at dischargeab30 (8.0)6 (4.4)11 (8.9)13 (11.5)0.10
Physical exam at dischargea62 (16.7)19 (13.8)16 (13.1)27 (24.1)0.04
Laboratory results at time of dischargea164 (43.9)63 (45.3)50 (40.7)51 (45.5)0.68
Back to baseline or other nonspecific remark about discharge statusa71 (19.0)30 (21.6)18 (14.8)23 (20.4)0.34
Any test or result still pending or specific comment that nothing is pendingb46 (12.2)9 (6.4)20 (16.3)17 (14.9)0.03
Recommendation for follow‐up tests/procedures157 (41.9)43 (30.9)54 (43.9)60 (53.1)0.002
Call‐back number of responsible in‐house physicianb23 (6.2)4 (2.9)2 (1.6)17 (15.2)<0.001
Resuscitation status27 (7.7)2 (1.5)18 (15.4)7 (6.7)<0.001
Etiology of heart failurec120 (82.8)44 (81.5)34 (87.2)42 (80.8)0.69
Reason/trigger for exacerbationc86 (58.9)30 (55.6)27 (67.5)29 (55.8)0.43
Ejection fractionc107 (73.3)40 (74.1)32 (80.0)35 (67.3)0.39
Discharge weightc6 (4.1)1 (1.9)1 (2.5)4 (7.7)0.33
Target weight rangec5 (3.4)0 (0)2 (5.0)3 (5.8)0.22
Discharge creatinine or GFRc34 (23.3)14 (25.9)10 (25.0)10 (19.2)0.69
If stent placed, whether drug‐eluting or notd89 (81.7)58 (87.9)27 (81.8)4 (40.0)0.001

On average, summaries included 5.6 of the 6 Joint Commission elements and 4.0 of the 7 TOCCC elements. A total of 63.0% of discharge summaries included all 6 elements required by The Joint Commission, whereas no discharge summary included all 7 TOCCC elements.

APRNs, housestaff and hospitalists included the same average number of The Joint Commission elements (5.6 each), but hospitalists on average included slightly more TOCCC elements (4.3) than did housestaff (4.0) or APRNs (3.8) (P<0.001). Summaries dictated on the day of discharge included an average of 4.2 TOCCC elements, compared to 3.9 TOCCC elements in delayed discharge. In multivariable analyses adjusted for diagnosis and length of stay, there was still no difference by training level in presence of The Joint Commission elements, but hospitalists were significantly more likely to include more TOCCC elements than APRNs (OR: 2.70; 95% CI: 1.49‐4.90) (Table 4). Summaries dictated on the day of discharge were significantly more likely to include more TOCCC elements (OR: 1.92; 95% CI: 1.23‐2.99).

Proportional Odds Model of Associations With Including More Elements Recommended by Specialty Societies (N=376)
Explanatory VariableAverage Number of TOCCC Elements IncludedOR (95% CI)Adjusted P Value
  • NOTE: Abbreviations: APRN, advanced practice registered nurse; CI, confidence interval; OR, odds ratio; TOCCC, Transitions of Care Consensus Conference (defined by Snow et al.[13]).

  • Patients could be categorized as having more than 1 eligible diagnosis.

Training level  0.004
APRN3.8REF 
Housestaff4.01.54 (0.90‐2.62) 
Hospitalist4.32.70 (1.49‐4.90) 
Timeliness   
Dictated after discharge3.9REF 
Dictated day of discharge4.21.92 (1.23‐2.99)0.004
Acute coronary syndrome vs nota3.90.72 (0.37‐1.39)0.33
Pneumonia vs nota4.21.02 (0.49‐2.14)0.95
Heart failure vs nota4.11.49 (0.80‐2.78)0.21
Length of stay, d 0.99 (0.90‐1.07)0.73

No discharge summary included all 7 TOCCC‐endorsed content elements, was dictated on the day of discharge, and was sent to an outside physician.

DISCUSSION

In this prospective single‐site study of medical patients with 3 common conditions, we found that discharge summaries were completed relatively promptly, but were often not sent to the appropriate outpatient physicians. We also found that summaries were uniformly excellent at providing details of the hospitalization, but less reliable at providing details relevant to transitional care such as the patient's condition on discharge or existence of pending tests. On average, summaries included 57% of the elements included in consensus guidelines by 6 major medical societies. The content of discharge summaries dictated by hospitalists was slightly more comprehensive than that of APRNs and trainees, but no group exhibited high performance. In fact, not one discharge summary fully met all 3 quality criteria of timeliness, transmission, and content.

Our study, unlike most in the field, focused on multiple dimensions of discharge summary quality simultaneously. For instance, previous studies have found that timely receipt of a discharge summary does not reduce readmission rates.[11, 14, 15] Yet, if the content of the discharge summary is inadequate for postdischarge care, the summary may not be useful even if it is received by the follow‐up visit. Conversely, high‐quality content is ineffective if the summary is not sent to the outpatient physician.

This study suggests several avenues for improving summary quality. Timely discharge summaries in this study were more likely to include key content and to be transmitted to the appropriate physician. Strategies to improve discharge summary quality should therefore prioritize timely summaries, which can be expected to have downstream benefits for other aspects of quality. Some studies have found that templates improve discharge summary content.[22] In our institution, a template exists, but it favors a hospitalization‐focused rather than transition‐focused approach to the discharge summary. For instance, it includes instructions to dictate the admission exam, but not the discharge exam. Thus, designing templates specifically for transitional care is key. Maximizing capabilities of electronic records may help; many content elements that were commonly missing (e.g., pending results, discharge vitals, discharge weight) could be automatically inserted from electronic records. Likewise, automatic transmission of the summary to care providers listed in the electronic record might ameliorate many transmission failures. Some efforts have been made to convert existing electronic data into discharge summaries.[23, 24, 25] However, these activities are very preliminary, and some studies have found the quality of electronic summaries to be lower than dictated or handwritten summaries.[26] As with all automated or electronic applications, it will be essential to consider workflow, readability, and ability to synthesize information prior to adoption.

Hospitalists consistently produced highest‐quality summaries, even though they did not receive explicit training, suggesting experience may be beneficial,[27, 28, 29] or that the hospitalist community focus on transitional care has been effective. In addition, hospitalists at our institution explicitly prioritize timely and comprehensive discharge dictations, because their business relies on maintaining good relationships with outpatient physicians who contract for their services. Housestaff and APRNs have no such incentives or policies; rather, they typically consider discharge summaries to be a useful source of patient history at the time of an admission or readmission. Other academic centers have found similar results.[6, 16] Nonetheless, even though hospitalists had slightly better performance in our study, large gaps in the quality of summaries remained for all groups including hospitalists.

This study has several limitations. First, as a single‐site study at an academic hospital, it may not be generalizable to other hospitals or other settings. It is noteworthy, however, that the average time to dictation in this study was much lower than that of other studies,[4, 14, 30, 31, 32] suggesting that practices at this institution are at least no worse and possibly better than elsewhere. Second, although there are some mandates and expert opinion‐based guidelines for discharge summary content, there is no validated evidence base to confirm what content ought to be present in discharge summaries to improve patient outcomes. Third, we had too few readmissions in the dataset to have enough power to determine whether discharge summary content, timeliness, or transmission predicts readmission. Fourth, we did not determine whether the information in discharge summaries was accurate or complete; we merely assessed whether it was present. For example, we gave every discharge summary full credit for including discharge medications because they are automatically appended. Yet medication reconciliation errors at discharge are common.[33, 34] In fact, in the DISCHARGE study cohort, more than a quarter of discharge medication lists contained a suspected error.[35]

In summary, this study demonstrated the inadequacy of the contemporary discharge summary for conveying information that is critical to the transition from hospital to home. It may be that hospital culture treats hospitalizations as discrete and self‐contained events rather than as components of a larger episode of care. As interest in reducing readmissions rises, reframing the discharge summary to serve as a transitional tool and targeting it for quality assessment will likely be necessary.

Acknowledgments

The authors would like to acknowledge Amy Browning and the staff of the Center for Outcomes Research and Evaluation Follow‐Up Center for conducting patient interviews, Mark Abroms and Katherine Herman for patient recruitment and screening, and Peter Charpentier for Web site development.

Disclosures

At the time this study was conducted, Dr. Horwitz was supported by the CTSA Grant UL1 RR024139 and KL2 RR024138 from the National Center for Advancing Translational Sciences (NCATS), a component of the National Institutes of Health (NIH), and NIH roadmap for Medical Research, and was a Centers of Excellence Scholar in Geriatric Medicine by the John A. Hartford Foundation and the American Federation for Aging Research. Dr. Horwitz is now supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. This work was also supported by a grant from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). Dr. Krumholz is supported by grant U01 HL105270‐01 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute. No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging, the National Center for Advancing Translational Sciences, the National Institutes of Health, The John A. Hartford Foundation, the National Heart, Lung, and Blood Institute, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as an oral presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 12, 2012. Dr. Krumholz chairs a cardiac scientific advisory board for UnitedHealth. Dr. Krumholz receives support from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures that are used for public reporting, including readmission measures.

APPENDIX

A

Dictation guidelines provided to house staff and hospitalists

DICTATION GUIDELINES

FORMAT OF DISCHARGE SUMMARY

 

  • Your name(spell it out), andPatient name(spell it out as well)
  • Medical record number, date of admission, date of discharge
  • Attending physician
  • Disposition
  • Principal and other diagnoses, Principal and other operations/procedures
  • Copies to be sent to other physicians
  • Begin narrative: CC, HPI, PMHx, Medications on admit, Social, Family Hx, Physical exam on admission, Data (labs on admission, plus labs relevant to workup, significant changes at discharge, admission EKG, radiologic and other data),Hospital course by problem, discharge meds, follow‐up appointments

 

APPENDIX

B

 

Content Items Abstracted
Diagnosis
Discharge Second Diagnosis
Hospital course
Procedures/tests performed during admission
Patient and Family Instructions
Social support or living situation of patient
Functional capacity at discharge
Cognitive capacity at discharge
Physical exam at discharge
Laboratory results at time of discharge
Back to baseline or other nonspecific remark about discharge status
Any test or result still pending
Specific comment that nothing is pending
Recommendation for follow up tests/procedures
Call back number of responsible in‐house physician
Resuscitation status
Etiology of heart failure
Reason/trigger for exacerbation
Ejection fraction
Discharge weight
Target weight range
Discharge creatinine or GFR
If stent placed, whether drug‐eluting or not
Joint Commission Composite Elements
Composite elementData elements abstracted that qualify as meeting measure
Reason for hospitalizationDiagnosis
Significant findingsHospital course
Procedures and treatment providedProcedures/tests performed during admission
Patient's discharge conditionFunctional capacity at discharge, Cognitive capacity at discharge, Physical exam at discharge, Laboratory results at time of discharge, Back to baseline or other nonspecific remark about discharge status
Patient and family instructionsSigns and symptoms to monitor at home
Attending physician's signatureAttending signature
Transitions of Care Consensus Conference Composite Elements
Composite elementData elements abstracted that qualify as meeting measure
Principal diagnosisDiagnosis
Problem listDischarge second diagnosis
Medication list[Automatically appended; full credit to every summary]
Transferring physician name and contact informationCall back number of responsible in‐house physician
Cognitive status of the patientCognitive capacity at discharge
Test resultsProcedures/tests performed during admission
Pending test resultsAny test or result still pending or specific comment that nothing is pending

APPENDIX

C

Histogram of days between discharge and dictation

 

 

 

References
  1. Alarcon R, Glanville H, Hodson JM. Value of the specialist's report. Br Med J. 1960;2(5213):16631664.
  2. Long A, Atkins JB. Communications between general practitioners and consultants. Br Med J. 1974;4(5942):456459.
  3. Swender PT, Schneider AJ, Oski FA. A functional hospital discharge summary. J Pediatr. 1975;86(1):9798.
  4. Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital‐based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007;297(8):831841.
  5. Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143(2):121128.
  6. Were MC, Li X, Kesterson J, et al. Adequacy of hospital discharge summaries in documenting tests with pending results and outpatient follow‐up providers. J Gen Intern Med. 2009;24(9):10021006.
  7. Moore C, McGinn T, Halm E. Tying up loose ends: discharging patients with unresolved medical issues. Arch Intern Med. 2007;167(12):13051311.
  8. Centers for Medicare and Medicaid Services. Condition of participation: medical record services. 42. Vol 482.C.F.R. § 482.24 (2012).
  9. Joint Commission on Accreditation of Healthcare Organizations. Hospital Accreditation Standards. Standard IM 6.10 EP 7–9. Oakbrook Terrace, IL: The Joint Commission; 2008.
  10. Kind AJH, Smith MA. Documentation of mandated discharge summary components in transitions from acute to subacute care. In: Agency for Healthcare Research and Quality, ed. Advances in Patient Safety: New Directions and Alternative Approaches. Vol 2: Culture and Redesign. AHRQ Publication No. 08-0034‐2. Rockville, MD: Agency for Healthcare Research and Quality; 2008:179–188.
  11. Hansen LO, Strater A, Smith L, et al. Hospital discharge documentation and risk of rehospitalisation. BMJ Qual Saf. 2011;20(9):773778.
  12. Halasyamani L, Kripalani S, Coleman E, et al. Transition of care for hospitalized elderly patients‐development of a discharge checklist for hospitalists. J Hosp Med. 2006;1(6):354360.
  13. Snow V, Beck D, Budnitz T, et al. Transitions of Care Consensus Policy Statement American College of Physicians‐Society of General Internal Medicine‐Society of Hospital Medicine‐American Geriatrics Society‐American College of Emergency Physicians‐Society of Academic Emergency Medicine. J Gen Intern Med. 2009;24(8):971976.
  14. Bell CM, Schnipper JL, Auerbach AD, et al. Association of communication between hospital‐based physicians and primary care providers with patient outcomes. J Gen Intern Med. 2009;24(3):381386.
  15. Walraven C, Seth R, Austin PC, Laupacis A. Effect of discharge summary availability during post‐discharge visits on hospital readmission. J Gen Intern Med. 2002;17(3):186192.
  16. Kind AJ, Thorpe CT, Sattin JA, Walz SE, Smith MA. Provider characteristics, clinical‐work processes and their relationship to discharge summary quality for sub‐acute care patients. J Gen Intern Med. 2012;27(1):7884.
  17. Anderson JL, Adams CD, Antman EM, et al. ACC/AHA 2007 guidelines for the management of patients with unstable angina/non‐ST‐elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines for the Management of Patients With Unstable Angina/Non‐ST‐Elevation Myocardial Infarction) developed in collaboration with the American College of Emergency Physicians, the Society for Cardiovascular Angiography and Interventions, and the Society of Thoracic Surgeons endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation and the Society for Academic Emergency Medicine. J Am Coll Cardiol. 2007;50(7):e1e157.
  18. Thygesen K, Alpert JS, White HD. Universal definition of myocardial infarction. Eur Heart J. 2007;28(20):25252538.
  19. Dickstein K, Cohen‐Solal A, Filippatos G, et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2008: the Task Force for the diagnosis and treatment of acute and chronic heart failure 2008 of the European Society of Cardiology. Developed in collaboration with the Heart Failure Association of the ESC (HFA) and endorsed by the European Society of Intensive Care Medicine (ESICM). Eur J Heart Fail. 2008;10(10):933989.
  20. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community‐acquired pneumonia in adults. Clin Infect Dis. 2007;44(suppl 2):S27S72.
  21. Sunderland T, Hill JL, Mellow AM, et al. Clock drawing in Alzheimer's disease. A novel measure of dementia severity. J Am Geriatr Soc. 1989;37(8):725729.
  22. Rao P, Andrei A, Fried A, Gonzalez D, Shine D. Assessing quality and efficiency of discharge summaries. Am J Med Qual. 2005;20(6):337343.
  23. Maslove DM, Leiter RE, Griesman J, et al. Electronic versus dictated hospital discharge summaries: a randomized controlled trial. J Gen Intern Med. 2009;24(9):9951001.
  24. Walraven C, Laupacis A, Seth R, Wells G. Dictated versus database‐generated discharge summaries: a randomized clinical trial. CMAJ. 1999;160(3):319326.
  25. Llewelyn DE, Ewins DL, Horn J, Evans TG, McGregor AM. Computerised updating of clinical summaries: new opportunities for clinical practice and research? BMJ. 1988;297(6662):15041506.
  26. Callen JL, Alderton M, McIntosh J. Evaluation of electronic discharge summaries: a comparison of documentation in electronic and handwritten discharge summaries. Int J Med Inform. 2008;77(9):613620.
  27. Davis MM, Devoe M, Kansagara D, Nicolaidis C, Englander H. Did I do as best as the system would let me? Healthcare professional views on hospital to home care transitions. J Gen Intern Med. 2012;27(12):16491656.
  28. Greysen SR, Schiliro D, Curry L, Bradley EH, Horwitz LI. Learning by doing—resident perspectives on developing competency in high‐quality discharge care. J Gen Intern Med. 2012;27(9):11881194.
  29. Greysen SR, Schiliro D, Horwitz LI, Curry L, Bradley EH. Out of sight, out of mind: housestaff perceptions of quality‐limiting factors in discharge care at teaching hospitals. J Hosp Med. 2012;7(5):376381.
  30. Walraven C, Seth R, Laupacis A. Dissemination of discharge summaries. Not reaching follow‐up physicians. Can Fam Physician. 2002;48:737742.
  31. Pantilat SZ, Lindenauer PK, Katz PP, Wachter RM. Primary care physician attitudes regarding communication with hospitalists. Am J Med. 2001;111(9B):15S20S.
  32. Wilson S, Ruscoe W, Chapman M, Miller R. General practitioner‐hospital communications: a review of discharge summaries. J Qual Clin Pract. 2001;21(4):104108.
  33. McMillan TE, Allan W, Black PN. Accuracy of information on medicines in hospital discharge summaries. Intern Med J. 2006;36(4):221225.
  34. Callen J, McIntosh J, Li J. Accuracy of medication documentation in hospital discharge summaries: A retrospective analysis of medication transcription errors in manual and electronic discharge summaries. Int J Med Inform. 2010;79(1):5864.
  35. Ziaeian B, Araujo KL, Ness PH, Horwitz LI. Medication reconciliation accuracy and patient understanding of intended medication changes on hospital discharge. J Gen Intern Med. 2012;27(11):15131520.
References
  1. Alarcon R, Glanville H, Hodson JM. Value of the specialist's report. Br Med J. 1960;2(5213):16631664.
  2. Long A, Atkins JB. Communications between general practitioners and consultants. Br Med J. 1974;4(5942):456459.
  3. Swender PT, Schneider AJ, Oski FA. A functional hospital discharge summary. J Pediatr. 1975;86(1):9798.
  4. Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital‐based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007;297(8):831841.
  5. Roy CL, Poon EG, Karson AS, et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med. 2005;143(2):121128.
  6. Were MC, Li X, Kesterson J, et al. Adequacy of hospital discharge summaries in documenting tests with pending results and outpatient follow‐up providers. J Gen Intern Med. 2009;24(9):10021006.
  7. Moore C, McGinn T, Halm E. Tying up loose ends: discharging patients with unresolved medical issues. Arch Intern Med. 2007;167(12):13051311.
  8. Centers for Medicare and Medicaid Services. Condition of participation: medical record services. 42. Vol 482.C.F.R. § 482.24 (2012).
  9. Joint Commission on Accreditation of Healthcare Organizations. Hospital Accreditation Standards. Standard IM 6.10 EP 7–9. Oakbrook Terrace, IL: The Joint Commission; 2008.
  10. Kind AJH, Smith MA. Documentation of mandated discharge summary components in transitions from acute to subacute care. In: Agency for Healthcare Research and Quality, ed. Advances in Patient Safety: New Directions and Alternative Approaches. Vol 2: Culture and Redesign. AHRQ Publication No. 08-0034‐2. Rockville, MD: Agency for Healthcare Research and Quality; 2008:179–188.
  11. Hansen LO, Strater A, Smith L, et al. Hospital discharge documentation and risk of rehospitalisation. BMJ Qual Saf. 2011;20(9):773778.
  12. Halasyamani L, Kripalani S, Coleman E, et al. Transition of care for hospitalized elderly patients‐development of a discharge checklist for hospitalists. J Hosp Med. 2006;1(6):354360.
  13. Snow V, Beck D, Budnitz T, et al. Transitions of Care Consensus Policy Statement American College of Physicians‐Society of General Internal Medicine‐Society of Hospital Medicine‐American Geriatrics Society‐American College of Emergency Physicians‐Society of Academic Emergency Medicine. J Gen Intern Med. 2009;24(8):971976.
  14. Bell CM, Schnipper JL, Auerbach AD, et al. Association of communication between hospital‐based physicians and primary care providers with patient outcomes. J Gen Intern Med. 2009;24(3):381386.
  15. Walraven C, Seth R, Austin PC, Laupacis A. Effect of discharge summary availability during post‐discharge visits on hospital readmission. J Gen Intern Med. 2002;17(3):186192.
  16. Kind AJ, Thorpe CT, Sattin JA, Walz SE, Smith MA. Provider characteristics, clinical‐work processes and their relationship to discharge summary quality for sub‐acute care patients. J Gen Intern Med. 2012;27(1):7884.
  17. Anderson JL, Adams CD, Antman EM, et al. ACC/AHA 2007 guidelines for the management of patients with unstable angina/non‐ST‐elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines for the Management of Patients With Unstable Angina/Non‐ST‐Elevation Myocardial Infarction) developed in collaboration with the American College of Emergency Physicians, the Society for Cardiovascular Angiography and Interventions, and the Society of Thoracic Surgeons endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation and the Society for Academic Emergency Medicine. J Am Coll Cardiol. 2007;50(7):e1e157.
  18. Thygesen K, Alpert JS, White HD. Universal definition of myocardial infarction. Eur Heart J. 2007;28(20):25252538.
  19. Dickstein K, Cohen‐Solal A, Filippatos G, et al. ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2008: the Task Force for the diagnosis and treatment of acute and chronic heart failure 2008 of the European Society of Cardiology. Developed in collaboration with the Heart Failure Association of the ESC (HFA) and endorsed by the European Society of Intensive Care Medicine (ESICM). Eur J Heart Fail. 2008;10(10):933989.
  20. Mandell LA, Wunderink RG, Anzueto A, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community‐acquired pneumonia in adults. Clin Infect Dis. 2007;44(suppl 2):S27S72.
  21. Sunderland T, Hill JL, Mellow AM, et al. Clock drawing in Alzheimer's disease. A novel measure of dementia severity. J Am Geriatr Soc. 1989;37(8):725729.
  22. Rao P, Andrei A, Fried A, Gonzalez D, Shine D. Assessing quality and efficiency of discharge summaries. Am J Med Qual. 2005;20(6):337343.
  23. Maslove DM, Leiter RE, Griesman J, et al. Electronic versus dictated hospital discharge summaries: a randomized controlled trial. J Gen Intern Med. 2009;24(9):9951001.
  24. Walraven C, Laupacis A, Seth R, Wells G. Dictated versus database‐generated discharge summaries: a randomized clinical trial. CMAJ. 1999;160(3):319326.
  25. Llewelyn DE, Ewins DL, Horn J, Evans TG, McGregor AM. Computerised updating of clinical summaries: new opportunities for clinical practice and research? BMJ. 1988;297(6662):15041506.
  26. Callen JL, Alderton M, McIntosh J. Evaluation of electronic discharge summaries: a comparison of documentation in electronic and handwritten discharge summaries. Int J Med Inform. 2008;77(9):613620.
  27. Davis MM, Devoe M, Kansagara D, Nicolaidis C, Englander H. Did I do as best as the system would let me? Healthcare professional views on hospital to home care transitions. J Gen Intern Med. 2012;27(12):16491656.
  28. Greysen SR, Schiliro D, Curry L, Bradley EH, Horwitz LI. Learning by doing—resident perspectives on developing competency in high‐quality discharge care. J Gen Intern Med. 2012;27(9):11881194.
  29. Greysen SR, Schiliro D, Horwitz LI, Curry L, Bradley EH. Out of sight, out of mind: housestaff perceptions of quality‐limiting factors in discharge care at teaching hospitals. J Hosp Med. 2012;7(5):376381.
  30. Walraven C, Seth R, Laupacis A. Dissemination of discharge summaries. Not reaching follow‐up physicians. Can Fam Physician. 2002;48:737742.
  31. Pantilat SZ, Lindenauer PK, Katz PP, Wachter RM. Primary care physician attitudes regarding communication with hospitalists. Am J Med. 2001;111(9B):15S20S.
  32. Wilson S, Ruscoe W, Chapman M, Miller R. General practitioner‐hospital communications: a review of discharge summaries. J Qual Clin Pract. 2001;21(4):104108.
  33. McMillan TE, Allan W, Black PN. Accuracy of information on medicines in hospital discharge summaries. Intern Med J. 2006;36(4):221225.
  34. Callen J, McIntosh J, Li J. Accuracy of medication documentation in hospital discharge summaries: A retrospective analysis of medication transcription errors in manual and electronic discharge summaries. Int J Med Inform. 2010;79(1):5864.
  35. Ziaeian B, Araujo KL, Ness PH, Horwitz LI. Medication reconciliation accuracy and patient understanding of intended medication changes on hospital discharge. J Gen Intern Med. 2012;27(11):15131520.
Issue
Journal of Hospital Medicine - 8(8)
Issue
Journal of Hospital Medicine - 8(8)
Page Number
436-443
Page Number
436-443
Publications
Publications
Article Type
Display Headline
Comprehensive quality of discharge summaries at an academic medical center
Display Headline
Comprehensive quality of discharge summaries at an academic medical center
Sections
Article Source

Copyright © 2013 Society of Hospital Medicine

Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Leora Horwitz, MD, Section of General Internal Medicine, Department of Internal Medicine, Yale School of Medicine, P.O. Box 208093, New Haven, CT 06520-8093; Telephone: 203-688-5678; Fax: 203-737‐3306; E‐mail: leora.horwitz@yale.edu
Content Gating
No Gating (article Unlocked/Free)
Alternative CME
Article PDF Media
Media Files