User login
Handoff CEX
Transfers among trainee physicians within the hospital typically occur at least twice a day and have been increasing among trainees as work hours have declined.[1] The 2011 Accreditation Council for Graduate Medical Education (ACGME) guidelines,[2] which restrict intern working hours to 16 hours from a previous maximum of 30, have likely increased the frequency of physician trainee handoffs even further. Similarly, transfers among hospitalist attendings occur at least twice a day, given typical shifts of 8 to 12 hours.
Given the frequency of transfers, and the potential for harm generated by failed transitions,[3, 4, 5, 6] the end‐of‐shift written and verbal handoffs have assumed increasingly greater importance in hospital care among both trainees and hospitalist attendings.
The ACGME now requires that programs assess the competency of trainees in handoff communication.[2] Yet, there are few tools for assessing the quality of sign‐out communication. Those that exist primarily focus on the written sign‐out, and are rarely validated.[7, 8, 9, 10, 11, 12] Furthermore, it is uncertain whether such assessments must be done by supervisors or whether peers can participate in the evaluation. In this prospective multi‐institutional study we assess the performance characteristics of a verbal sign‐out evaluation tool for internal medicine housestaff and hospitalist attendings, and examine whether it can be used by peers as well as by external evaluators. This tool has previously been found to effectively discriminate between experienced and inexperienced nurses conducting nursing handoffs.[13]
METHODS
Tool Design and Measures
The Handoff CEX (clinical evaluation exercise) is a structured assessment based on the format of the mini‐CEX, an instrument used to assess the quality of history and physical examination by trainees for which validation studies have previously been conducted.[14, 15, 16, 17] We developed the tool based on themes we identified from our own expertise,[1, 5, 6, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29] the ACGME core competencies for trainees,[2] and the literature to maximize content validity. First, standardization has numerous demonstrable benefits for safety in general and handoffs in particular.[30, 31, 32] Consequently we created a domain for organization in which standardization was a characteristic of high performance.
Second, there is evidence that people engaged in conversation routinely overestimate peer comprehension,[27] and that explicit strategies to combat this overestimation, such as confirming understanding, explicitly assigning tasks rather than using open‐ended language, and using concrete language, are effective.[33] Accordingly we created a domain for communication skills, which is also an ACGME competency.
Third, although there were no formal guidelines for sign‐out content when we developed this tool, our own research had demonstrated that the content elements most often missing and felt to be important by stakeholders were related to clinical condition and explicating thinking processes,[5, 6] so we created a domain for content that highlighted these areas and met the ACGME competency of medical knowledge. In accordance with standards for evaluation of learners, we incorporated a domain for judgment to identify where trainees were in the RIME spectrum of reporter, interpreter, master, and educator.
Next, we added a section for professionalism in accordance with the ACGME core competencies of professionalism and patient care.[34] To avoid the disinclination of peers to label each other unprofessional, we labeled the professionalism domain as patient‐focused on the tool.
Finally, we included a domain for setting because of an extensive literature demonstrating increased handoff failures in noisy or interruptive settings.[35, 36, 37] We then revised the tool slightly based on our experiences among nurses and students.[13, 38] The final tool included the 6 domains described above and an assessment of overall competency. Each domain was scored on a 9‐point scale and included descriptive anchors at high and low ends of performance. We further divided the scale into 3 main sections: unsatisfactory (score 13), satisfactory (46), and superior (79). We designed 2 tools, 1 to assess the person providing the handoff and 1 to assess the handoff recipient, each with its own descriptive anchors. The recipient tool did not include a content domain (see Supporting Information, Appendix 1, in the online version of this article).
Setting and Subjects
We tested the tool in 2 different urban academic medical centers: the University of Chicago Medicine (UCM) and Yale‐New Haven Hospital (Yale). At UCM, we tested the tool among hospitalists, nurse practitioners, and physician assistants during the Monday and Tuesday morning and Friday evening sign‐out sessions. At Yale, we tested the tool among housestaff during the evening sign‐out session from the primary team to the on‐call covering team.
The UCM is a 550‐bed urban academic medical center in which the nonteaching hospitalist service cares for patients with liver disease, or end‐stage renal or lung disease awaiting transplant, and a small fraction of general medicine and oncology patients when the housestaff service exceeds its cap. No formal training on sign‐out is provided to attending or midlevel providers. The nonteaching hospitalist service operates as a separate service from the housestaff service and consists of 38 hospitalist clinicians (hospitalist attendings, nurse practitioners, and physicians assistants). There are 2 handoffs each day. In the morning the departing night hospitalist hands off to the incoming daytime hospitalist or midlevel provider. These handoffs occur at 7:30 am in a dedicated room. In the evening the daytime hospitalist or midlevel provider hands off to an incoming night hospitalist. This handoff occurs at 5:30 pm or 7:30 pm in a dedicated location. The written sign‐out is maintained on a Microsoft Word (Microsoft Corp., Redmond, WA) document on a password‐protected server and updated daily.
Yale is a 946‐bed urban academic medical center with a large internal medicine training program. Formal sign‐out education that covers the main domains of the tool is provided to new interns during the first 3 months of the year,[19] and a templated electronic medical record‐based electronic written handoff report is produced by the housestaff for all patients.[22] Approximately half of inpatient medicine patients are cared for by housestaff teams, which are entirely separate from the hospitalist service. Housestaff sign‐out occurs between 4 pm and 7 pm every night. At a minimum, the departing intern signs out to the incoming intern; this handoff is typically supervised by at least 1 second‐ or third‐year resident. All patients are signed out verbally; in addition, the written handoff report is provided to the incoming team. Most handoffs occur in a quiet charting room.
Data Collection
Data collection at UCM occurred between March and December 2010 on 3 days of each week: Mondays, Tuesdays, and Fridays. On Mondays and Tuesdays the morning handoffs were observed; on Fridays the evening handoffs were observed. Data collection at Yale occurred between March and May 2011. Only evening handoffs from the primary team to the overnight coverage were observed. At both sites, participants provided verbal informed consent prior to data collection. At the time of an eligible sign‐out session, a research assistant (D.R. at Yale, P.S. at UCM) provided the evaluation tools to all members of the incoming and outgoing teams, and observed the sign‐out session himself. Each person providing a handoff was asked to evaluate the recipient of the handoff; each person receiving a handoff was asked to evaluate the provider of the handoff. In addition, the trained third‐party observer (D.R., P.S.) evaluated both the provider and recipient of the handoff. The external evaluators were trained in principles of effective communication and the use of the tool, with specific review of anchors at each end of each domain. One evaluator had a DO degree and was completing an MPH degree. The second evaluator was an experienced clinical research assistant whose training consisted of supervised observation of 10 handoffs by a physician investigator. At Yale, if a resident was present, she or he was also asked to evaluate both the provider and recipient of the handoff. Consequently, every sign‐out session included at least 2 evaluations of each participant, 1 by a peer evaluator and 1 by a consistent external evaluator who did not know the patients. At Yale, many sign‐outs also included a third evaluation by a resident supervisor.
The study was approved by the institutional review boards at both UCM and Yale.
Statistical Analysis
We obtained mean, median, and interquartile range of scores for each subdomain of the tool as well as the overall assessment of handoff quality. We assessed convergent construct validity by assessing performance of the tool in different contexts. To do so, we determined whether scores differed by type of participant (provider or recipient), by site, by training level of evaluatee, or by type of evaluator (external, resident supervisor, or peer) by using Wilcoxon rank sum tests and Kruskal‐Wallis tests. For the assessment of differences in ratings by training level, we used evaluations of sign‐out providers only, because the 2 sites differed in scores for recipients. We also assessed construct validity by using Spearman rank correlation coefficients to describe the internal consistency of the tool in terms of the correlation between domains of the tool, and we conducted an exploratory factor analysis to gain insight into whether the subdomains of the tool were measuring the same construct. In conducting this analysis, we restricted the dataset to evaluations of sign‐out providers only, and used a principal components estimation method, a promax rotation, and squared multiple correlation communality priors. Finally, we conducted some preliminary studies of reliability by testing whether different types of evaluators provided similar assessments. We calculated a weighted kappa using Fleiss‐Cohen weights for external versus peer scores and again for supervising resident versus peer scores (Yale only). We were not able to assess test‐retest reliability by nature of the sign‐out process. Statistical significance was defined by a P value 0.05, and analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).
RESULTS
A total of 149 handoff sessions were observed: 89 at UCM and 60 at Yale. Each site conducted a similar total number of evaluations: 336 at UCM, 337 at Yale. These sessions involved 97 unique individuals, 34 at UCM and 63 at Yale. Overall scores were high at both sites, but a wide range of scores was applied (Table 1).
| Domain | Provider, N=343 | Recipient, N=330 | P Value | ||||
|---|---|---|---|---|---|---|---|
| Median (IQR) | Mean (SD) | Range | Median (IQR) | Mean (SD) | Range | ||
| |||||||
| Setting | 7 (69) | 7.0 (1.7) | 29 | 7 (69) | 7.3 (1.6) | 29 | 0.05 |
| Organization | 7 (68) | 7.2 (1.5) | 29 | 8 (69) | 7.4 (1.4) | 29 | 0.07 |
| Communication | 7 (69) | 7.2 (1.6) | 19 | 8 (79) | 7.4 (1.5) | 29 | 0.22 |
| Content | 7 (68) | 7.0 (1.6) | 29 | ||||
| Judgment | 8 (68) | 7.3 (1.4) | 39 | 8 (79) | 7.5 (1.4) | 39 | 0.06 |
| Professionalism | 8 (79) | 7.4 (1.5) | 29 | 8 (79) | 7.6 (1.4) | 39 | 0.23 |
| Overall | 7 (68) | 7.1 (1.5) | 29 | 7 (68) | 7.4 (1.4) | 29 | 0.02 |
Handoff Providers
A total of 343 evaluations of handoff providers were completed regarding 67 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism (median: 8; interquartile range [IQR]: 79). The lowest rated domain was content (median: 7; IQR: 68) (Table 1).
Handoff Recipients
A total of 330 evaluations of handoff recipients were completed regarding 58 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism, with a median of 8 (IQR: 79). The lowest rated domain was setting, with a median score of 7 (IQR: 6‐9) (Table 1).
Validity Testing
Comparing provider scores to recipient scores, recipients received significantly higher scores for overall assessment (Table 1). Scores at UCM and Yale were similar in all domains for providers but were slightly lower at UCM in several domains for recipients (see Supporting Information, Appendix 2, in the online version of this article). Scores did not differ significantly by training level (Table 2). Third‐party external evaluators consistently gave lower marks for the same handoff than peer evaluators did (Table 3).
| Domain | Median (Range) | P Value | |||
|---|---|---|---|---|---|
| NP/PA, N=33 | Subintern or Intern, N=170 | Resident, N=44 | Hospitalist, N=95 | ||
| |||||
| Setting | 7 (29) | 7 (39) | 7 (49) | 7 (29) | 0.89 |
| Organization | 8 (49) | 7 (29) | 7 (49) | 8 (39) | 0.11 |
| Communication | 8 (49) | 7 (29) | 7 (49) | 8 (19) | 0.72 |
| Content | 7 (39) | 7 (29) | 7 (49) | 7 (29) | 0.92 |
| Judgment | 8 (59) | 7 (39) | 8 (49) | 8 (49) | 0.09 |
| Professionalism | 8 (49) | 7 (29) | 8 (39) | 8 (49) | 0.82 |
| Overall | 7 (39) | 7 (29) | 8 (49) | 7 (29) | 0.28 |
| Provider, Median (Range) | Recipient, Median (Range) | |||||||
|---|---|---|---|---|---|---|---|---|
| Domain | Peer, N=152 | Resident, Supervisor, N=43 | External, N=147 | P Value | Peer, N=145 | Resident Supervisor, N=43 | External, N=142 | P Value |
| ||||||||
| Setting | 8 (39) | 7 (39) | 7 (29) | 0.02 | 8 (29) | 7 (39) | 7 (29) | <0.001 |
| Organization | 8 (39) | 8 (39) | 7 (29) | 0.18 | 8 (39) | 8 (69) | 7 (29) | <0.001 |
| Communication | 8 (39) | 8 (39) | 7 (19) | <0.001 | 8 (39) | 8 (49) | 7 (29) | <0.001 |
| Content | 8 (39) | 8 (29) | 7 (29) | <0.001 | N/A | N/A | N/A | N/A |
| Judgment | 8 (49) | 8 (39) | 7 (39) | <0.001 | 8 (39) | 8 (49) | 7 (39) | <0.001 |
| Professionalism | 8 (39) | 8 (59) | 7 (29) | 0.02 | 8 (39) | 8 (69) | 7 (39) | <0.001 |
| Overall | 8 (39) | 8 (39) | 7 (29) | 0.001 | 8 (29) | 8 (49) | 7 (29) | <0.001 |
Spearman rank correlation coefficients among the CEX subdomains for provider scores ranged from 0.71 to 0.86, except for setting (Table 4). Setting was less well correlated with the other subdomains, with correlation coefficients ranging from 0.39 to 0.41. Correlations between individual domains and the overall rating ranged from 0.80 to 0.86, except setting, which had a correlation of 0.55. Every correlation was significant at P<0.001. Correlation coefficients for recipient scores were very similar to those for provider scores (see Supporting Information, Appendix 3, in the online version of this article).
| Spearman Correlation Coefficients | ||||||
|---|---|---|---|---|---|---|
| Setting | Organization | Communication | Content | Judgment | Professionalism | |
| ||||||
| Setting | 1.000 | 0.40 | 0.40 | 0.39 | 0.39 | 0.41 |
| Organization | 0.40 | 1.00 | 0.80 | 0.71 | 0.77 | 0.73 |
| Communication | 0.40 | 0.80 | 1.00 | 0.79 | 0.82 | 0.77 |
| Content | 0.39 | 0.71 | 0.79 | 1.00 | 0.80 | 0.74 |
| Judgment | 0.39 | 0.77 | 0.82 | 0.80 | 1.00 | 0.78 |
| Professionalism | 0.41 | 0.73 | 0.77 | 0.74 | 0.78 | 1.00 |
| Overall | 0.55 | 0.80 | 0.84 | 0.83 | 0.86 | 0.82 |
We analyzed 343 provider evaluations in the factor analysis; there were 6 missing values. The scree plot of eigenvalues did not support more than 1 factor; however, the rotated factor pattern for standardized regression coefficients for the first factor and the final communality estimates showed the setting component yielding smaller values than did other scale components (see Supporting Information, Appendix 4, in the online version of this article).
Reliability Testing
Weighted kappa scores for provider evaluations ranged from 0.28 (95% confidence interval [CI]: 0.01, 0.56) for setting to 0.59 (95% CI: 0.38, 0.80) for organization, and were generally higher for resident versus peer comparisons than for external versus peer comparisons. Weighted kappa scores for recipient evaluation were slightly lower for external versus peer evaluations, but agreement was no better than chance for resident versus peer evaluations (Table 5).
| Domain | Provider | Recipient | ||
|---|---|---|---|---|
| External vs Peer, N=144 (95% CI) | Resident vs Peer, N=42 (95% CI) | External vs Peer, N=134 (95% CI) | Resident vs Peer, N=43 (95% CI) | |
| ||||
| Setting | 0.39 (0.24, 0.54) | 0.28 (0.01, 0.56) | 0.34 (0.20, 0.48) | 0.48 (0.27, 0.69) |
| Organization | 0.43 (0.29, 0.58) | 0.59 (0.39, 0.80) | 0.39 (0.22, 0.55) | 0.03 (0.23, 0.29) |
| Communication | 0.34 (0.19, 0.49) | 0.52 (0.37, 0.68) | 0.36 (0.22, 0.51) | 0.02 (0.18, 0.23) |
| Content | 0.38 (0.25, 0.51) | 0.53 (0.27, 0.80) | N/A (N/A) | N/A (N/A) |
| Judgment | 0.36 (0.22, 0.49) | 0.54 (0.25, 0.83) | 0.28 (0.15, 0.42) | 0.12 (0.34, 0.09) |
| Professionalism | 0.47 (0.32, 0.63) | 0.47 (0.23, 0.72) | 0.35 (0.18, 0.51) | 0.01 (0.29, 0.26) |
| Overall | 0.50 (0.36, 0.64) | 0.45 (0.24, 0.67) | 0.31 (0.16, 0.48) | 0.07 (0.20, 0.34) |
DISCUSSION
In this study we found that an evaluation tool for direct observation of housestaff and hospitalists generated a range of scores and was well validated in the sense of performing similarly across 2 different institutions and among both trainees and attendings, while having high internal consistency. However, external evaluators gave consistently lower marks than peer evaluators at both sites, resulting in low reliability when comparing these 2 groups of raters.
It has traditionally been difficult to conduct direct evaluations of handoffs, because they may occur at haphazard times, in variable locations, and without very much advance notice. For this reason, several attempts have been made to incorporate peers in evaluations of handoff practices.[5, 39, 40] Using peers to conduct evaluations also has the advantage that peers are more likely to be familiar with the patients being handed off and might recognize handoff flaws that external evaluators would miss. Nonetheless, peer evaluations have some important liabilities. Peers may be unwilling or unable to provide honest critiques of their colleagues given that they must work closely together for years. Trainee peers may also lack sufficient clinical expertise or experience to accurately assess competence. In our study, we found that peers gave consistently higher marks to their colleagues than did external evaluators, suggesting they may have found it difficult to criticize their colleagues. We conclude that peer evaluation alone is likely an insufficient means of evaluating handoff quality.
Supervising residents gave very similar marks as intern peers, suggesting that they also are unwilling to criticize, are insufficiently experienced to evaluate, or alternatively, that the peer evaluations were reasonable. We suspect the latter is unlikely given that external evaluator scores were consistently lower than peers. One would expect the external evaluators to be biased toward higher scores given that they are not familiar with the patients and are not able to comment on inaccuracies or omissions in the sign‐out.
The tool appeared to perform less well in most cases for recipients than for providers, with a narrower range of scores and low‐weighted kappa scores. Although recipients play a key role in ensuring a high‐quality sign‐out by paying close attention, ensuring it is a bidirectional conversation, asking appropriate questions, and reading back key information, it may be that evaluators were unable to place these activities within the same domains that were used for the provider evaluation. An altogether different recipient evaluation approach may be necessary.[41]
In general, scores were clustered at the top of the score range, as is typical for evaluations. One strategy to spread out scores further would be to refine the tool by adding anchors for satisfactory performance not just the extremes. A second approach might be to reduce the grading scale to only 3 points (unsatisfactory, satisfactory, superior) to force more scores to the middle. However, this approach might limit the discrimination ability of the tool.
We have previously studied the use of this tool among nurses. In that study, we also found consistently higher scores by peers than by external evaluators. We did, however, find a positive effect of experience, in which more experienced nurses received higher scores on average. We did not observe a similar training effect in this study. There are several possible explanations for the lack of a training effect. It is possible that the types of handoffs assessed played a role. At UCM, some assessed handoffs were night staff to day staff, which might be lower quality than day staff to night staff handoffs, whereas at Yale, all handoffs were day to night teams. Thus, average scores at UCM (primarily hospitalists) might have been lowered by the type of handoff provided. Given that hospitalist evaluations were conducted exclusively at UCM and housestaff evaluations exclusively at Yale, lack of difference between hospitalists and housestaff may also have been related to differences in evaluation practice or handoff practice at the 2 sites, not necessarily related to training level. Third, in our experience, attending physicians provide briefer less‐comprehensive sign‐outs than trainees, particularly when communicating with equally experienced attendings; these sign‐outs may appropriately be scored lower on the tool. Fourth, the great majority of the hospitalists at UCM were within 5 years of residency and therefore not very much more experienced than the trainees. Finally, it is possible that skills do not improve over time given widespread lack of observation and feedback during training years for this important skill.
The high internal consistency of most of the subdomains and the loading of all subdomains except setting onto 1 factor are evidence of convergent construct validity, but also suggest that evaluators have difficulty distinguishing among components of sign‐out quality. Internal consistency may also reflect a halo effect, in which scores on different domains are all influenced by a common overall judgment.[42] We are currently testing a shorter version of the tool including domains only for content, professionalism, and setting in addition to overall score. The fact that setting did not correlate as well with the other domains suggests that sign‐out practitioners may not have or exercise control over their surroundings. Consequently, it may ultimately be reasonable to drop this domain from the tool, or alternatively, to refocus on the need to ensure a quiet setting during sign‐out skills training.
There are several limitations to this study. External evaluations were conducted by personnel who were not familiar with the patients, and they may therefore have overestimated the quality of sign‐out. Studying different types of physicians at different sites might have limited our ability to identify differences by training level. As is commonly seen in evaluation studies, scores were skewed to the high end, although we did observe some use of the full range of the tool. Finally, we were limited in our ability to test inter‐rater reliability because of the multiple sources of variability in the data (numerous different raters, with different backgrounds at different settings, rating different individuals).
In summary, we developed a handoff evaluation tool that was easily completed by housestaff and attendings without training, that performed similarly in a variety of different settings at 2 institutions, and that can in principle be used either for peer evaluations or for external evaluations, although peer evaluations may be positively biased. Further work will be done to refine and simplify the tool.
ACKNOWLEDGMENTS
Disclosures: Development and evaluation of the sign‐out CEX was supported by a grant from the Agency for Healthcare Research and Quality (1R03HS018278‐01). Dr. Arora is supported by a National Institute on Aging (K23 AG033763). Dr. Horwitz is supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. Dr. Horwitz is also a Pepper Scholar with support from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality, the National Institute on Aging, the National Institutes of Health, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as a poster presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 9, 2012. Dr. Rand is now with the Department of Medicine, University of Vermont College of Medicine, Burlington, Vermont. Mr. Staisiunas is now with the Law School, Marquette University, Milwaukee, Wisconsin. The authors declare they have no conflicts of interest.
Appendix
A
PROVIDER HAND‐OFF CEX TOOL
RECIPIENT HAND‐OFF CEX TOOL
Appendix
B
Handoff CEX scores by site of evaluation
| Domain | Provider | Recipient | ||||
| Median (Range) | P‐value | Median (Range) | P‐value | |||
| UC | Yale | UC | Yale | |||
| N=172 | N=170 | N=163 | N=167 | |||
| Setting | 7 (29) | 7 (39) | 0.32 | 7 (29) | 7 (39) | 0.36 |
| Organization | 8 (29) | 7 (39) | 0.30 | 7 (29) | 8 (59) | 0.001 |
| Communication | 7 (19) | 7 (39) | 0.67 | 7 (29) | 8 (49) | 0.03 |
| Content | 7 (29) | 7 (29) | N/A | N/A | N/A | |
| Judgment | 8 (39) | 7 (39) | 0.60 | 7 (39) | 8 (49) | 0.001 |
| Professionalism | 8 (29) | 8 (39) | 0.67 | 8 (39) | 8 (49) | 0.35 |
| Overall | 7 (29) | 7 (39) | 0.41 | 7 (29) | 8 (49) | 0.005 |
Appendix
C
Spearman correlation, recipients (N=330)
| SpearmanCorrelationCoefficients | |||||
| Setting | Organization | Communication | Judgment | Professionalism | |
| Setting | 1.0 | 0.46 | 0.48 | 0.47 | 0.40 |
| Organization | 0.46 | 1.00 | 0.78 | 0.75 | 0.75 |
| Communication | 0.48 | 0.78 | 1.00 | 0.85 | 0.77 |
| Judgment | 0.47 | 0.75 | 0.85 | 1.00 | 0.74 |
| Professionalism | 0.40 | 0.75 | 0.77 | 0.74 | 1.00 |
| Overall | 0.60 | 0.77 | 0.84 | 0.82 | 0.77 |
All p values <0.0001
Appendix
D
Factor analysis results for provider evaluations
| Rotated Factor Pattern (Standardized Regression Coefficients) N=336 | ||
| Factor1 | Factor2 | |
| Organization | 0.64 | 0.27 |
| Communication | 0.79 | 0.16 |
| Content | 0.82 | 0.06 |
| Judgment | 0.86 | 0.06 |
| Professionalism | 0.66 | 0.23 |
| Setting | 0.18 | 0.29 |
- , , , . Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):1173–1177.
- Accreditation Council for Graduate Medical Education. Common program requirements. 2011; http://www.acgme‐2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed August 23, 2011.
- , , , , . Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866–872.
- , , . Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186–194.
- , , , , . Communication failures in patient sign‐out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401–407.
- , , , , . Consequences of inadequate sign‐out for patient care. Arch Intern Med. 2008;168(16):1755–1760.
- , , , . Adequacy of information transferred at resident sign‐out (in‐hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):6–10.
- , , , , . What are covering doctors told about their patients? Analysis of sign‐out among internal medicine house staff. Qual Saf Health Care. 2009;18(4):248–255.
- , . Using direct observation, formal evaluation, and an interactive curriculum to improve the sign‐out practices of internal medicine interns. Acad Med. 2010;85(7):1182–1188.
- , , , . Doctors' handovers in hospitals: a literature review. Qual Saf Health Care. 2011;20(2):128–133.
- , , , et al. Resident sign‐out and patient hand‐offs: opportunities for improvement. Teach Learn Med. 2011;23(2):105–111.
- , , , et al. Use of an appreciative inquiry approach to improve resident sign‐out in an era of multiple shift changes. J Gen Intern Med. 2012;27(3):287–291.
- , , , , , . Validation of a handoff assessment tool: the Handoff CEX [published online ahead of print June 7, 2012]. J Clin Nurs. doi: 10.1111/j.1365–2702.2012.04131.x.
- , , , . The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123(10):795–799.
- , , , . Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):27–33.
- , , , . Assessing the reliability and validity of the mini‐clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900–904.
- , , , , . Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826–830.
- , , , , , . Dropping the baton: a qualitative analysis of failures during the transition from emergency department to inpatient care. Ann Emerg Med. 2009;53(6):701–710.e4.
- , , . Development and implementation of an oral sign‐out skills curriculum. J Gen Intern Med. 2007;22(10):1470–1474.
- , , , . Mixed methods evaluation of oral sign‐out practices. J Gen Intern Med. 2007;22(S1):S114.
- , , , et al. Evaluation of an asynchronous physician voicemail sign‐out for emergency department admissions. Ann Emerg Med. 2009;54(3):368–378.
- , , , et al. An institution‐wide handoff task force to standardise and improve physician handoffs. BMJ Qual Saf. 2012;21(10):863–871.
- , . A model for building a standardized hand‐off protocol. Jt Comm J Qual Patient Saf. 2006;32(11):646–655.
- , , , , . Medication discrepancies in resident sign‐outs and their potential to harm. J Gen Intern Med. 2007;22(12):1751–1755.
- , , , . A theoretical framework and competency‐based approach to improving handoffs. Qual Saf Health Care. 2008;17(1):11–14.
- , , , , , . Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med. 2009;4(7):433–440.
- , , , , . Interns overestimate the effectiveness of their hand‐off communication. Pediatrics. 2010;125(3):491–496.
- , . Improving clinical handovers: creating local solutions for a global problem. Qual Saf Health Care. 2009;18(4):244–245.
- , , , , . Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign‐out. J Hosp Med. 2006;1(4):257–266.
- , , . Standardized sign‐out reduces intern perception of medical errors on the general internal medicine ward. Teach Learn Med. 2009;21(2):121–126.
- , , . SBAR: a shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf. 2006;32(3):167–175.
- . Structuring flexibility: the potential good, bad and ugly in standardisation of handovers. Qual Saf Health Care. 2008;17(1):4–5.
- , , , , . Handoff strategies in settings with high consequences for failure: lessons for health care operations. Int J Qual Health Care. 2004;16(2):125–132.
- , , , , , . Residents' perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758–763.
- , . Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673–676.
- , , , , . Communication loads on clinical staff in the emergency department. Med J Aust. 2002;176(9):415–418.
- , . A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274–284.
- , , , et al. Hand‐off education and evaluation: piloting the observed simulated hand‐off experience (OSHE). J Gen Intern Med. 2010;25(2):129–134.
- , , , et al. Handoffs causing patient harm: a survey of medical and surgical house staff. Jt Comm J Qual Patient Saf. 2008;34(10):563–570.
- , , . A prospective observational study of physician handoff for intensive‐care‐unit‐to‐ward patient transfers. Am J Med. 2011;124(9):860–867.
- , , , , . Characterizing physician listening behavior during hospitalist handoffs using the HEAR checklist (published online ahead of print December 20, 2012]. BMJ Qual Saf. doi:10.1136/bmjqs‐2012‐001138.
- . A constant error in psychological ratings. J Appl Psychol. 1920;4(1):25.
Transfers among trainee physicians within the hospital typically occur at least twice a day and have been increasing among trainees as work hours have declined.[1] The 2011 Accreditation Council for Graduate Medical Education (ACGME) guidelines,[2] which restrict intern working hours to 16 hours from a previous maximum of 30, have likely increased the frequency of physician trainee handoffs even further. Similarly, transfers among hospitalist attendings occur at least twice a day, given typical shifts of 8 to 12 hours.
Given the frequency of transfers, and the potential for harm generated by failed transitions,[3, 4, 5, 6] the end‐of‐shift written and verbal handoffs have assumed increasingly greater importance in hospital care among both trainees and hospitalist attendings.
The ACGME now requires that programs assess the competency of trainees in handoff communication.[2] Yet, there are few tools for assessing the quality of sign‐out communication. Those that exist primarily focus on the written sign‐out, and are rarely validated.[7, 8, 9, 10, 11, 12] Furthermore, it is uncertain whether such assessments must be done by supervisors or whether peers can participate in the evaluation. In this prospective multi‐institutional study we assess the performance characteristics of a verbal sign‐out evaluation tool for internal medicine housestaff and hospitalist attendings, and examine whether it can be used by peers as well as by external evaluators. This tool has previously been found to effectively discriminate between experienced and inexperienced nurses conducting nursing handoffs.[13]
METHODS
Tool Design and Measures
The Handoff CEX (clinical evaluation exercise) is a structured assessment based on the format of the mini‐CEX, an instrument used to assess the quality of history and physical examination by trainees for which validation studies have previously been conducted.[14, 15, 16, 17] We developed the tool based on themes we identified from our own expertise,[1, 5, 6, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29] the ACGME core competencies for trainees,[2] and the literature to maximize content validity. First, standardization has numerous demonstrable benefits for safety in general and handoffs in particular.[30, 31, 32] Consequently we created a domain for organization in which standardization was a characteristic of high performance.
Second, there is evidence that people engaged in conversation routinely overestimate peer comprehension,[27] and that explicit strategies to combat this overestimation, such as confirming understanding, explicitly assigning tasks rather than using open‐ended language, and using concrete language, are effective.[33] Accordingly we created a domain for communication skills, which is also an ACGME competency.
Third, although there were no formal guidelines for sign‐out content when we developed this tool, our own research had demonstrated that the content elements most often missing and felt to be important by stakeholders were related to clinical condition and explicating thinking processes,[5, 6] so we created a domain for content that highlighted these areas and met the ACGME competency of medical knowledge. In accordance with standards for evaluation of learners, we incorporated a domain for judgment to identify where trainees were in the RIME spectrum of reporter, interpreter, master, and educator.
Next, we added a section for professionalism in accordance with the ACGME core competencies of professionalism and patient care.[34] To avoid the disinclination of peers to label each other unprofessional, we labeled the professionalism domain as patient‐focused on the tool.
Finally, we included a domain for setting because of an extensive literature demonstrating increased handoff failures in noisy or interruptive settings.[35, 36, 37] We then revised the tool slightly based on our experiences among nurses and students.[13, 38] The final tool included the 6 domains described above and an assessment of overall competency. Each domain was scored on a 9‐point scale and included descriptive anchors at high and low ends of performance. We further divided the scale into 3 main sections: unsatisfactory (score 13), satisfactory (46), and superior (79). We designed 2 tools, 1 to assess the person providing the handoff and 1 to assess the handoff recipient, each with its own descriptive anchors. The recipient tool did not include a content domain (see Supporting Information, Appendix 1, in the online version of this article).
Setting and Subjects
We tested the tool in 2 different urban academic medical centers: the University of Chicago Medicine (UCM) and Yale‐New Haven Hospital (Yale). At UCM, we tested the tool among hospitalists, nurse practitioners, and physician assistants during the Monday and Tuesday morning and Friday evening sign‐out sessions. At Yale, we tested the tool among housestaff during the evening sign‐out session from the primary team to the on‐call covering team.
The UCM is a 550‐bed urban academic medical center in which the nonteaching hospitalist service cares for patients with liver disease, or end‐stage renal or lung disease awaiting transplant, and a small fraction of general medicine and oncology patients when the housestaff service exceeds its cap. No formal training on sign‐out is provided to attending or midlevel providers. The nonteaching hospitalist service operates as a separate service from the housestaff service and consists of 38 hospitalist clinicians (hospitalist attendings, nurse practitioners, and physicians assistants). There are 2 handoffs each day. In the morning the departing night hospitalist hands off to the incoming daytime hospitalist or midlevel provider. These handoffs occur at 7:30 am in a dedicated room. In the evening the daytime hospitalist or midlevel provider hands off to an incoming night hospitalist. This handoff occurs at 5:30 pm or 7:30 pm in a dedicated location. The written sign‐out is maintained on a Microsoft Word (Microsoft Corp., Redmond, WA) document on a password‐protected server and updated daily.
Yale is a 946‐bed urban academic medical center with a large internal medicine training program. Formal sign‐out education that covers the main domains of the tool is provided to new interns during the first 3 months of the year,[19] and a templated electronic medical record‐based electronic written handoff report is produced by the housestaff for all patients.[22] Approximately half of inpatient medicine patients are cared for by housestaff teams, which are entirely separate from the hospitalist service. Housestaff sign‐out occurs between 4 pm and 7 pm every night. At a minimum, the departing intern signs out to the incoming intern; this handoff is typically supervised by at least 1 second‐ or third‐year resident. All patients are signed out verbally; in addition, the written handoff report is provided to the incoming team. Most handoffs occur in a quiet charting room.
Data Collection
Data collection at UCM occurred between March and December 2010 on 3 days of each week: Mondays, Tuesdays, and Fridays. On Mondays and Tuesdays the morning handoffs were observed; on Fridays the evening handoffs were observed. Data collection at Yale occurred between March and May 2011. Only evening handoffs from the primary team to the overnight coverage were observed. At both sites, participants provided verbal informed consent prior to data collection. At the time of an eligible sign‐out session, a research assistant (D.R. at Yale, P.S. at UCM) provided the evaluation tools to all members of the incoming and outgoing teams, and observed the sign‐out session himself. Each person providing a handoff was asked to evaluate the recipient of the handoff; each person receiving a handoff was asked to evaluate the provider of the handoff. In addition, the trained third‐party observer (D.R., P.S.) evaluated both the provider and recipient of the handoff. The external evaluators were trained in principles of effective communication and the use of the tool, with specific review of anchors at each end of each domain. One evaluator had a DO degree and was completing an MPH degree. The second evaluator was an experienced clinical research assistant whose training consisted of supervised observation of 10 handoffs by a physician investigator. At Yale, if a resident was present, she or he was also asked to evaluate both the provider and recipient of the handoff. Consequently, every sign‐out session included at least 2 evaluations of each participant, 1 by a peer evaluator and 1 by a consistent external evaluator who did not know the patients. At Yale, many sign‐outs also included a third evaluation by a resident supervisor.
The study was approved by the institutional review boards at both UCM and Yale.
Statistical Analysis
We obtained mean, median, and interquartile range of scores for each subdomain of the tool as well as the overall assessment of handoff quality. We assessed convergent construct validity by assessing performance of the tool in different contexts. To do so, we determined whether scores differed by type of participant (provider or recipient), by site, by training level of evaluatee, or by type of evaluator (external, resident supervisor, or peer) by using Wilcoxon rank sum tests and Kruskal‐Wallis tests. For the assessment of differences in ratings by training level, we used evaluations of sign‐out providers only, because the 2 sites differed in scores for recipients. We also assessed construct validity by using Spearman rank correlation coefficients to describe the internal consistency of the tool in terms of the correlation between domains of the tool, and we conducted an exploratory factor analysis to gain insight into whether the subdomains of the tool were measuring the same construct. In conducting this analysis, we restricted the dataset to evaluations of sign‐out providers only, and used a principal components estimation method, a promax rotation, and squared multiple correlation communality priors. Finally, we conducted some preliminary studies of reliability by testing whether different types of evaluators provided similar assessments. We calculated a weighted kappa using Fleiss‐Cohen weights for external versus peer scores and again for supervising resident versus peer scores (Yale only). We were not able to assess test‐retest reliability by nature of the sign‐out process. Statistical significance was defined by a P value 0.05, and analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).
RESULTS
A total of 149 handoff sessions were observed: 89 at UCM and 60 at Yale. Each site conducted a similar total number of evaluations: 336 at UCM, 337 at Yale. These sessions involved 97 unique individuals, 34 at UCM and 63 at Yale. Overall scores were high at both sites, but a wide range of scores was applied (Table 1).
| Domain | Provider, N=343 | Recipient, N=330 | P Value | ||||
|---|---|---|---|---|---|---|---|
| Median (IQR) | Mean (SD) | Range | Median (IQR) | Mean (SD) | Range | ||
| |||||||
| Setting | 7 (69) | 7.0 (1.7) | 29 | 7 (69) | 7.3 (1.6) | 29 | 0.05 |
| Organization | 7 (68) | 7.2 (1.5) | 29 | 8 (69) | 7.4 (1.4) | 29 | 0.07 |
| Communication | 7 (69) | 7.2 (1.6) | 19 | 8 (79) | 7.4 (1.5) | 29 | 0.22 |
| Content | 7 (68) | 7.0 (1.6) | 29 | ||||
| Judgment | 8 (68) | 7.3 (1.4) | 39 | 8 (79) | 7.5 (1.4) | 39 | 0.06 |
| Professionalism | 8 (79) | 7.4 (1.5) | 29 | 8 (79) | 7.6 (1.4) | 39 | 0.23 |
| Overall | 7 (68) | 7.1 (1.5) | 29 | 7 (68) | 7.4 (1.4) | 29 | 0.02 |
Handoff Providers
A total of 343 evaluations of handoff providers were completed regarding 67 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism (median: 8; interquartile range [IQR]: 79). The lowest rated domain was content (median: 7; IQR: 68) (Table 1).
Handoff Recipients
A total of 330 evaluations of handoff recipients were completed regarding 58 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism, with a median of 8 (IQR: 79). The lowest rated domain was setting, with a median score of 7 (IQR: 6‐9) (Table 1).
Validity Testing
Comparing provider scores to recipient scores, recipients received significantly higher scores for overall assessment (Table 1). Scores at UCM and Yale were similar in all domains for providers but were slightly lower at UCM in several domains for recipients (see Supporting Information, Appendix 2, in the online version of this article). Scores did not differ significantly by training level (Table 2). Third‐party external evaluators consistently gave lower marks for the same handoff than peer evaluators did (Table 3).
| Domain | Median (Range) | P Value | |||
|---|---|---|---|---|---|
| NP/PA, N=33 | Subintern or Intern, N=170 | Resident, N=44 | Hospitalist, N=95 | ||
| |||||
| Setting | 7 (29) | 7 (39) | 7 (49) | 7 (29) | 0.89 |
| Organization | 8 (49) | 7 (29) | 7 (49) | 8 (39) | 0.11 |
| Communication | 8 (49) | 7 (29) | 7 (49) | 8 (19) | 0.72 |
| Content | 7 (39) | 7 (29) | 7 (49) | 7 (29) | 0.92 |
| Judgment | 8 (59) | 7 (39) | 8 (49) | 8 (49) | 0.09 |
| Professionalism | 8 (49) | 7 (29) | 8 (39) | 8 (49) | 0.82 |
| Overall | 7 (39) | 7 (29) | 8 (49) | 7 (29) | 0.28 |
| Provider, Median (Range) | Recipient, Median (Range) | |||||||
|---|---|---|---|---|---|---|---|---|
| Domain | Peer, N=152 | Resident, Supervisor, N=43 | External, N=147 | P Value | Peer, N=145 | Resident Supervisor, N=43 | External, N=142 | P Value |
| ||||||||
| Setting | 8 (39) | 7 (39) | 7 (29) | 0.02 | 8 (29) | 7 (39) | 7 (29) | <0.001 |
| Organization | 8 (39) | 8 (39) | 7 (29) | 0.18 | 8 (39) | 8 (69) | 7 (29) | <0.001 |
| Communication | 8 (39) | 8 (39) | 7 (19) | <0.001 | 8 (39) | 8 (49) | 7 (29) | <0.001 |
| Content | 8 (39) | 8 (29) | 7 (29) | <0.001 | N/A | N/A | N/A | N/A |
| Judgment | 8 (49) | 8 (39) | 7 (39) | <0.001 | 8 (39) | 8 (49) | 7 (39) | <0.001 |
| Professionalism | 8 (39) | 8 (59) | 7 (29) | 0.02 | 8 (39) | 8 (69) | 7 (39) | <0.001 |
| Overall | 8 (39) | 8 (39) | 7 (29) | 0.001 | 8 (29) | 8 (49) | 7 (29) | <0.001 |
Spearman rank correlation coefficients among the CEX subdomains for provider scores ranged from 0.71 to 0.86, except for setting (Table 4). Setting was less well correlated with the other subdomains, with correlation coefficients ranging from 0.39 to 0.41. Correlations between individual domains and the overall rating ranged from 0.80 to 0.86, except setting, which had a correlation of 0.55. Every correlation was significant at P<0.001. Correlation coefficients for recipient scores were very similar to those for provider scores (see Supporting Information, Appendix 3, in the online version of this article).
| Spearman Correlation Coefficients | ||||||
|---|---|---|---|---|---|---|
| Setting | Organization | Communication | Content | Judgment | Professionalism | |
| ||||||
| Setting | 1.000 | 0.40 | 0.40 | 0.39 | 0.39 | 0.41 |
| Organization | 0.40 | 1.00 | 0.80 | 0.71 | 0.77 | 0.73 |
| Communication | 0.40 | 0.80 | 1.00 | 0.79 | 0.82 | 0.77 |
| Content | 0.39 | 0.71 | 0.79 | 1.00 | 0.80 | 0.74 |
| Judgment | 0.39 | 0.77 | 0.82 | 0.80 | 1.00 | 0.78 |
| Professionalism | 0.41 | 0.73 | 0.77 | 0.74 | 0.78 | 1.00 |
| Overall | 0.55 | 0.80 | 0.84 | 0.83 | 0.86 | 0.82 |
We analyzed 343 provider evaluations in the factor analysis; there were 6 missing values. The scree plot of eigenvalues did not support more than 1 factor; however, the rotated factor pattern for standardized regression coefficients for the first factor and the final communality estimates showed the setting component yielding smaller values than did other scale components (see Supporting Information, Appendix 4, in the online version of this article).
Reliability Testing
Weighted kappa scores for provider evaluations ranged from 0.28 (95% confidence interval [CI]: 0.01, 0.56) for setting to 0.59 (95% CI: 0.38, 0.80) for organization, and were generally higher for resident versus peer comparisons than for external versus peer comparisons. Weighted kappa scores for recipient evaluation were slightly lower for external versus peer evaluations, but agreement was no better than chance for resident versus peer evaluations (Table 5).
| Domain | Provider | Recipient | ||
|---|---|---|---|---|
| External vs Peer, N=144 (95% CI) | Resident vs Peer, N=42 (95% CI) | External vs Peer, N=134 (95% CI) | Resident vs Peer, N=43 (95% CI) | |
| ||||
| Setting | 0.39 (0.24, 0.54) | 0.28 (0.01, 0.56) | 0.34 (0.20, 0.48) | 0.48 (0.27, 0.69) |
| Organization | 0.43 (0.29, 0.58) | 0.59 (0.39, 0.80) | 0.39 (0.22, 0.55) | 0.03 (0.23, 0.29) |
| Communication | 0.34 (0.19, 0.49) | 0.52 (0.37, 0.68) | 0.36 (0.22, 0.51) | 0.02 (0.18, 0.23) |
| Content | 0.38 (0.25, 0.51) | 0.53 (0.27, 0.80) | N/A (N/A) | N/A (N/A) |
| Judgment | 0.36 (0.22, 0.49) | 0.54 (0.25, 0.83) | 0.28 (0.15, 0.42) | 0.12 (0.34, 0.09) |
| Professionalism | 0.47 (0.32, 0.63) | 0.47 (0.23, 0.72) | 0.35 (0.18, 0.51) | 0.01 (0.29, 0.26) |
| Overall | 0.50 (0.36, 0.64) | 0.45 (0.24, 0.67) | 0.31 (0.16, 0.48) | 0.07 (0.20, 0.34) |
DISCUSSION
In this study we found that an evaluation tool for direct observation of housestaff and hospitalists generated a range of scores and was well validated in the sense of performing similarly across 2 different institutions and among both trainees and attendings, while having high internal consistency. However, external evaluators gave consistently lower marks than peer evaluators at both sites, resulting in low reliability when comparing these 2 groups of raters.
It has traditionally been difficult to conduct direct evaluations of handoffs, because they may occur at haphazard times, in variable locations, and without very much advance notice. For this reason, several attempts have been made to incorporate peers in evaluations of handoff practices.[5, 39, 40] Using peers to conduct evaluations also has the advantage that peers are more likely to be familiar with the patients being handed off and might recognize handoff flaws that external evaluators would miss. Nonetheless, peer evaluations have some important liabilities. Peers may be unwilling or unable to provide honest critiques of their colleagues given that they must work closely together for years. Trainee peers may also lack sufficient clinical expertise or experience to accurately assess competence. In our study, we found that peers gave consistently higher marks to their colleagues than did external evaluators, suggesting they may have found it difficult to criticize their colleagues. We conclude that peer evaluation alone is likely an insufficient means of evaluating handoff quality.
Supervising residents gave very similar marks as intern peers, suggesting that they also are unwilling to criticize, are insufficiently experienced to evaluate, or alternatively, that the peer evaluations were reasonable. We suspect the latter is unlikely given that external evaluator scores were consistently lower than peers. One would expect the external evaluators to be biased toward higher scores given that they are not familiar with the patients and are not able to comment on inaccuracies or omissions in the sign‐out.
The tool appeared to perform less well in most cases for recipients than for providers, with a narrower range of scores and low‐weighted kappa scores. Although recipients play a key role in ensuring a high‐quality sign‐out by paying close attention, ensuring it is a bidirectional conversation, asking appropriate questions, and reading back key information, it may be that evaluators were unable to place these activities within the same domains that were used for the provider evaluation. An altogether different recipient evaluation approach may be necessary.[41]
In general, scores were clustered at the top of the score range, as is typical for evaluations. One strategy to spread out scores further would be to refine the tool by adding anchors for satisfactory performance not just the extremes. A second approach might be to reduce the grading scale to only 3 points (unsatisfactory, satisfactory, superior) to force more scores to the middle. However, this approach might limit the discrimination ability of the tool.
We have previously studied the use of this tool among nurses. In that study, we also found consistently higher scores by peers than by external evaluators. We did, however, find a positive effect of experience, in which more experienced nurses received higher scores on average. We did not observe a similar training effect in this study. There are several possible explanations for the lack of a training effect. It is possible that the types of handoffs assessed played a role. At UCM, some assessed handoffs were night staff to day staff, which might be lower quality than day staff to night staff handoffs, whereas at Yale, all handoffs were day to night teams. Thus, average scores at UCM (primarily hospitalists) might have been lowered by the type of handoff provided. Given that hospitalist evaluations were conducted exclusively at UCM and housestaff evaluations exclusively at Yale, lack of difference between hospitalists and housestaff may also have been related to differences in evaluation practice or handoff practice at the 2 sites, not necessarily related to training level. Third, in our experience, attending physicians provide briefer less‐comprehensive sign‐outs than trainees, particularly when communicating with equally experienced attendings; these sign‐outs may appropriately be scored lower on the tool. Fourth, the great majority of the hospitalists at UCM were within 5 years of residency and therefore not very much more experienced than the trainees. Finally, it is possible that skills do not improve over time given widespread lack of observation and feedback during training years for this important skill.
The high internal consistency of most of the subdomains and the loading of all subdomains except setting onto 1 factor are evidence of convergent construct validity, but also suggest that evaluators have difficulty distinguishing among components of sign‐out quality. Internal consistency may also reflect a halo effect, in which scores on different domains are all influenced by a common overall judgment.[42] We are currently testing a shorter version of the tool including domains only for content, professionalism, and setting in addition to overall score. The fact that setting did not correlate as well with the other domains suggests that sign‐out practitioners may not have or exercise control over their surroundings. Consequently, it may ultimately be reasonable to drop this domain from the tool, or alternatively, to refocus on the need to ensure a quiet setting during sign‐out skills training.
There are several limitations to this study. External evaluations were conducted by personnel who were not familiar with the patients, and they may therefore have overestimated the quality of sign‐out. Studying different types of physicians at different sites might have limited our ability to identify differences by training level. As is commonly seen in evaluation studies, scores were skewed to the high end, although we did observe some use of the full range of the tool. Finally, we were limited in our ability to test inter‐rater reliability because of the multiple sources of variability in the data (numerous different raters, with different backgrounds at different settings, rating different individuals).
In summary, we developed a handoff evaluation tool that was easily completed by housestaff and attendings without training, that performed similarly in a variety of different settings at 2 institutions, and that can in principle be used either for peer evaluations or for external evaluations, although peer evaluations may be positively biased. Further work will be done to refine and simplify the tool.
ACKNOWLEDGMENTS
Disclosures: Development and evaluation of the sign‐out CEX was supported by a grant from the Agency for Healthcare Research and Quality (1R03HS018278‐01). Dr. Arora is supported by a National Institute on Aging (K23 AG033763). Dr. Horwitz is supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. Dr. Horwitz is also a Pepper Scholar with support from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality, the National Institute on Aging, the National Institutes of Health, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as a poster presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 9, 2012. Dr. Rand is now with the Department of Medicine, University of Vermont College of Medicine, Burlington, Vermont. Mr. Staisiunas is now with the Law School, Marquette University, Milwaukee, Wisconsin. The authors declare they have no conflicts of interest.
Appendix
A
PROVIDER HAND‐OFF CEX TOOL
RECIPIENT HAND‐OFF CEX TOOL
Appendix
B
Handoff CEX scores by site of evaluation
| Domain | Provider | Recipient | ||||
| Median (Range) | P‐value | Median (Range) | P‐value | |||
| UC | Yale | UC | Yale | |||
| N=172 | N=170 | N=163 | N=167 | |||
| Setting | 7 (29) | 7 (39) | 0.32 | 7 (29) | 7 (39) | 0.36 |
| Organization | 8 (29) | 7 (39) | 0.30 | 7 (29) | 8 (59) | 0.001 |
| Communication | 7 (19) | 7 (39) | 0.67 | 7 (29) | 8 (49) | 0.03 |
| Content | 7 (29) | 7 (29) | N/A | N/A | N/A | |
| Judgment | 8 (39) | 7 (39) | 0.60 | 7 (39) | 8 (49) | 0.001 |
| Professionalism | 8 (29) | 8 (39) | 0.67 | 8 (39) | 8 (49) | 0.35 |
| Overall | 7 (29) | 7 (39) | 0.41 | 7 (29) | 8 (49) | 0.005 |
Appendix
C
Spearman correlation, recipients (N=330)
| SpearmanCorrelationCoefficients | |||||
| Setting | Organization | Communication | Judgment | Professionalism | |
| Setting | 1.0 | 0.46 | 0.48 | 0.47 | 0.40 |
| Organization | 0.46 | 1.00 | 0.78 | 0.75 | 0.75 |
| Communication | 0.48 | 0.78 | 1.00 | 0.85 | 0.77 |
| Judgment | 0.47 | 0.75 | 0.85 | 1.00 | 0.74 |
| Professionalism | 0.40 | 0.75 | 0.77 | 0.74 | 1.00 |
| Overall | 0.60 | 0.77 | 0.84 | 0.82 | 0.77 |
All p values <0.0001
Appendix
D
Factor analysis results for provider evaluations
| Rotated Factor Pattern (Standardized Regression Coefficients) N=336 | ||
| Factor1 | Factor2 | |
| Organization | 0.64 | 0.27 |
| Communication | 0.79 | 0.16 |
| Content | 0.82 | 0.06 |
| Judgment | 0.86 | 0.06 |
| Professionalism | 0.66 | 0.23 |
| Setting | 0.18 | 0.29 |
Transfers among trainee physicians within the hospital typically occur at least twice a day and have been increasing among trainees as work hours have declined.[1] The 2011 Accreditation Council for Graduate Medical Education (ACGME) guidelines,[2] which restrict intern working hours to 16 hours from a previous maximum of 30, have likely increased the frequency of physician trainee handoffs even further. Similarly, transfers among hospitalist attendings occur at least twice a day, given typical shifts of 8 to 12 hours.
Given the frequency of transfers, and the potential for harm generated by failed transitions,[3, 4, 5, 6] the end‐of‐shift written and verbal handoffs have assumed increasingly greater importance in hospital care among both trainees and hospitalist attendings.
The ACGME now requires that programs assess the competency of trainees in handoff communication.[2] Yet, there are few tools for assessing the quality of sign‐out communication. Those that exist primarily focus on the written sign‐out, and are rarely validated.[7, 8, 9, 10, 11, 12] Furthermore, it is uncertain whether such assessments must be done by supervisors or whether peers can participate in the evaluation. In this prospective multi‐institutional study we assess the performance characteristics of a verbal sign‐out evaluation tool for internal medicine housestaff and hospitalist attendings, and examine whether it can be used by peers as well as by external evaluators. This tool has previously been found to effectively discriminate between experienced and inexperienced nurses conducting nursing handoffs.[13]
METHODS
Tool Design and Measures
The Handoff CEX (clinical evaluation exercise) is a structured assessment based on the format of the mini‐CEX, an instrument used to assess the quality of history and physical examination by trainees for which validation studies have previously been conducted.[14, 15, 16, 17] We developed the tool based on themes we identified from our own expertise,[1, 5, 6, 8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29] the ACGME core competencies for trainees,[2] and the literature to maximize content validity. First, standardization has numerous demonstrable benefits for safety in general and handoffs in particular.[30, 31, 32] Consequently we created a domain for organization in which standardization was a characteristic of high performance.
Second, there is evidence that people engaged in conversation routinely overestimate peer comprehension,[27] and that explicit strategies to combat this overestimation, such as confirming understanding, explicitly assigning tasks rather than using open‐ended language, and using concrete language, are effective.[33] Accordingly we created a domain for communication skills, which is also an ACGME competency.
Third, although there were no formal guidelines for sign‐out content when we developed this tool, our own research had demonstrated that the content elements most often missing and felt to be important by stakeholders were related to clinical condition and explicating thinking processes,[5, 6] so we created a domain for content that highlighted these areas and met the ACGME competency of medical knowledge. In accordance with standards for evaluation of learners, we incorporated a domain for judgment to identify where trainees were in the RIME spectrum of reporter, interpreter, master, and educator.
Next, we added a section for professionalism in accordance with the ACGME core competencies of professionalism and patient care.[34] To avoid the disinclination of peers to label each other unprofessional, we labeled the professionalism domain as patient‐focused on the tool.
Finally, we included a domain for setting because of an extensive literature demonstrating increased handoff failures in noisy or interruptive settings.[35, 36, 37] We then revised the tool slightly based on our experiences among nurses and students.[13, 38] The final tool included the 6 domains described above and an assessment of overall competency. Each domain was scored on a 9‐point scale and included descriptive anchors at high and low ends of performance. We further divided the scale into 3 main sections: unsatisfactory (score 13), satisfactory (46), and superior (79). We designed 2 tools, 1 to assess the person providing the handoff and 1 to assess the handoff recipient, each with its own descriptive anchors. The recipient tool did not include a content domain (see Supporting Information, Appendix 1, in the online version of this article).
Setting and Subjects
We tested the tool in 2 different urban academic medical centers: the University of Chicago Medicine (UCM) and Yale‐New Haven Hospital (Yale). At UCM, we tested the tool among hospitalists, nurse practitioners, and physician assistants during the Monday and Tuesday morning and Friday evening sign‐out sessions. At Yale, we tested the tool among housestaff during the evening sign‐out session from the primary team to the on‐call covering team.
The UCM is a 550‐bed urban academic medical center in which the nonteaching hospitalist service cares for patients with liver disease, or end‐stage renal or lung disease awaiting transplant, and a small fraction of general medicine and oncology patients when the housestaff service exceeds its cap. No formal training on sign‐out is provided to attending or midlevel providers. The nonteaching hospitalist service operates as a separate service from the housestaff service and consists of 38 hospitalist clinicians (hospitalist attendings, nurse practitioners, and physicians assistants). There are 2 handoffs each day. In the morning the departing night hospitalist hands off to the incoming daytime hospitalist or midlevel provider. These handoffs occur at 7:30 am in a dedicated room. In the evening the daytime hospitalist or midlevel provider hands off to an incoming night hospitalist. This handoff occurs at 5:30 pm or 7:30 pm in a dedicated location. The written sign‐out is maintained on a Microsoft Word (Microsoft Corp., Redmond, WA) document on a password‐protected server and updated daily.
Yale is a 946‐bed urban academic medical center with a large internal medicine training program. Formal sign‐out education that covers the main domains of the tool is provided to new interns during the first 3 months of the year,[19] and a templated electronic medical record‐based electronic written handoff report is produced by the housestaff for all patients.[22] Approximately half of inpatient medicine patients are cared for by housestaff teams, which are entirely separate from the hospitalist service. Housestaff sign‐out occurs between 4 pm and 7 pm every night. At a minimum, the departing intern signs out to the incoming intern; this handoff is typically supervised by at least 1 second‐ or third‐year resident. All patients are signed out verbally; in addition, the written handoff report is provided to the incoming team. Most handoffs occur in a quiet charting room.
Data Collection
Data collection at UCM occurred between March and December 2010 on 3 days of each week: Mondays, Tuesdays, and Fridays. On Mondays and Tuesdays the morning handoffs were observed; on Fridays the evening handoffs were observed. Data collection at Yale occurred between March and May 2011. Only evening handoffs from the primary team to the overnight coverage were observed. At both sites, participants provided verbal informed consent prior to data collection. At the time of an eligible sign‐out session, a research assistant (D.R. at Yale, P.S. at UCM) provided the evaluation tools to all members of the incoming and outgoing teams, and observed the sign‐out session himself. Each person providing a handoff was asked to evaluate the recipient of the handoff; each person receiving a handoff was asked to evaluate the provider of the handoff. In addition, the trained third‐party observer (D.R., P.S.) evaluated both the provider and recipient of the handoff. The external evaluators were trained in principles of effective communication and the use of the tool, with specific review of anchors at each end of each domain. One evaluator had a DO degree and was completing an MPH degree. The second evaluator was an experienced clinical research assistant whose training consisted of supervised observation of 10 handoffs by a physician investigator. At Yale, if a resident was present, she or he was also asked to evaluate both the provider and recipient of the handoff. Consequently, every sign‐out session included at least 2 evaluations of each participant, 1 by a peer evaluator and 1 by a consistent external evaluator who did not know the patients. At Yale, many sign‐outs also included a third evaluation by a resident supervisor.
The study was approved by the institutional review boards at both UCM and Yale.
Statistical Analysis
We obtained mean, median, and interquartile range of scores for each subdomain of the tool as well as the overall assessment of handoff quality. We assessed convergent construct validity by assessing performance of the tool in different contexts. To do so, we determined whether scores differed by type of participant (provider or recipient), by site, by training level of evaluatee, or by type of evaluator (external, resident supervisor, or peer) by using Wilcoxon rank sum tests and Kruskal‐Wallis tests. For the assessment of differences in ratings by training level, we used evaluations of sign‐out providers only, because the 2 sites differed in scores for recipients. We also assessed construct validity by using Spearman rank correlation coefficients to describe the internal consistency of the tool in terms of the correlation between domains of the tool, and we conducted an exploratory factor analysis to gain insight into whether the subdomains of the tool were measuring the same construct. In conducting this analysis, we restricted the dataset to evaluations of sign‐out providers only, and used a principal components estimation method, a promax rotation, and squared multiple correlation communality priors. Finally, we conducted some preliminary studies of reliability by testing whether different types of evaluators provided similar assessments. We calculated a weighted kappa using Fleiss‐Cohen weights for external versus peer scores and again for supervising resident versus peer scores (Yale only). We were not able to assess test‐retest reliability by nature of the sign‐out process. Statistical significance was defined by a P value 0.05, and analyses were performed using SAS 9.2 (SAS Institute, Cary, NC).
RESULTS
A total of 149 handoff sessions were observed: 89 at UCM and 60 at Yale. Each site conducted a similar total number of evaluations: 336 at UCM, 337 at Yale. These sessions involved 97 unique individuals, 34 at UCM and 63 at Yale. Overall scores were high at both sites, but a wide range of scores was applied (Table 1).
| Domain | Provider, N=343 | Recipient, N=330 | P Value | ||||
|---|---|---|---|---|---|---|---|
| Median (IQR) | Mean (SD) | Range | Median (IQR) | Mean (SD) | Range | ||
| |||||||
| Setting | 7 (69) | 7.0 (1.7) | 29 | 7 (69) | 7.3 (1.6) | 29 | 0.05 |
| Organization | 7 (68) | 7.2 (1.5) | 29 | 8 (69) | 7.4 (1.4) | 29 | 0.07 |
| Communication | 7 (69) | 7.2 (1.6) | 19 | 8 (79) | 7.4 (1.5) | 29 | 0.22 |
| Content | 7 (68) | 7.0 (1.6) | 29 | ||||
| Judgment | 8 (68) | 7.3 (1.4) | 39 | 8 (79) | 7.5 (1.4) | 39 | 0.06 |
| Professionalism | 8 (79) | 7.4 (1.5) | 29 | 8 (79) | 7.6 (1.4) | 39 | 0.23 |
| Overall | 7 (68) | 7.1 (1.5) | 29 | 7 (68) | 7.4 (1.4) | 29 | 0.02 |
Handoff Providers
A total of 343 evaluations of handoff providers were completed regarding 67 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism (median: 8; interquartile range [IQR]: 79). The lowest rated domain was content (median: 7; IQR: 68) (Table 1).
Handoff Recipients
A total of 330 evaluations of handoff recipients were completed regarding 58 unique individuals. For each domain, scores spanned the full range from unsatisfactory to superior. The highest rated domain on the handoff provider evaluation tool was professionalism, with a median of 8 (IQR: 79). The lowest rated domain was setting, with a median score of 7 (IQR: 6‐9) (Table 1).
Validity Testing
Comparing provider scores to recipient scores, recipients received significantly higher scores for overall assessment (Table 1). Scores at UCM and Yale were similar in all domains for providers but were slightly lower at UCM in several domains for recipients (see Supporting Information, Appendix 2, in the online version of this article). Scores did not differ significantly by training level (Table 2). Third‐party external evaluators consistently gave lower marks for the same handoff than peer evaluators did (Table 3).
| Domain | Median (Range) | P Value | |||
|---|---|---|---|---|---|
| NP/PA, N=33 | Subintern or Intern, N=170 | Resident, N=44 | Hospitalist, N=95 | ||
| |||||
| Setting | 7 (29) | 7 (39) | 7 (49) | 7 (29) | 0.89 |
| Organization | 8 (49) | 7 (29) | 7 (49) | 8 (39) | 0.11 |
| Communication | 8 (49) | 7 (29) | 7 (49) | 8 (19) | 0.72 |
| Content | 7 (39) | 7 (29) | 7 (49) | 7 (29) | 0.92 |
| Judgment | 8 (59) | 7 (39) | 8 (49) | 8 (49) | 0.09 |
| Professionalism | 8 (49) | 7 (29) | 8 (39) | 8 (49) | 0.82 |
| Overall | 7 (39) | 7 (29) | 8 (49) | 7 (29) | 0.28 |
| Provider, Median (Range) | Recipient, Median (Range) | |||||||
|---|---|---|---|---|---|---|---|---|
| Domain | Peer, N=152 | Resident, Supervisor, N=43 | External, N=147 | P Value | Peer, N=145 | Resident Supervisor, N=43 | External, N=142 | P Value |
| ||||||||
| Setting | 8 (39) | 7 (39) | 7 (29) | 0.02 | 8 (29) | 7 (39) | 7 (29) | <0.001 |
| Organization | 8 (39) | 8 (39) | 7 (29) | 0.18 | 8 (39) | 8 (69) | 7 (29) | <0.001 |
| Communication | 8 (39) | 8 (39) | 7 (19) | <0.001 | 8 (39) | 8 (49) | 7 (29) | <0.001 |
| Content | 8 (39) | 8 (29) | 7 (29) | <0.001 | N/A | N/A | N/A | N/A |
| Judgment | 8 (49) | 8 (39) | 7 (39) | <0.001 | 8 (39) | 8 (49) | 7 (39) | <0.001 |
| Professionalism | 8 (39) | 8 (59) | 7 (29) | 0.02 | 8 (39) | 8 (69) | 7 (39) | <0.001 |
| Overall | 8 (39) | 8 (39) | 7 (29) | 0.001 | 8 (29) | 8 (49) | 7 (29) | <0.001 |
Spearman rank correlation coefficients among the CEX subdomains for provider scores ranged from 0.71 to 0.86, except for setting (Table 4). Setting was less well correlated with the other subdomains, with correlation coefficients ranging from 0.39 to 0.41. Correlations between individual domains and the overall rating ranged from 0.80 to 0.86, except setting, which had a correlation of 0.55. Every correlation was significant at P<0.001. Correlation coefficients for recipient scores were very similar to those for provider scores (see Supporting Information, Appendix 3, in the online version of this article).
| Spearman Correlation Coefficients | ||||||
|---|---|---|---|---|---|---|
| Setting | Organization | Communication | Content | Judgment | Professionalism | |
| ||||||
| Setting | 1.000 | 0.40 | 0.40 | 0.39 | 0.39 | 0.41 |
| Organization | 0.40 | 1.00 | 0.80 | 0.71 | 0.77 | 0.73 |
| Communication | 0.40 | 0.80 | 1.00 | 0.79 | 0.82 | 0.77 |
| Content | 0.39 | 0.71 | 0.79 | 1.00 | 0.80 | 0.74 |
| Judgment | 0.39 | 0.77 | 0.82 | 0.80 | 1.00 | 0.78 |
| Professionalism | 0.41 | 0.73 | 0.77 | 0.74 | 0.78 | 1.00 |
| Overall | 0.55 | 0.80 | 0.84 | 0.83 | 0.86 | 0.82 |
We analyzed 343 provider evaluations in the factor analysis; there were 6 missing values. The scree plot of eigenvalues did not support more than 1 factor; however, the rotated factor pattern for standardized regression coefficients for the first factor and the final communality estimates showed the setting component yielding smaller values than did other scale components (see Supporting Information, Appendix 4, in the online version of this article).
Reliability Testing
Weighted kappa scores for provider evaluations ranged from 0.28 (95% confidence interval [CI]: 0.01, 0.56) for setting to 0.59 (95% CI: 0.38, 0.80) for organization, and were generally higher for resident versus peer comparisons than for external versus peer comparisons. Weighted kappa scores for recipient evaluation were slightly lower for external versus peer evaluations, but agreement was no better than chance for resident versus peer evaluations (Table 5).
| Domain | Provider | Recipient | ||
|---|---|---|---|---|
| External vs Peer, N=144 (95% CI) | Resident vs Peer, N=42 (95% CI) | External vs Peer, N=134 (95% CI) | Resident vs Peer, N=43 (95% CI) | |
| ||||
| Setting | 0.39 (0.24, 0.54) | 0.28 (0.01, 0.56) | 0.34 (0.20, 0.48) | 0.48 (0.27, 0.69) |
| Organization | 0.43 (0.29, 0.58) | 0.59 (0.39, 0.80) | 0.39 (0.22, 0.55) | 0.03 (0.23, 0.29) |
| Communication | 0.34 (0.19, 0.49) | 0.52 (0.37, 0.68) | 0.36 (0.22, 0.51) | 0.02 (0.18, 0.23) |
| Content | 0.38 (0.25, 0.51) | 0.53 (0.27, 0.80) | N/A (N/A) | N/A (N/A) |
| Judgment | 0.36 (0.22, 0.49) | 0.54 (0.25, 0.83) | 0.28 (0.15, 0.42) | 0.12 (0.34, 0.09) |
| Professionalism | 0.47 (0.32, 0.63) | 0.47 (0.23, 0.72) | 0.35 (0.18, 0.51) | 0.01 (0.29, 0.26) |
| Overall | 0.50 (0.36, 0.64) | 0.45 (0.24, 0.67) | 0.31 (0.16, 0.48) | 0.07 (0.20, 0.34) |
DISCUSSION
In this study we found that an evaluation tool for direct observation of housestaff and hospitalists generated a range of scores and was well validated in the sense of performing similarly across 2 different institutions and among both trainees and attendings, while having high internal consistency. However, external evaluators gave consistently lower marks than peer evaluators at both sites, resulting in low reliability when comparing these 2 groups of raters.
It has traditionally been difficult to conduct direct evaluations of handoffs, because they may occur at haphazard times, in variable locations, and without very much advance notice. For this reason, several attempts have been made to incorporate peers in evaluations of handoff practices.[5, 39, 40] Using peers to conduct evaluations also has the advantage that peers are more likely to be familiar with the patients being handed off and might recognize handoff flaws that external evaluators would miss. Nonetheless, peer evaluations have some important liabilities. Peers may be unwilling or unable to provide honest critiques of their colleagues given that they must work closely together for years. Trainee peers may also lack sufficient clinical expertise or experience to accurately assess competence. In our study, we found that peers gave consistently higher marks to their colleagues than did external evaluators, suggesting they may have found it difficult to criticize their colleagues. We conclude that peer evaluation alone is likely an insufficient means of evaluating handoff quality.
Supervising residents gave very similar marks as intern peers, suggesting that they also are unwilling to criticize, are insufficiently experienced to evaluate, or alternatively, that the peer evaluations were reasonable. We suspect the latter is unlikely given that external evaluator scores were consistently lower than peers. One would expect the external evaluators to be biased toward higher scores given that they are not familiar with the patients and are not able to comment on inaccuracies or omissions in the sign‐out.
The tool appeared to perform less well in most cases for recipients than for providers, with a narrower range of scores and low‐weighted kappa scores. Although recipients play a key role in ensuring a high‐quality sign‐out by paying close attention, ensuring it is a bidirectional conversation, asking appropriate questions, and reading back key information, it may be that evaluators were unable to place these activities within the same domains that were used for the provider evaluation. An altogether different recipient evaluation approach may be necessary.[41]
In general, scores were clustered at the top of the score range, as is typical for evaluations. One strategy to spread out scores further would be to refine the tool by adding anchors for satisfactory performance not just the extremes. A second approach might be to reduce the grading scale to only 3 points (unsatisfactory, satisfactory, superior) to force more scores to the middle. However, this approach might limit the discrimination ability of the tool.
We have previously studied the use of this tool among nurses. In that study, we also found consistently higher scores by peers than by external evaluators. We did, however, find a positive effect of experience, in which more experienced nurses received higher scores on average. We did not observe a similar training effect in this study. There are several possible explanations for the lack of a training effect. It is possible that the types of handoffs assessed played a role. At UCM, some assessed handoffs were night staff to day staff, which might be lower quality than day staff to night staff handoffs, whereas at Yale, all handoffs were day to night teams. Thus, average scores at UCM (primarily hospitalists) might have been lowered by the type of handoff provided. Given that hospitalist evaluations were conducted exclusively at UCM and housestaff evaluations exclusively at Yale, lack of difference between hospitalists and housestaff may also have been related to differences in evaluation practice or handoff practice at the 2 sites, not necessarily related to training level. Third, in our experience, attending physicians provide briefer less‐comprehensive sign‐outs than trainees, particularly when communicating with equally experienced attendings; these sign‐outs may appropriately be scored lower on the tool. Fourth, the great majority of the hospitalists at UCM were within 5 years of residency and therefore not very much more experienced than the trainees. Finally, it is possible that skills do not improve over time given widespread lack of observation and feedback during training years for this important skill.
The high internal consistency of most of the subdomains and the loading of all subdomains except setting onto 1 factor are evidence of convergent construct validity, but also suggest that evaluators have difficulty distinguishing among components of sign‐out quality. Internal consistency may also reflect a halo effect, in which scores on different domains are all influenced by a common overall judgment.[42] We are currently testing a shorter version of the tool including domains only for content, professionalism, and setting in addition to overall score. The fact that setting did not correlate as well with the other domains suggests that sign‐out practitioners may not have or exercise control over their surroundings. Consequently, it may ultimately be reasonable to drop this domain from the tool, or alternatively, to refocus on the need to ensure a quiet setting during sign‐out skills training.
There are several limitations to this study. External evaluations were conducted by personnel who were not familiar with the patients, and they may therefore have overestimated the quality of sign‐out. Studying different types of physicians at different sites might have limited our ability to identify differences by training level. As is commonly seen in evaluation studies, scores were skewed to the high end, although we did observe some use of the full range of the tool. Finally, we were limited in our ability to test inter‐rater reliability because of the multiple sources of variability in the data (numerous different raters, with different backgrounds at different settings, rating different individuals).
In summary, we developed a handoff evaluation tool that was easily completed by housestaff and attendings without training, that performed similarly in a variety of different settings at 2 institutions, and that can in principle be used either for peer evaluations or for external evaluations, although peer evaluations may be positively biased. Further work will be done to refine and simplify the tool.
ACKNOWLEDGMENTS
Disclosures: Development and evaluation of the sign‐out CEX was supported by a grant from the Agency for Healthcare Research and Quality (1R03HS018278‐01). Dr. Arora is supported by a National Institute on Aging (K23 AG033763). Dr. Horwitz is supported by the National Institute on Aging (K08 AG038336) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program. Dr. Horwitz is also a Pepper Scholar with support from the Claude D. Pepper Older Americans Independence Center at Yale University School of Medicine (P30AG021342 NIH/NIA). No funding source had any role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality, the National Institute on Aging, the National Institutes of Health, or the American Federation for Aging Research. Dr. Horwitz had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. An earlier version of this work was presented as a poster presentation at the Society of General Internal Medicine Annual Meeting in Orlando, Florida on May 9, 2012. Dr. Rand is now with the Department of Medicine, University of Vermont College of Medicine, Burlington, Vermont. Mr. Staisiunas is now with the Law School, Marquette University, Milwaukee, Wisconsin. The authors declare they have no conflicts of interest.
Appendix
A
PROVIDER HAND‐OFF CEX TOOL
RECIPIENT HAND‐OFF CEX TOOL
Appendix
B
Handoff CEX scores by site of evaluation
| Domain | Provider | Recipient | ||||
| Median (Range) | P‐value | Median (Range) | P‐value | |||
| UC | Yale | UC | Yale | |||
| N=172 | N=170 | N=163 | N=167 | |||
| Setting | 7 (29) | 7 (39) | 0.32 | 7 (29) | 7 (39) | 0.36 |
| Organization | 8 (29) | 7 (39) | 0.30 | 7 (29) | 8 (59) | 0.001 |
| Communication | 7 (19) | 7 (39) | 0.67 | 7 (29) | 8 (49) | 0.03 |
| Content | 7 (29) | 7 (29) | N/A | N/A | N/A | |
| Judgment | 8 (39) | 7 (39) | 0.60 | 7 (39) | 8 (49) | 0.001 |
| Professionalism | 8 (29) | 8 (39) | 0.67 | 8 (39) | 8 (49) | 0.35 |
| Overall | 7 (29) | 7 (39) | 0.41 | 7 (29) | 8 (49) | 0.005 |
Appendix
C
Spearman correlation, recipients (N=330)
| SpearmanCorrelationCoefficients | |||||
| Setting | Organization | Communication | Judgment | Professionalism | |
| Setting | 1.0 | 0.46 | 0.48 | 0.47 | 0.40 |
| Organization | 0.46 | 1.00 | 0.78 | 0.75 | 0.75 |
| Communication | 0.48 | 0.78 | 1.00 | 0.85 | 0.77 |
| Judgment | 0.47 | 0.75 | 0.85 | 1.00 | 0.74 |
| Professionalism | 0.40 | 0.75 | 0.77 | 0.74 | 1.00 |
| Overall | 0.60 | 0.77 | 0.84 | 0.82 | 0.77 |
All p values <0.0001
Appendix
D
Factor analysis results for provider evaluations
| Rotated Factor Pattern (Standardized Regression Coefficients) N=336 | ||
| Factor1 | Factor2 | |
| Organization | 0.64 | 0.27 |
| Communication | 0.79 | 0.16 |
| Content | 0.82 | 0.06 |
| Judgment | 0.86 | 0.06 |
| Professionalism | 0.66 | 0.23 |
| Setting | 0.18 | 0.29 |
- , , , . Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):1173–1177.
- Accreditation Council for Graduate Medical Education. Common program requirements. 2011; http://www.acgme‐2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed August 23, 2011.
- , , , , . Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866–872.
- , , . Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186–194.
- , , , , . Communication failures in patient sign‐out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401–407.
- , , , , . Consequences of inadequate sign‐out for patient care. Arch Intern Med. 2008;168(16):1755–1760.
- , , , . Adequacy of information transferred at resident sign‐out (in‐hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):6–10.
- , , , , . What are covering doctors told about their patients? Analysis of sign‐out among internal medicine house staff. Qual Saf Health Care. 2009;18(4):248–255.
- , . Using direct observation, formal evaluation, and an interactive curriculum to improve the sign‐out practices of internal medicine interns. Acad Med. 2010;85(7):1182–1188.
- , , , . Doctors' handovers in hospitals: a literature review. Qual Saf Health Care. 2011;20(2):128–133.
- , , , et al. Resident sign‐out and patient hand‐offs: opportunities for improvement. Teach Learn Med. 2011;23(2):105–111.
- , , , et al. Use of an appreciative inquiry approach to improve resident sign‐out in an era of multiple shift changes. J Gen Intern Med. 2012;27(3):287–291.
- , , , , , . Validation of a handoff assessment tool: the Handoff CEX [published online ahead of print June 7, 2012]. J Clin Nurs. doi: 10.1111/j.1365–2702.2012.04131.x.
- , , , . The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123(10):795–799.
- , , , . Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):27–33.
- , , , . Assessing the reliability and validity of the mini‐clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900–904.
- , , , , . Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826–830.
- , , , , , . Dropping the baton: a qualitative analysis of failures during the transition from emergency department to inpatient care. Ann Emerg Med. 2009;53(6):701–710.e4.
- , , . Development and implementation of an oral sign‐out skills curriculum. J Gen Intern Med. 2007;22(10):1470–1474.
- , , , . Mixed methods evaluation of oral sign‐out practices. J Gen Intern Med. 2007;22(S1):S114.
- , , , et al. Evaluation of an asynchronous physician voicemail sign‐out for emergency department admissions. Ann Emerg Med. 2009;54(3):368–378.
- , , , et al. An institution‐wide handoff task force to standardise and improve physician handoffs. BMJ Qual Saf. 2012;21(10):863–871.
- , . A model for building a standardized hand‐off protocol. Jt Comm J Qual Patient Saf. 2006;32(11):646–655.
- , , , , . Medication discrepancies in resident sign‐outs and their potential to harm. J Gen Intern Med. 2007;22(12):1751–1755.
- , , , . A theoretical framework and competency‐based approach to improving handoffs. Qual Saf Health Care. 2008;17(1):11–14.
- , , , , , . Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med. 2009;4(7):433–440.
- , , , , . Interns overestimate the effectiveness of their hand‐off communication. Pediatrics. 2010;125(3):491–496.
- , . Improving clinical handovers: creating local solutions for a global problem. Qual Saf Health Care. 2009;18(4):244–245.
- , , , , . Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign‐out. J Hosp Med. 2006;1(4):257–266.
- , , . Standardized sign‐out reduces intern perception of medical errors on the general internal medicine ward. Teach Learn Med. 2009;21(2):121–126.
- , , . SBAR: a shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf. 2006;32(3):167–175.
- . Structuring flexibility: the potential good, bad and ugly in standardisation of handovers. Qual Saf Health Care. 2008;17(1):4–5.
- , , , , . Handoff strategies in settings with high consequences for failure: lessons for health care operations. Int J Qual Health Care. 2004;16(2):125–132.
- , , , , , . Residents' perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758–763.
- , . Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673–676.
- , , , , . Communication loads on clinical staff in the emergency department. Med J Aust. 2002;176(9):415–418.
- , . A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274–284.
- , , , et al. Hand‐off education and evaluation: piloting the observed simulated hand‐off experience (OSHE). J Gen Intern Med. 2010;25(2):129–134.
- , , , et al. Handoffs causing patient harm: a survey of medical and surgical house staff. Jt Comm J Qual Patient Saf. 2008;34(10):563–570.
- , , . A prospective observational study of physician handoff for intensive‐care‐unit‐to‐ward patient transfers. Am J Med. 2011;124(9):860–867.
- , , , , . Characterizing physician listening behavior during hospitalist handoffs using the HEAR checklist (published online ahead of print December 20, 2012]. BMJ Qual Saf. doi:10.1136/bmjqs‐2012‐001138.
- . A constant error in psychological ratings. J Appl Psychol. 1920;4(1):25.
- , , , . Transfers of patient care between house staff on internal medicine wards: a national survey. Arch Intern Med. 2006;166(11):1173–1177.
- Accreditation Council for Graduate Medical Education. Common program requirements. 2011; http://www.acgme‐2010standards.org/pdf/Common_Program_Requirements_07012011.pdf. Accessed August 23, 2011.
- , , , , . Does housestaff discontinuity of care increase the risk for preventable adverse events? Ann Intern Med. 1994;121(11):866–872.
- , , . Communication failures: an insidious contributor to medical mishaps. Acad Med. 2004;79(2):186–194.
- , , , , . Communication failures in patient sign‐out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care. 2005;14(6):401–407.
- , , , , . Consequences of inadequate sign‐out for patient care. Arch Intern Med. 2008;168(16):1755–1760.
- , , , . Adequacy of information transferred at resident sign‐out (in‐hospital handover of care): a prospective survey. Qual Saf Health Care. 2008;17(1):6–10.
- , , , , . What are covering doctors told about their patients? Analysis of sign‐out among internal medicine house staff. Qual Saf Health Care. 2009;18(4):248–255.
- , . Using direct observation, formal evaluation, and an interactive curriculum to improve the sign‐out practices of internal medicine interns. Acad Med. 2010;85(7):1182–1188.
- , , , . Doctors' handovers in hospitals: a literature review. Qual Saf Health Care. 2011;20(2):128–133.
- , , , et al. Resident sign‐out and patient hand‐offs: opportunities for improvement. Teach Learn Med. 2011;23(2):105–111.
- , , , et al. Use of an appreciative inquiry approach to improve resident sign‐out in an era of multiple shift changes. J Gen Intern Med. 2012;27(3):287–291.
- , , , , , . Validation of a handoff assessment tool: the Handoff CEX [published online ahead of print June 7, 2012]. J Clin Nurs. doi: 10.1111/j.1365–2702.2012.04131.x.
- , , , . The mini‐CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995;123(10):795–799.
- , , , . Examiner differences in the mini‐CEX. Adv Health Sci Educ Theory Pract. 1997;2(1):27–33.
- , , , . Assessing the reliability and validity of the mini‐clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900–904.
- , , , , . Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826–830.
- , , , , , . Dropping the baton: a qualitative analysis of failures during the transition from emergency department to inpatient care. Ann Emerg Med. 2009;53(6):701–710.e4.
- , , . Development and implementation of an oral sign‐out skills curriculum. J Gen Intern Med. 2007;22(10):1470–1474.
- , , , . Mixed methods evaluation of oral sign‐out practices. J Gen Intern Med. 2007;22(S1):S114.
- , , , et al. Evaluation of an asynchronous physician voicemail sign‐out for emergency department admissions. Ann Emerg Med. 2009;54(3):368–378.
- , , , et al. An institution‐wide handoff task force to standardise and improve physician handoffs. BMJ Qual Saf. 2012;21(10):863–871.
- , . A model for building a standardized hand‐off protocol. Jt Comm J Qual Patient Saf. 2006;32(11):646–655.
- , , , , . Medication discrepancies in resident sign‐outs and their potential to harm. J Gen Intern Med. 2007;22(12):1751–1755.
- , , , . A theoretical framework and competency‐based approach to improving handoffs. Qual Saf Health Care. 2008;17(1):11–14.
- , , , , , . Hospitalist handoffs: a systematic review and task force recommendations. J Hosp Med. 2009;4(7):433–440.
- , , , , . Interns overestimate the effectiveness of their hand‐off communication. Pediatrics. 2010;125(3):491–496.
- , . Improving clinical handovers: creating local solutions for a global problem. Qual Saf Health Care. 2009;18(4):244–245.
- , , , , . Managing discontinuity in academic medical centers: strategies for a safe and effective resident sign‐out. J Hosp Med. 2006;1(4):257–266.
- , , . Standardized sign‐out reduces intern perception of medical errors on the general internal medicine ward. Teach Learn Med. 2009;21(2):121–126.
- , , . SBAR: a shared mental model for improving communication between clinicians. Jt Comm J Qual Patient Saf. 2006;32(3):167–175.
- . Structuring flexibility: the potential good, bad and ugly in standardisation of handovers. Qual Saf Health Care. 2008;17(1):4–5.
- , , , , . Handoff strategies in settings with high consequences for failure: lessons for health care operations. Int J Qual Health Care. 2004;16(2):125–132.
- , , , , , . Residents' perceptions of professionalism in training and practice: barriers, promoters, and duty hour requirements. J Gen Intern Med. 2006;21(7):758–763.
- , . Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673–676.
- , , , , . Communication loads on clinical staff in the emergency department. Med J Aust. 2002;176(9):415–418.
- , . A systematic review of failures in handoff communication during intrahospital transfers. Jt Comm J Qual Patient Saf. 2011;37(6):274–284.
- , , , et al. Hand‐off education and evaluation: piloting the observed simulated hand‐off experience (OSHE). J Gen Intern Med. 2010;25(2):129–134.
- , , , et al. Handoffs causing patient harm: a survey of medical and surgical house staff. Jt Comm J Qual Patient Saf. 2008;34(10):563–570.
- , , . A prospective observational study of physician handoff for intensive‐care‐unit‐to‐ward patient transfers. Am J Med. 2011;124(9):860–867.
- , , , , . Characterizing physician listening behavior during hospitalist handoffs using the HEAR checklist (published online ahead of print December 20, 2012]. BMJ Qual Saf. doi:10.1136/bmjqs‐2012‐001138.
- . A constant error in psychological ratings. J Appl Psychol. 1920;4(1):25.
Copyright © 2013 Society of Hospital Medicine
In the Literature
In This Edition
Literature at a Glance
A guide to this month’s studies
- ICU volume and outcomes
- Bicarbonate and contrast-induced nephropathy
- Rate of incidental findings in chest CT angiography
- Consequences of adverse-event reporting by physicians
- Hip fracture comanagement
- Niacin vs. ezetimibe in atherosclerosis
- Effect of hospital pharmacists on readmission rates
- Antibiotics in acute exacerbations of COPD
Higher Patient ICU Inflow Volumes Are Associated with Unplanned Readmissions to ICU
Clinical question: Do higher rates of unplanned ICU readmissions occur on days with high patient inflow volumes?
Background: Patients readmitted to ICUs have longer lengths of stay (LOS) and higher rates of in-hospital mortality. Previous studies suggest many ICU readmissions might be due to premature discharge, but there is little evidence evaluating the impact of patient inflow volumes on the incidence of ICU readmissions.
Study design: Retrospective, cohort study.
Setting: Large, urban, tertiary-care academic medical center in Baltimore.
Synopsis: A retrospective review of 3,233 discharges from a neurosciences critical-care unit revealed 95 unplanned readmissions to the ICU setting within 72 hours of discharge to lower level of care. The odds of one or more discharges becoming an unplanned readmission became significantly higher on days when ≥8 patients were admitted to the ICU (OR, 1.66; 95% CI, 1.03-2.68), and the odds of an unplanned readmission were almost five times higher on days when ≥10 patients were admitted, compared with days when ≤9 patients were admitted (OR, 4.99; 95% CI, 2.45-10.17).
After adjusting for patient complexity, patients discharged on days with ≥10 admissions had higher than twice the odds of becoming an unplanned readmission than patients discharged on days with ≤9 admissions (OR, 2.34; 95% CI, 1.27-4.34).
This study was limited to patients in a neurosciences critical-care unit at a single academic medical center. Further research is needed to better understand how high admission volumes lead to increased unplanned readmission rates.
Bottom line: Days with high patient inflow volumes to the ICU are associated with higher rates of unplanned readmissions to the ICU, and the rate of unplanned readmissions becomes significant once a daily threshold of eight admissions is reached.
Citation: Baker DR, Pronovost PJ, Morlock LL, Geocadin RG, Holzmueller CG. Patient flow variability and unplanned readmissions to an intensive care unit. Crit Care Med. 2009;37(11):2882-2887.
Effectiveness of Sodium Bicarbonate in Contrast-Induced Nephropathy Prevention
Clinical question: Is IV sodium bicarbonate effective for prevention of contrast-induced nephropathy (CIN) in high-risk patients?
Background: CIN is a leading cause of acute kidney injury in the hospital setting. Some studies have suggested IV sodium bicarbonate might reduce risk for CIN; other studies challenge this conclusion.
Study design: Systematic review.
Setting: Published and unpublished randomized, controlled trials performed worldwide.
Synopsis: The research examined in this study was composed of randomized, controlled trials that investigated CIN prevention and included IV sodium bicarbonate in one of the treatment groups. Nine published and 15 unpublished trials were selected for a total of 3,563 patients studied. The overall pooled relative risk of CIN in patients treated with IV sodium bicarbonate compared with normal saline was 0.62 (95% CI, 0.45-0.86), though the strength of this evidence was questioned.
Significant heterogeneity across studies was found (I2=49.1%; P=0.004), partially related to substantially greater treatment effect in published (RR 0.43, 95% CI, 0.25-0.75) versus unpublished (RR 0.78, 95% CI, 0.52-1.17) studies. Publication bias was confirmed statistically. Among the published studies, greater treatment effect favoring bicarbonate over saline tended to be reported in those published before 2008, had fewer patients (<200) and events (<15), had measured events within 48 hours, and were studies of lower quality.
No effects regarding the risk of heart failure, the need for dialysis, or mortality were found, though the studies were not specifically designed to investigate those clinical outcomes. Larger studies are needed to better assess these questions.
Bottom line: IV sodium bicarbonate for CIN prevention in high-risk patients could be less effective than previous reports have suggested.
Citation: Zoungas S, Ninomiya T, Huxley R, et al. Systematic review: sodium bicarbonate treatment regimens for the prevention of contrast-induced nephropathy. Ann Intern Med. 2009;151(9):631-638.
Incidental Findings More Frequent than PE in Chest CT Angiograms
Clinical question: What is the prevalence of incidental findings on chest-computed tomographic angiograms (CTAs) ordered by an ED to evaluate for pulmonary embolism (PE)?
Background: CTAs commonly are ordered by ED physicians to assess for PE. While CTAs might yield findings to support an alternate diagnosis to PE, incidental findings might be found that often require further radiographic or clinical followup. The workup of these incidental findings can be burdensome and low-yield.
Study design: Retrospective chart review.
Setting: Single, academic, tertiary-care hospital in North Carolina.
Synopsis: All patients who underwent CTA evaluation for PE in the ED over two enrollment periods were selected; radiographic findings were compiled and their medical records reviewed. Fifty-five of 589 CTAs (9%) were positive for PE. New incidental findings requiring radiographic or clinical followup were found in 141 cases (24%).
Overall, pulmonary nodules were most common, requiring followup in 73 (13%) cases. Adenopathy requiring followup was seen in 51 cases (9%), and new masses requiring followup were found in 12 cases (2%). Findings to support alternate diagnoses for shortness of breath, hypoxemia, or tachycardia were found in 195 patients (33%), most commonly pleural effusion (19%) and infiltrates (11%). Other incidental findings that required less-urgent clinical attention were common with 615 total findings, most frequently nonmalignant bone findings in 144 cases (24%), mild dependent atelectasis in 137 cases (23%), and emphysema in 69 cases (12%).
Bottom line: Incidental findings requiring followup were more than twice as common as PE (24% vs. 9%) in CTAs ordered to evaluate for PE in an ED.
Citation: Hall WB, Truitt SG, Scheunemann LP, et al. The prevalence of clinically relevant incidental findings on chest computed tomographic angiograms ordered to diagnose pulmonary embolism. Arch Intern Med. 2009;169(21):1961-1965.
Patients Don’t Penalize for Adverse-Outcome Disclosure
Clinical question: What patient or clinical characteristics affect the likelihood of physician reporting of an adverse outcome, and how does adverse-outcome disclosure affect patient perceptions of quality of care?
Background: Harmful adverse events (AE), injuries caused by medical management rather than by the underlying condition of the patient, are common in the U.S. Previous studies have focused on physician and provider attitudes about disclosure. Little is known about how characteristics of the AE affect disclosure, and how disclosure affects patients’ perceptions of quality of care.
Study design: Retrospective cohort study.
Setting: Acute-care hospitals in Massachusetts.
Synopsis: Of 4,143 eligible patients, 2,582 (62%) agreed to a telephone interview that asked about patient experiences with adverse events during their hospital stay. Respondents reporting an AE were asked about disclosure by medical staff, effects of adverse events on their hospital course, and the quality of their hospital care.
Of the 845 AEs reported by 608 patients, only 40% were disclosed, defined as “anyone from the hospital explaining why the negative effects occurred.” The majority of the AEs were related to newly prescribed medications (40%) and surgical procedures (34%). Researchers determined that 31% of the AEs were preventable and 75% were severe. In multivariate analysis, disclosure was less likely if the AE was preventable or if patients had long-term effects from the AE. Patients with an AE were more likely to rate the quality of their hospitalization higher if there had been disclosure.
Bottom line: Disclosure of adverse events by medical personnel is low (40%) in hospitalized patients, even though disclosure of adverse events increases patients’ ratings of quality of care.
Citation: López L, Weismann JS, Schneider EC, Weingart SN, Cohen AP, Epstein AM. Disclosure of hospital adverse events and its association with patients’ ratings of the quality of care. Arch Intern Med. 2009;169(20):1888-1894.
Comanagement of Hip-Fracture Patients by Geriatricians Decreases Time to Surgery, LOS, and Complications
Clinical question: Does comanagement of hip-fracture patients by geriatricians and orthopedic surgeons improve short-term outcomes?
Background: Hip fractures in older adults are associated with considerable morbidity and mortality. A model at a single center, where hip fracture patients are comanaged by geriatricians and orthopedic surgeons, demonstrated decreased LOS, readmission rates, and mortality when compared with national data. This study compares results to a usual-care site.
Study design: Retrospective cohort study.
Setting: Community-based teaching hospital and a tertiary-care hospital in Rochester, N.Y.
Synopsis: Researchers enrolled 314 patients with hip fractures. The 193 patients in the intervention group were comanaged by geriatricians and orthopedic surgeons. The 121 patients in the usual-care group were admitted under the care of orthopedic surgeons, and hospitalists were consulted when deemed necessary. Retrospective chart reviews were performed; complications were defined a priori.
When compared with usual care, patients in the intervention group had significantly shorter times to surgery (24.1 hours vs. 37.4 hours), shorter LOS (4.6 days vs. 8.3 days), fewer complications (30.6% vs. 46.3%), including fewer postoperative infections (2.3% vs. 19.8%), cardiac complications (1.0% vs. 7.4%), cases of thromboembolism (0.5% vs. 5.0%), episodes of bleeding (0% vs. 3.3%), and episodes of hypoxia (6.7% vs. 14.1%). There was no difference in inpatient mortality or 30-day readmission rates.
Further assessment comanagement by hospitalists and comanagement by geriatricians is needed.
Bottom line: Perioperative comanagement of hip-fracture patients by geriatricians and orthopedic surgeons significantly improves short-term outcomes.
Citation: Friedman SM, Mendelson DA, Bingham KW, Kates SL. Impact of a comanaged geriatric fracture center on short-term hip fracture outcomes. Arch Intern Med. 2009;169 (18):1712-1717.
Niacin Is Superior to Ezetimibe in Causing Significant Regression of Carotid Intima-Media Thickness when Combined with a Statin
Clinical question: Is ezetimibe superior to niacin for reducing carotid intima-media thickness (CIMT) in patients with coronary artery disease (CAD) already on statin monotherapy?
Background: Statin montherapy significantly reduces the risk of cardiovascular events, and further lowering of this risk can be achieved by reducing the LDL, using statin intensification, or adding ezetimibe, or by raising the HDL levels by adding niacin therapy. This comparative-effectiveness trial compared the efficacy of these two approaches.
Study design: Prospective, randomized, parallel-group, open-label study.
Setting: Tertiary-care military medical center, and private tertiary-care hospital in Washington, D.C.
Synopsis: Three hundred sixty-three patients with known CAD or CAD equivalent were enrolled, and all of the patients were maintained on statin therapy with LDL <100 and HDL <50. Patients were randomized to ezetimibe 10 mg/day or niacin, starting at 500 mg at bedtime and titrated to 2 g/day. Primary endpoint was a mean change in CIMT after 14 months. Secondary endpoints were change in lipid levels, composite of major cardiovascular events, drug discontinuation, and quality of life.
The trial was terminated early after 208 patients had completed the trial. Although ezetimibe showed greater reduction of LDL, niacin showed significantly greater reduction in the progression of CIMT. Patients receiving niacin experienced fewer cardiovascular events (1% vs. 5%) but had higher rates of withdrawal (15% vs. 9%) due to flushing.
Limitations of the study are small sample size, short follow-up period, and use of CIMT as a surrogate marker for clinical endpoint.
Bottom line: Niacin is superior to ezetimibe in reducing CIMT and raising HDL levels and might be more efficacious in reducing cardiovascular risk.
Citation: Taylor AJ, Villines TC, Stanek EJ, et al. Extended-release niacin or ezetimibe and carotid intima-media thickness. N Engl J Med. 2009;361(22):2113-2122.
Pharmacist-Facilitated Hospital Discharge Program Didn’t Reduce Post-Discharge Healthcare Resource Utilization
Clinical question: Does pharmacist-facilitated hospital discharge reduce hospital readmission rates?
Background: Medication discrepancies at the time of discharge often lead to confusion, medical errors, and readmission to the hospital. Patients who are at high risk of medication errors often are on multiple medications and experience adverse drug events upon discharge.
Study design: Prospective cohort study.
Setting: Tertiary-care, academic teaching hospital in Michigan.
Synopsis: One pharmacist alternated between the resident service and hospitalist service every month. The pharmacist monitored the patients being discharged for appropriateness and accuracy of medications. The pharmacist assessed medication therapy, reconciled medications, screened for adherence concerns, counseled and educated patients, and performed post-discharge telephone follow-up.
Primary outcomes were ED visits within 72 hours and readmission rates by day 14 and day 30.
The study found high numbers of medication discrepancies in the control (33.5%) and intervention (59.6%) groups, and these discrepancies were resolved prior to discharge; however, there was no significant impact on post-discharge ED visits, or 14- and 30-day readmission rates. Post-discharge telephone calls reduced 14-day readmission rates.
Bottom line: Pharmacist-facilitated hospital discharge did not significantly reduce post-discharge ED visits or readmissions.
Citation: Walker PC, Bernstein SJ, Jones JN, et al. Impact of a pharmacist-facilitated hospital discharge program. Arch Intern Med. 2009;169(21):2003-2010.
Questionable Antibiotic Benefit for Patients with Acute COPD Exacerbations
Clinical question: Does the addition of antibiotics to systemic corticosteroids provide additional benefits for patients hospitalized with acute exacerbations of chronic obstructive pulmonary disease (AECOPD)?
Background: The role of antibiotics in the treatment of AECOPD is unclear, particularly in addition to systemic corticosteroids. Many of the studies demonstrating the benefit of antibiotics were conducted several decades before systemic steroids were used routinely for the treatment of AECOPD.
Study design: Randomized, double-blinded, placebo-controlled study.
Setting: Two academic teaching hospitals in the Netherlands.
Synopsis: Two hundred sixty-five acute exacerbations of COPD were enrolled in the study, and patients were randomized to a seven-day course of 200 mg/day of doxycycline or placebo. All patients received systemic corticosteroids, nebulized bronchodilator therapy, and physiotherapy. The study found that doxycycline was equivalent to placebo for the primary endpoint of clinical success on day 30; however, doxycycline was superior to placebo for secondary outcomes of clinical success, clinical cure, symptomatic improvement, microbiological success, and reducing open label antibiotic use on day 10, but not on day 30.
Because the population studied had low levels of advanced antimicrobial resistance, the findings might not be generalizable. Results suggested a difference of treatment effect between subgroups based on C-reactive protein values, but further research is needed.
Bottom line: Patients treated with doxycycline for acute exacerbation of COPD had improved clinical outcomes at day 10, but the benefits were not significant at day 30. Data are still equivocal regarding benefits of antibiotics in patients with acute exacerbations of COPD.
Citation: Daniels JM, Snijders D, de Graaff CS, Vlaspolder F, Jansen HM, Boersma WG. Antibiotics in addition to systemic corticosteroids for acute exacerbations of chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2010;181(2):150-157. TH
PEDIATRIC HM LITERATURE
By Mark Shen, MD
Less Is More in the Management of Acute Gastroenteritis
Clinical question: What is the effect of a guideline-training program for pediatricians on the management of acute gastroenteritis (AGE)?
Background: Guidelines recommending oral rehydration, early refeeding, and limited interventions have been developed for AGE management, but uptake by pediatricians has been slow. While individual components of these guidelines have been shown to be effective in comparison to other resource-intensive interventions, outcomes of global guideline implementation have not been studied well.
Study design: Prospective, randomized, controlled trial.
Setting: One hundred forty-nine primary-care pediatricians in Italy.
Synopsis: Pediatricians from the Italian national healthcare system were randomized to AGE guideline training or no training (usual practice; control group). The physicians enrolled 1,309 children in the study. Parents completed daily forms documenting the interventions for, and outcomes of, each illness.
Management by the trained group of pediatricians was in full compliance with the guidelines 66% of the time, as opposed to 33% of the time in the untrained group.
Significant differences between the groups were noted for each basic tenet of the guidelines: oral rehydration use, restrictive diets, medication use, and microbiologic testing. Notably, by both intention-to-treat and per-protocol analysis, the mean duration of diarrhea was shorter in children managed by the trained group of pediatricians.
As underadherence to evidence-based guidelines remains commonplace, effective implementation will become a greater area of focus. Despite a modest effect in an ambulatory population, these investigators demonstrated a clear improvement in outcomes with a simple, two-hour training program. While not studied, it is likely that the cost of care was significantly lower in the guideline group, given the dramatic differences in medication use and testing.
Bottom line: Effective implementation of AGE guidelines will improve outcomes and reduce costs.
Citation: Albano F, Lo Vecchio A, Guarino A. The applicability and efficacy of guidelines for the management of acute gastroenteritis in outpatient children: a field-randomized trial on primary care pediatricians. J Pediatr. 2010;156(2):226-230.
In This Edition
Literature at a Glance
A guide to this month’s studies
- ICU volume and outcomes
- Bicarbonate and contrast-induced nephropathy
- Rate of incidental findings in chest CT angiography
- Consequences of adverse-event reporting by physicians
- Hip fracture comanagement
- Niacin vs. ezetimibe in atherosclerosis
- Effect of hospital pharmacists on readmission rates
- Antibiotics in acute exacerbations of COPD
Higher Patient ICU Inflow Volumes Are Associated with Unplanned Readmissions to ICU
Clinical question: Do higher rates of unplanned ICU readmissions occur on days with high patient inflow volumes?
Background: Patients readmitted to ICUs have longer lengths of stay (LOS) and higher rates of in-hospital mortality. Previous studies suggest many ICU readmissions might be due to premature discharge, but there is little evidence evaluating the impact of patient inflow volumes on the incidence of ICU readmissions.
Study design: Retrospective, cohort study.
Setting: Large, urban, tertiary-care academic medical center in Baltimore.
Synopsis: A retrospective review of 3,233 discharges from a neurosciences critical-care unit revealed 95 unplanned readmissions to the ICU setting within 72 hours of discharge to lower level of care. The odds of one or more discharges becoming an unplanned readmission became significantly higher on days when ≥8 patients were admitted to the ICU (OR, 1.66; 95% CI, 1.03-2.68), and the odds of an unplanned readmission were almost five times higher on days when ≥10 patients were admitted, compared with days when ≤9 patients were admitted (OR, 4.99; 95% CI, 2.45-10.17).
After adjusting for patient complexity, patients discharged on days with ≥10 admissions had higher than twice the odds of becoming an unplanned readmission than patients discharged on days with ≤9 admissions (OR, 2.34; 95% CI, 1.27-4.34).
This study was limited to patients in a neurosciences critical-care unit at a single academic medical center. Further research is needed to better understand how high admission volumes lead to increased unplanned readmission rates.
Bottom line: Days with high patient inflow volumes to the ICU are associated with higher rates of unplanned readmissions to the ICU, and the rate of unplanned readmissions becomes significant once a daily threshold of eight admissions is reached.
Citation: Baker DR, Pronovost PJ, Morlock LL, Geocadin RG, Holzmueller CG. Patient flow variability and unplanned readmissions to an intensive care unit. Crit Care Med. 2009;37(11):2882-2887.
Effectiveness of Sodium Bicarbonate in Contrast-Induced Nephropathy Prevention
Clinical question: Is IV sodium bicarbonate effective for prevention of contrast-induced nephropathy (CIN) in high-risk patients?
Background: CIN is a leading cause of acute kidney injury in the hospital setting. Some studies have suggested IV sodium bicarbonate might reduce risk for CIN; other studies challenge this conclusion.
Study design: Systematic review.
Setting: Published and unpublished randomized, controlled trials performed worldwide.
Synopsis: The research examined in this study was composed of randomized, controlled trials that investigated CIN prevention and included IV sodium bicarbonate in one of the treatment groups. Nine published and 15 unpublished trials were selected for a total of 3,563 patients studied. The overall pooled relative risk of CIN in patients treated with IV sodium bicarbonate compared with normal saline was 0.62 (95% CI, 0.45-0.86), though the strength of this evidence was questioned.
Significant heterogeneity across studies was found (I2=49.1%; P=0.004), partially related to substantially greater treatment effect in published (RR 0.43, 95% CI, 0.25-0.75) versus unpublished (RR 0.78, 95% CI, 0.52-1.17) studies. Publication bias was confirmed statistically. Among the published studies, greater treatment effect favoring bicarbonate over saline tended to be reported in those published before 2008, had fewer patients (<200) and events (<15), had measured events within 48 hours, and were studies of lower quality.
No effects regarding the risk of heart failure, the need for dialysis, or mortality were found, though the studies were not specifically designed to investigate those clinical outcomes. Larger studies are needed to better assess these questions.
Bottom line: IV sodium bicarbonate for CIN prevention in high-risk patients could be less effective than previous reports have suggested.
Citation: Zoungas S, Ninomiya T, Huxley R, et al. Systematic review: sodium bicarbonate treatment regimens for the prevention of contrast-induced nephropathy. Ann Intern Med. 2009;151(9):631-638.
Incidental Findings More Frequent than PE in Chest CT Angiograms
Clinical question: What is the prevalence of incidental findings on chest-computed tomographic angiograms (CTAs) ordered by an ED to evaluate for pulmonary embolism (PE)?
Background: CTAs commonly are ordered by ED physicians to assess for PE. While CTAs might yield findings to support an alternate diagnosis to PE, incidental findings might be found that often require further radiographic or clinical followup. The workup of these incidental findings can be burdensome and low-yield.
Study design: Retrospective chart review.
Setting: Single, academic, tertiary-care hospital in North Carolina.
Synopsis: All patients who underwent CTA evaluation for PE in the ED over two enrollment periods were selected; radiographic findings were compiled and their medical records reviewed. Fifty-five of 589 CTAs (9%) were positive for PE. New incidental findings requiring radiographic or clinical followup were found in 141 cases (24%).
Overall, pulmonary nodules were most common, requiring followup in 73 (13%) cases. Adenopathy requiring followup was seen in 51 cases (9%), and new masses requiring followup were found in 12 cases (2%). Findings to support alternate diagnoses for shortness of breath, hypoxemia, or tachycardia were found in 195 patients (33%), most commonly pleural effusion (19%) and infiltrates (11%). Other incidental findings that required less-urgent clinical attention were common with 615 total findings, most frequently nonmalignant bone findings in 144 cases (24%), mild dependent atelectasis in 137 cases (23%), and emphysema in 69 cases (12%).
Bottom line: Incidental findings requiring followup were more than twice as common as PE (24% vs. 9%) in CTAs ordered to evaluate for PE in an ED.
Citation: Hall WB, Truitt SG, Scheunemann LP, et al. The prevalence of clinically relevant incidental findings on chest computed tomographic angiograms ordered to diagnose pulmonary embolism. Arch Intern Med. 2009;169(21):1961-1965.
Patients Don’t Penalize for Adverse-Outcome Disclosure
Clinical question: What patient or clinical characteristics affect the likelihood of physician reporting of an adverse outcome, and how does adverse-outcome disclosure affect patient perceptions of quality of care?
Background: Harmful adverse events (AE), injuries caused by medical management rather than by the underlying condition of the patient, are common in the U.S. Previous studies have focused on physician and provider attitudes about disclosure. Little is known about how characteristics of the AE affect disclosure, and how disclosure affects patients’ perceptions of quality of care.
Study design: Retrospective cohort study.
Setting: Acute-care hospitals in Massachusetts.
Synopsis: Of 4,143 eligible patients, 2,582 (62%) agreed to a telephone interview that asked about patient experiences with adverse events during their hospital stay. Respondents reporting an AE were asked about disclosure by medical staff, effects of adverse events on their hospital course, and the quality of their hospital care.
Of the 845 AEs reported by 608 patients, only 40% were disclosed, defined as “anyone from the hospital explaining why the negative effects occurred.” The majority of the AEs were related to newly prescribed medications (40%) and surgical procedures (34%). Researchers determined that 31% of the AEs were preventable and 75% were severe. In multivariate analysis, disclosure was less likely if the AE was preventable or if patients had long-term effects from the AE. Patients with an AE were more likely to rate the quality of their hospitalization higher if there had been disclosure.
Bottom line: Disclosure of adverse events by medical personnel is low (40%) in hospitalized patients, even though disclosure of adverse events increases patients’ ratings of quality of care.
Citation: López L, Weismann JS, Schneider EC, Weingart SN, Cohen AP, Epstein AM. Disclosure of hospital adverse events and its association with patients’ ratings of the quality of care. Arch Intern Med. 2009;169(20):1888-1894.
Comanagement of Hip-Fracture Patients by Geriatricians Decreases Time to Surgery, LOS, and Complications
Clinical question: Does comanagement of hip-fracture patients by geriatricians and orthopedic surgeons improve short-term outcomes?
Background: Hip fractures in older adults are associated with considerable morbidity and mortality. A model at a single center, where hip fracture patients are comanaged by geriatricians and orthopedic surgeons, demonstrated decreased LOS, readmission rates, and mortality when compared with national data. This study compares results to a usual-care site.
Study design: Retrospective cohort study.
Setting: Community-based teaching hospital and a tertiary-care hospital in Rochester, N.Y.
Synopsis: Researchers enrolled 314 patients with hip fractures. The 193 patients in the intervention group were comanaged by geriatricians and orthopedic surgeons. The 121 patients in the usual-care group were admitted under the care of orthopedic surgeons, and hospitalists were consulted when deemed necessary. Retrospective chart reviews were performed; complications were defined a priori.
When compared with usual care, patients in the intervention group had significantly shorter times to surgery (24.1 hours vs. 37.4 hours), shorter LOS (4.6 days vs. 8.3 days), fewer complications (30.6% vs. 46.3%), including fewer postoperative infections (2.3% vs. 19.8%), cardiac complications (1.0% vs. 7.4%), cases of thromboembolism (0.5% vs. 5.0%), episodes of bleeding (0% vs. 3.3%), and episodes of hypoxia (6.7% vs. 14.1%). There was no difference in inpatient mortality or 30-day readmission rates.
Further assessment comanagement by hospitalists and comanagement by geriatricians is needed.
Bottom line: Perioperative comanagement of hip-fracture patients by geriatricians and orthopedic surgeons significantly improves short-term outcomes.
Citation: Friedman SM, Mendelson DA, Bingham KW, Kates SL. Impact of a comanaged geriatric fracture center on short-term hip fracture outcomes. Arch Intern Med. 2009;169 (18):1712-1717.
Niacin Is Superior to Ezetimibe in Causing Significant Regression of Carotid Intima-Media Thickness when Combined with a Statin
Clinical question: Is ezetimibe superior to niacin for reducing carotid intima-media thickness (CIMT) in patients with coronary artery disease (CAD) already on statin monotherapy?
Background: Statin montherapy significantly reduces the risk of cardiovascular events, and further lowering of this risk can be achieved by reducing the LDL, using statin intensification, or adding ezetimibe, or by raising the HDL levels by adding niacin therapy. This comparative-effectiveness trial compared the efficacy of these two approaches.
Study design: Prospective, randomized, parallel-group, open-label study.
Setting: Tertiary-care military medical center, and private tertiary-care hospital in Washington, D.C.
Synopsis: Three hundred sixty-three patients with known CAD or CAD equivalent were enrolled, and all of the patients were maintained on statin therapy with LDL <100 and HDL <50. Patients were randomized to ezetimibe 10 mg/day or niacin, starting at 500 mg at bedtime and titrated to 2 g/day. Primary endpoint was a mean change in CIMT after 14 months. Secondary endpoints were change in lipid levels, composite of major cardiovascular events, drug discontinuation, and quality of life.
The trial was terminated early after 208 patients had completed the trial. Although ezetimibe showed greater reduction of LDL, niacin showed significantly greater reduction in the progression of CIMT. Patients receiving niacin experienced fewer cardiovascular events (1% vs. 5%) but had higher rates of withdrawal (15% vs. 9%) due to flushing.
Limitations of the study are small sample size, short follow-up period, and use of CIMT as a surrogate marker for clinical endpoint.
Bottom line: Niacin is superior to ezetimibe in reducing CIMT and raising HDL levels and might be more efficacious in reducing cardiovascular risk.
Citation: Taylor AJ, Villines TC, Stanek EJ, et al. Extended-release niacin or ezetimibe and carotid intima-media thickness. N Engl J Med. 2009;361(22):2113-2122.
Pharmacist-Facilitated Hospital Discharge Program Didn’t Reduce Post-Discharge Healthcare Resource Utilization
Clinical question: Does pharmacist-facilitated hospital discharge reduce hospital readmission rates?
Background: Medication discrepancies at the time of discharge often lead to confusion, medical errors, and readmission to the hospital. Patients who are at high risk of medication errors often are on multiple medications and experience adverse drug events upon discharge.
Study design: Prospective cohort study.
Setting: Tertiary-care, academic teaching hospital in Michigan.
Synopsis: One pharmacist alternated between the resident service and hospitalist service every month. The pharmacist monitored the patients being discharged for appropriateness and accuracy of medications. The pharmacist assessed medication therapy, reconciled medications, screened for adherence concerns, counseled and educated patients, and performed post-discharge telephone follow-up.
Primary outcomes were ED visits within 72 hours and readmission rates by day 14 and day 30.
The study found high numbers of medication discrepancies in the control (33.5%) and intervention (59.6%) groups, and these discrepancies were resolved prior to discharge; however, there was no significant impact on post-discharge ED visits, or 14- and 30-day readmission rates. Post-discharge telephone calls reduced 14-day readmission rates.
Bottom line: Pharmacist-facilitated hospital discharge did not significantly reduce post-discharge ED visits or readmissions.
Citation: Walker PC, Bernstein SJ, Jones JN, et al. Impact of a pharmacist-facilitated hospital discharge program. Arch Intern Med. 2009;169(21):2003-2010.
Questionable Antibiotic Benefit for Patients with Acute COPD Exacerbations
Clinical question: Does the addition of antibiotics to systemic corticosteroids provide additional benefits for patients hospitalized with acute exacerbations of chronic obstructive pulmonary disease (AECOPD)?
Background: The role of antibiotics in the treatment of AECOPD is unclear, particularly in addition to systemic corticosteroids. Many of the studies demonstrating the benefit of antibiotics were conducted several decades before systemic steroids were used routinely for the treatment of AECOPD.
Study design: Randomized, double-blinded, placebo-controlled study.
Setting: Two academic teaching hospitals in the Netherlands.
Synopsis: Two hundred sixty-five acute exacerbations of COPD were enrolled in the study, and patients were randomized to a seven-day course of 200 mg/day of doxycycline or placebo. All patients received systemic corticosteroids, nebulized bronchodilator therapy, and physiotherapy. The study found that doxycycline was equivalent to placebo for the primary endpoint of clinical success on day 30; however, doxycycline was superior to placebo for secondary outcomes of clinical success, clinical cure, symptomatic improvement, microbiological success, and reducing open label antibiotic use on day 10, but not on day 30.
Because the population studied had low levels of advanced antimicrobial resistance, the findings might not be generalizable. Results suggested a difference of treatment effect between subgroups based on C-reactive protein values, but further research is needed.
Bottom line: Patients treated with doxycycline for acute exacerbation of COPD had improved clinical outcomes at day 10, but the benefits were not significant at day 30. Data are still equivocal regarding benefits of antibiotics in patients with acute exacerbations of COPD.
Citation: Daniels JM, Snijders D, de Graaff CS, Vlaspolder F, Jansen HM, Boersma WG. Antibiotics in addition to systemic corticosteroids for acute exacerbations of chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2010;181(2):150-157. TH
PEDIATRIC HM LITERATURE
By Mark Shen, MD
Less Is More in the Management of Acute Gastroenteritis
Clinical question: What is the effect of a guideline-training program for pediatricians on the management of acute gastroenteritis (AGE)?
Background: Guidelines recommending oral rehydration, early refeeding, and limited interventions have been developed for AGE management, but uptake by pediatricians has been slow. While individual components of these guidelines have been shown to be effective in comparison to other resource-intensive interventions, outcomes of global guideline implementation have not been studied well.
Study design: Prospective, randomized, controlled trial.
Setting: One hundred forty-nine primary-care pediatricians in Italy.
Synopsis: Pediatricians from the Italian national healthcare system were randomized to AGE guideline training or no training (usual practice; control group). The physicians enrolled 1,309 children in the study. Parents completed daily forms documenting the interventions for, and outcomes of, each illness.
Management by the trained group of pediatricians was in full compliance with the guidelines 66% of the time, as opposed to 33% of the time in the untrained group.
Significant differences between the groups were noted for each basic tenet of the guidelines: oral rehydration use, restrictive diets, medication use, and microbiologic testing. Notably, by both intention-to-treat and per-protocol analysis, the mean duration of diarrhea was shorter in children managed by the trained group of pediatricians.
As underadherence to evidence-based guidelines remains commonplace, effective implementation will become a greater area of focus. Despite a modest effect in an ambulatory population, these investigators demonstrated a clear improvement in outcomes with a simple, two-hour training program. While not studied, it is likely that the cost of care was significantly lower in the guideline group, given the dramatic differences in medication use and testing.
Bottom line: Effective implementation of AGE guidelines will improve outcomes and reduce costs.
Citation: Albano F, Lo Vecchio A, Guarino A. The applicability and efficacy of guidelines for the management of acute gastroenteritis in outpatient children: a field-randomized trial on primary care pediatricians. J Pediatr. 2010;156(2):226-230.
In This Edition
Literature at a Glance
A guide to this month’s studies
- ICU volume and outcomes
- Bicarbonate and contrast-induced nephropathy
- Rate of incidental findings in chest CT angiography
- Consequences of adverse-event reporting by physicians
- Hip fracture comanagement
- Niacin vs. ezetimibe in atherosclerosis
- Effect of hospital pharmacists on readmission rates
- Antibiotics in acute exacerbations of COPD
Higher Patient ICU Inflow Volumes Are Associated with Unplanned Readmissions to ICU
Clinical question: Do higher rates of unplanned ICU readmissions occur on days with high patient inflow volumes?
Background: Patients readmitted to ICUs have longer lengths of stay (LOS) and higher rates of in-hospital mortality. Previous studies suggest many ICU readmissions might be due to premature discharge, but there is little evidence evaluating the impact of patient inflow volumes on the incidence of ICU readmissions.
Study design: Retrospective, cohort study.
Setting: Large, urban, tertiary-care academic medical center in Baltimore.
Synopsis: A retrospective review of 3,233 discharges from a neurosciences critical-care unit revealed 95 unplanned readmissions to the ICU setting within 72 hours of discharge to lower level of care. The odds of one or more discharges becoming an unplanned readmission became significantly higher on days when ≥8 patients were admitted to the ICU (OR, 1.66; 95% CI, 1.03-2.68), and the odds of an unplanned readmission were almost five times higher on days when ≥10 patients were admitted, compared with days when ≤9 patients were admitted (OR, 4.99; 95% CI, 2.45-10.17).
After adjusting for patient complexity, patients discharged on days with ≥10 admissions had higher than twice the odds of becoming an unplanned readmission than patients discharged on days with ≤9 admissions (OR, 2.34; 95% CI, 1.27-4.34).
This study was limited to patients in a neurosciences critical-care unit at a single academic medical center. Further research is needed to better understand how high admission volumes lead to increased unplanned readmission rates.
Bottom line: Days with high patient inflow volumes to the ICU are associated with higher rates of unplanned readmissions to the ICU, and the rate of unplanned readmissions becomes significant once a daily threshold of eight admissions is reached.
Citation: Baker DR, Pronovost PJ, Morlock LL, Geocadin RG, Holzmueller CG. Patient flow variability and unplanned readmissions to an intensive care unit. Crit Care Med. 2009;37(11):2882-2887.
Effectiveness of Sodium Bicarbonate in Contrast-Induced Nephropathy Prevention
Clinical question: Is IV sodium bicarbonate effective for prevention of contrast-induced nephropathy (CIN) in high-risk patients?
Background: CIN is a leading cause of acute kidney injury in the hospital setting. Some studies have suggested IV sodium bicarbonate might reduce risk for CIN; other studies challenge this conclusion.
Study design: Systematic review.
Setting: Published and unpublished randomized, controlled trials performed worldwide.
Synopsis: The research examined in this study was composed of randomized, controlled trials that investigated CIN prevention and included IV sodium bicarbonate in one of the treatment groups. Nine published and 15 unpublished trials were selected for a total of 3,563 patients studied. The overall pooled relative risk of CIN in patients treated with IV sodium bicarbonate compared with normal saline was 0.62 (95% CI, 0.45-0.86), though the strength of this evidence was questioned.
Significant heterogeneity across studies was found (I2=49.1%; P=0.004), partially related to substantially greater treatment effect in published (RR 0.43, 95% CI, 0.25-0.75) versus unpublished (RR 0.78, 95% CI, 0.52-1.17) studies. Publication bias was confirmed statistically. Among the published studies, greater treatment effect favoring bicarbonate over saline tended to be reported in those published before 2008, had fewer patients (<200) and events (<15), had measured events within 48 hours, and were studies of lower quality.
No effects regarding the risk of heart failure, the need for dialysis, or mortality were found, though the studies were not specifically designed to investigate those clinical outcomes. Larger studies are needed to better assess these questions.
Bottom line: IV sodium bicarbonate for CIN prevention in high-risk patients could be less effective than previous reports have suggested.
Citation: Zoungas S, Ninomiya T, Huxley R, et al. Systematic review: sodium bicarbonate treatment regimens for the prevention of contrast-induced nephropathy. Ann Intern Med. 2009;151(9):631-638.
Incidental Findings More Frequent than PE in Chest CT Angiograms
Clinical question: What is the prevalence of incidental findings on chest-computed tomographic angiograms (CTAs) ordered by an ED to evaluate for pulmonary embolism (PE)?
Background: CTAs commonly are ordered by ED physicians to assess for PE. While CTAs might yield findings to support an alternate diagnosis to PE, incidental findings might be found that often require further radiographic or clinical followup. The workup of these incidental findings can be burdensome and low-yield.
Study design: Retrospective chart review.
Setting: Single, academic, tertiary-care hospital in North Carolina.
Synopsis: All patients who underwent CTA evaluation for PE in the ED over two enrollment periods were selected; radiographic findings were compiled and their medical records reviewed. Fifty-five of 589 CTAs (9%) were positive for PE. New incidental findings requiring radiographic or clinical followup were found in 141 cases (24%).
Overall, pulmonary nodules were most common, requiring followup in 73 (13%) cases. Adenopathy requiring followup was seen in 51 cases (9%), and new masses requiring followup were found in 12 cases (2%). Findings to support alternate diagnoses for shortness of breath, hypoxemia, or tachycardia were found in 195 patients (33%), most commonly pleural effusion (19%) and infiltrates (11%). Other incidental findings that required less-urgent clinical attention were common with 615 total findings, most frequently nonmalignant bone findings in 144 cases (24%), mild dependent atelectasis in 137 cases (23%), and emphysema in 69 cases (12%).
Bottom line: Incidental findings requiring followup were more than twice as common as PE (24% vs. 9%) in CTAs ordered to evaluate for PE in an ED.
Citation: Hall WB, Truitt SG, Scheunemann LP, et al. The prevalence of clinically relevant incidental findings on chest computed tomographic angiograms ordered to diagnose pulmonary embolism. Arch Intern Med. 2009;169(21):1961-1965.
Patients Don’t Penalize for Adverse-Outcome Disclosure
Clinical question: What patient or clinical characteristics affect the likelihood of physician reporting of an adverse outcome, and how does adverse-outcome disclosure affect patient perceptions of quality of care?
Background: Harmful adverse events (AE), injuries caused by medical management rather than by the underlying condition of the patient, are common in the U.S. Previous studies have focused on physician and provider attitudes about disclosure. Little is known about how characteristics of the AE affect disclosure, and how disclosure affects patients’ perceptions of quality of care.
Study design: Retrospective cohort study.
Setting: Acute-care hospitals in Massachusetts.
Synopsis: Of 4,143 eligible patients, 2,582 (62%) agreed to a telephone interview that asked about patient experiences with adverse events during their hospital stay. Respondents reporting an AE were asked about disclosure by medical staff, effects of adverse events on their hospital course, and the quality of their hospital care.
Of the 845 AEs reported by 608 patients, only 40% were disclosed, defined as “anyone from the hospital explaining why the negative effects occurred.” The majority of the AEs were related to newly prescribed medications (40%) and surgical procedures (34%). Researchers determined that 31% of the AEs were preventable and 75% were severe. In multivariate analysis, disclosure was less likely if the AE was preventable or if patients had long-term effects from the AE. Patients with an AE were more likely to rate the quality of their hospitalization higher if there had been disclosure.
Bottom line: Disclosure of adverse events by medical personnel is low (40%) in hospitalized patients, even though disclosure of adverse events increases patients’ ratings of quality of care.
Citation: López L, Weismann JS, Schneider EC, Weingart SN, Cohen AP, Epstein AM. Disclosure of hospital adverse events and its association with patients’ ratings of the quality of care. Arch Intern Med. 2009;169(20):1888-1894.
Comanagement of Hip-Fracture Patients by Geriatricians Decreases Time to Surgery, LOS, and Complications
Clinical question: Does comanagement of hip-fracture patients by geriatricians and orthopedic surgeons improve short-term outcomes?
Background: Hip fractures in older adults are associated with considerable morbidity and mortality. A model at a single center, where hip fracture patients are comanaged by geriatricians and orthopedic surgeons, demonstrated decreased LOS, readmission rates, and mortality when compared with national data. This study compares results to a usual-care site.
Study design: Retrospective cohort study.
Setting: Community-based teaching hospital and a tertiary-care hospital in Rochester, N.Y.
Synopsis: Researchers enrolled 314 patients with hip fractures. The 193 patients in the intervention group were comanaged by geriatricians and orthopedic surgeons. The 121 patients in the usual-care group were admitted under the care of orthopedic surgeons, and hospitalists were consulted when deemed necessary. Retrospective chart reviews were performed; complications were defined a priori.
When compared with usual care, patients in the intervention group had significantly shorter times to surgery (24.1 hours vs. 37.4 hours), shorter LOS (4.6 days vs. 8.3 days), fewer complications (30.6% vs. 46.3%), including fewer postoperative infections (2.3% vs. 19.8%), cardiac complications (1.0% vs. 7.4%), cases of thromboembolism (0.5% vs. 5.0%), episodes of bleeding (0% vs. 3.3%), and episodes of hypoxia (6.7% vs. 14.1%). There was no difference in inpatient mortality or 30-day readmission rates.
Further assessment comanagement by hospitalists and comanagement by geriatricians is needed.
Bottom line: Perioperative comanagement of hip-fracture patients by geriatricians and orthopedic surgeons significantly improves short-term outcomes.
Citation: Friedman SM, Mendelson DA, Bingham KW, Kates SL. Impact of a comanaged geriatric fracture center on short-term hip fracture outcomes. Arch Intern Med. 2009;169 (18):1712-1717.
Niacin Is Superior to Ezetimibe in Causing Significant Regression of Carotid Intima-Media Thickness when Combined with a Statin
Clinical question: Is ezetimibe superior to niacin for reducing carotid intima-media thickness (CIMT) in patients with coronary artery disease (CAD) already on statin monotherapy?
Background: Statin montherapy significantly reduces the risk of cardiovascular events, and further lowering of this risk can be achieved by reducing the LDL, using statin intensification, or adding ezetimibe, or by raising the HDL levels by adding niacin therapy. This comparative-effectiveness trial compared the efficacy of these two approaches.
Study design: Prospective, randomized, parallel-group, open-label study.
Setting: Tertiary-care military medical center, and private tertiary-care hospital in Washington, D.C.
Synopsis: Three hundred sixty-three patients with known CAD or CAD equivalent were enrolled, and all of the patients were maintained on statin therapy with LDL <100 and HDL <50. Patients were randomized to ezetimibe 10 mg/day or niacin, starting at 500 mg at bedtime and titrated to 2 g/day. Primary endpoint was a mean change in CIMT after 14 months. Secondary endpoints were change in lipid levels, composite of major cardiovascular events, drug discontinuation, and quality of life.
The trial was terminated early after 208 patients had completed the trial. Although ezetimibe showed greater reduction of LDL, niacin showed significantly greater reduction in the progression of CIMT. Patients receiving niacin experienced fewer cardiovascular events (1% vs. 5%) but had higher rates of withdrawal (15% vs. 9%) due to flushing.
Limitations of the study are small sample size, short follow-up period, and use of CIMT as a surrogate marker for clinical endpoint.
Bottom line: Niacin is superior to ezetimibe in reducing CIMT and raising HDL levels and might be more efficacious in reducing cardiovascular risk.
Citation: Taylor AJ, Villines TC, Stanek EJ, et al. Extended-release niacin or ezetimibe and carotid intima-media thickness. N Engl J Med. 2009;361(22):2113-2122.
Pharmacist-Facilitated Hospital Discharge Program Didn’t Reduce Post-Discharge Healthcare Resource Utilization
Clinical question: Does pharmacist-facilitated hospital discharge reduce hospital readmission rates?
Background: Medication discrepancies at the time of discharge often lead to confusion, medical errors, and readmission to the hospital. Patients who are at high risk of medication errors often are on multiple medications and experience adverse drug events upon discharge.
Study design: Prospective cohort study.
Setting: Tertiary-care, academic teaching hospital in Michigan.
Synopsis: One pharmacist alternated between the resident service and hospitalist service every month. The pharmacist monitored the patients being discharged for appropriateness and accuracy of medications. The pharmacist assessed medication therapy, reconciled medications, screened for adherence concerns, counseled and educated patients, and performed post-discharge telephone follow-up.
Primary outcomes were ED visits within 72 hours and readmission rates by day 14 and day 30.
The study found high numbers of medication discrepancies in the control (33.5%) and intervention (59.6%) groups, and these discrepancies were resolved prior to discharge; however, there was no significant impact on post-discharge ED visits, or 14- and 30-day readmission rates. Post-discharge telephone calls reduced 14-day readmission rates.
Bottom line: Pharmacist-facilitated hospital discharge did not significantly reduce post-discharge ED visits or readmissions.
Citation: Walker PC, Bernstein SJ, Jones JN, et al. Impact of a pharmacist-facilitated hospital discharge program. Arch Intern Med. 2009;169(21):2003-2010.
Questionable Antibiotic Benefit for Patients with Acute COPD Exacerbations
Clinical question: Does the addition of antibiotics to systemic corticosteroids provide additional benefits for patients hospitalized with acute exacerbations of chronic obstructive pulmonary disease (AECOPD)?
Background: The role of antibiotics in the treatment of AECOPD is unclear, particularly in addition to systemic corticosteroids. Many of the studies demonstrating the benefit of antibiotics were conducted several decades before systemic steroids were used routinely for the treatment of AECOPD.
Study design: Randomized, double-blinded, placebo-controlled study.
Setting: Two academic teaching hospitals in the Netherlands.
Synopsis: Two hundred sixty-five acute exacerbations of COPD were enrolled in the study, and patients were randomized to a seven-day course of 200 mg/day of doxycycline or placebo. All patients received systemic corticosteroids, nebulized bronchodilator therapy, and physiotherapy. The study found that doxycycline was equivalent to placebo for the primary endpoint of clinical success on day 30; however, doxycycline was superior to placebo for secondary outcomes of clinical success, clinical cure, symptomatic improvement, microbiological success, and reducing open label antibiotic use on day 10, but not on day 30.
Because the population studied had low levels of advanced antimicrobial resistance, the findings might not be generalizable. Results suggested a difference of treatment effect between subgroups based on C-reactive protein values, but further research is needed.
Bottom line: Patients treated with doxycycline for acute exacerbation of COPD had improved clinical outcomes at day 10, but the benefits were not significant at day 30. Data are still equivocal regarding benefits of antibiotics in patients with acute exacerbations of COPD.
Citation: Daniels JM, Snijders D, de Graaff CS, Vlaspolder F, Jansen HM, Boersma WG. Antibiotics in addition to systemic corticosteroids for acute exacerbations of chronic obstructive pulmonary disease. Am J Respir Crit Care Med. 2010;181(2):150-157. TH
PEDIATRIC HM LITERATURE
By Mark Shen, MD
Less Is More in the Management of Acute Gastroenteritis
Clinical question: What is the effect of a guideline-training program for pediatricians on the management of acute gastroenteritis (AGE)?
Background: Guidelines recommending oral rehydration, early refeeding, and limited interventions have been developed for AGE management, but uptake by pediatricians has been slow. While individual components of these guidelines have been shown to be effective in comparison to other resource-intensive interventions, outcomes of global guideline implementation have not been studied well.
Study design: Prospective, randomized, controlled trial.
Setting: One hundred forty-nine primary-care pediatricians in Italy.
Synopsis: Pediatricians from the Italian national healthcare system were randomized to AGE guideline training or no training (usual practice; control group). The physicians enrolled 1,309 children in the study. Parents completed daily forms documenting the interventions for, and outcomes of, each illness.
Management by the trained group of pediatricians was in full compliance with the guidelines 66% of the time, as opposed to 33% of the time in the untrained group.
Significant differences between the groups were noted for each basic tenet of the guidelines: oral rehydration use, restrictive diets, medication use, and microbiologic testing. Notably, by both intention-to-treat and per-protocol analysis, the mean duration of diarrhea was shorter in children managed by the trained group of pediatricians.
As underadherence to evidence-based guidelines remains commonplace, effective implementation will become a greater area of focus. Despite a modest effect in an ambulatory population, these investigators demonstrated a clear improvement in outcomes with a simple, two-hour training program. While not studied, it is likely that the cost of care was significantly lower in the guideline group, given the dramatic differences in medication use and testing.
Bottom line: Effective implementation of AGE guidelines will improve outcomes and reduce costs.
Citation: Albano F, Lo Vecchio A, Guarino A. The applicability and efficacy of guidelines for the management of acute gastroenteritis in outpatient children: a field-randomized trial on primary care pediatricians. J Pediatr. 2010;156(2):226-230.