Eric Vittinghoff, PhD

METHODS

RESULTS

DISCUSSION

METHODS

RESULTS

DISCUSSION

METHODS

RESULTS

DISCUSSION

Medication reconciliation, when performed well, effectively identifies discrepancies and reduces medication errors in the hospital setting.[1, 2, 3] This process involves 4 major steps: (1) obtain and document a comprehensive medication history on admission, (2) compare the medication history to medication orders in the hospital and identify and resolve discrepancies, (3) provide the patient with a written list of discharge medications, and (4) educate the patient about their discharge medication regimen.[4, 5, 6]

However, medication reconciliation has been challenging to implement given difficulties with accurate medication information, patients' ability to communicate or remember, and clinician's not having enough time, motivation, or clear roles.[5, 7, 8, 9, 10, 11] Lack of role clarity is generally a barrier to quality improvement; therefore, we studied the perceptions of physicians, nurses, and pharmacists about their roles and responsibilities in completing inpatient medication reconciliation.

METHODS

We independently surveyed attending and resident physicians, nurses, and pharmacists at the University of California San Francisco (UCSF) Medical Center via email who were actively caring for hospitalized patients in April 2010. We collected data on demographics, roles on specific tasks in the medication reconciliation process from admission through discharge, and attitudes and barriers toward medication reconciliation and health information technology systems. Responses to questions used a 4‐point Likert scale. We calculated frequencies and proportions, and used the Fisher exact test to evaluate differences in role agreement for specific medication reconciliation tasks.

RESULTS

Of 256 active clinicians, 78 completed the survey (30.5% overall response rate) providing care in various hospital services (medicine, surgery, cardiology, neurology, pediatrics, obstetrics/gynecology). We received responses from 7 attending physicians (16% response rate), 14 resident physicians (19% response rate), 35 nurses (43% response rate), and 22 pharmacists (43% response rate). Most clinicians worked more than 5 years at UCSF, except residents (14 years).

Overall agreement was poor to fair on whose primary role it was for specific medication reconciliation tasks from admission through discharge (Table 1). Clinicians mainly agreed that it was a physician's responsibility to decide which medications should be continued or discontinued on admission and discharge, although agreement between attending and resident physicians varied. Fisher exact test revealed significant differences in agreement among attending and resident physicians, nurses, and pharmacists to obtain and document a medication history on admission (P=0.001), provide a list of the discharge medications (P<0.001), or educate patients on the postdischarge medication regimen (P<0.001). For these tasks, the physician, nurse, pharmacist or a combination of these clinicians (multiple category) were each identified to be responsible.

Most clinicians believed that maintaining a patient's list of medications improves patient care (94%100% agreement). However, when asked whether clinicians other than yourself should be responsible for an accurate medication list, most nurses (73%) and pharmacists (52%) agreed with this statement compared to resident (50%) and attending physicians (29%). Most clinicians agreed that information technology systems for reconciling medications were complicated, and that patients who do not know their medications, accessing outside medical records, working with inaccurate lists, or nonEnglish‐speaking patients are barriers to reconciliation.

DISCUSSION

We found fair agreement among clinicians that physicians were responsible for reconciling medications on admission and discharge. However, attending and resident physicians each believed it was their primary responsibility, respectively, suggesting the need for better communication between each other. We found poor agreement among clinicians about whose primary role it was to perform the other main steps of medication reconciliation including obtaining and documenting a medication history, and providing a medication list and educating the patient at discharge. For these tasks, there was more confusion among physicians, nurses, and pharmacists. Our findings highlight the need for better role clarity and good communication among team members, particularly at discharge.

Nearly all clinicians agreed that updating patients' medication lists improves patient care. However, most nurses and pharmacists preferred that physicians be responsible for updating information and reconciling medications. They also noted a number of patient‐related and information system barriers to effective reconciliation as others have identified.[7, 8, 9, 10, 11] Although standardizing medication information reporting and implementing technology that can integrate medical records to create, update, and share information between patients and providers can help streamline the medication reconciliation process,[4, 5, 7, 8, 12] these procedures are unlikely to be effective unless good interprofessional communication, role clarity, and clinician understanding of how the system works are in place.

When this study was conducted, our institution's policy required that medication reconciliation be completed, but no specific roles or standard work documents existed. Since then, we have clarified the role of the physician to be responsible for completing medication reconciliation with ancillary help from nurses, pharmacists, and other clinicians, particularly when obtaining a medication history and preparing the patient for discharge. This role clarity has led to focused training and standard work guide documents as guidance to clinicians in different hospital settings about expectations and how to complete medication reconciliation. Clearly, no single reconciliation workflow process will meet the needs of all hospitals. However, it is crucial that interprofessional teams are established with clearly defined roles and responsibilities, and how these roles and responsibilities may change in various situations or services.[8]

Our study had several limitations. We surveyed 1 academic medical center, thus limiting the generalizability of our findings to other organizations or settings. Our small sample size and low response rate could be susceptible to selection bias. However, our findings are similar to other studies.[7, 10, 11] Finally, we included clinicians practicing on various services throughout our hospital, and the local medication reconciliation process could have contributed to the poor agreement. Nonetheless, differences in perceived roles and attitudes for completing medication reconciliation were observed.

In conclusion, lack of agreement among clinicians about their specific roles and responsibilities in the medication reconciliation process exists, and this may result in incomplete reconciliation, inefficiency, duplication of work, and possibly more confusion about a patient's medication regimen. Clinically meaningful and efficient medication reconciliation requires interprofessional teamwork with clear roles and responsibilities, good communication and better information reporting, and tracking systems to successfully combine the steps of medication reconciliation and ensure patient safety.[8, 12]

Disclosures: Funded by research grant NHLBI R01 HL086473 to Dr. Auerbach, and through UCSF‐ CTSI grant number KL2 RR024130 to Dr. Lee from the National Center for Research Resources, the National Center for Advancing Translational Sciences, and the Office of the Director, National Institutes of Health. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. Dr. Lee had full access to all study data and takes responsibility for data integrity and data analysis accuracy. The authors report no conflicts of interest.

Medication reconciliation, when performed well, effectively identifies discrepancies and reduces medication errors in the hospital setting.[1, 2, 3] This process involves 4 major steps: (1) obtain and document a comprehensive medication history on admission, (2) compare the medication history to medication orders in the hospital and identify and resolve discrepancies, (3) provide the patient with a written list of discharge medications, and (4) educate the patient about their discharge medication regimen.[4, 5, 6]

However, medication reconciliation has been challenging to implement given difficulties with accurate medication information, patients' ability to communicate or remember, and clinician's not having enough time, motivation, or clear roles.[5, 7, 8, 9, 10, 11] Lack of role clarity is generally a barrier to quality improvement; therefore, we studied the perceptions of physicians, nurses, and pharmacists about their roles and responsibilities in completing inpatient medication reconciliation.

METHODS

We independently surveyed attending and resident physicians, nurses, and pharmacists at the University of California San Francisco (UCSF) Medical Center via email who were actively caring for hospitalized patients in April 2010. We collected data on demographics, roles on specific tasks in the medication reconciliation process from admission through discharge, and attitudes and barriers toward medication reconciliation and health information technology systems. Responses to questions used a 4‐point Likert scale. We calculated frequencies and proportions, and used the Fisher exact test to evaluate differences in role agreement for specific medication reconciliation tasks.

RESULTS

Of 256 active clinicians, 78 completed the survey (30.5% overall response rate) providing care in various hospital services (medicine, surgery, cardiology, neurology, pediatrics, obstetrics/gynecology). We received responses from 7 attending physicians (16% response rate), 14 resident physicians (19% response rate), 35 nurses (43% response rate), and 22 pharmacists (43% response rate). Most clinicians worked more than 5 years at UCSF, except residents (14 years).

Overall agreement was poor to fair on whose primary role it was for specific medication reconciliation tasks from admission through discharge (Table 1). Clinicians mainly agreed that it was a physician's responsibility to decide which medications should be continued or discontinued on admission and discharge, although agreement between attending and resident physicians varied. Fisher exact test revealed significant differences in agreement among attending and resident physicians, nurses, and pharmacists to obtain and document a medication history on admission (P=0.001), provide a list of the discharge medications (P<0.001), or educate patients on the postdischarge medication regimen (P<0.001). For these tasks, the physician, nurse, pharmacist or a combination of these clinicians (multiple category) were each identified to be responsible.

Most clinicians believed that maintaining a patient's list of medications improves patient care (94%100% agreement). However, when asked whether clinicians other than yourself should be responsible for an accurate medication list, most nurses (73%) and pharmacists (52%) agreed with this statement compared to resident (50%) and attending physicians (29%). Most clinicians agreed that information technology systems for reconciling medications were complicated, and that patients who do not know their medications, accessing outside medical records, working with inaccurate lists, or nonEnglish‐speaking patients are barriers to reconciliation.

DISCUSSION

We found fair agreement among clinicians that physicians were responsible for reconciling medications on admission and discharge. However, attending and resident physicians each believed it was their primary responsibility, respectively, suggesting the need for better communication between each other. We found poor agreement among clinicians about whose primary role it was to perform the other main steps of medication reconciliation including obtaining and documenting a medication history, and providing a medication list and educating the patient at discharge. For these tasks, there was more confusion among physicians, nurses, and pharmacists. Our findings highlight the need for better role clarity and good communication among team members, particularly at discharge.

Nearly all clinicians agreed that updating patients' medication lists improves patient care. However, most nurses and pharmacists preferred that physicians be responsible for updating information and reconciling medications. They also noted a number of patient‐related and information system barriers to effective reconciliation as others have identified.[7, 8, 9, 10, 11] Although standardizing medication information reporting and implementing technology that can integrate medical records to create, update, and share information between patients and providers can help streamline the medication reconciliation process,[4, 5, 7, 8, 12] these procedures are unlikely to be effective unless good interprofessional communication, role clarity, and clinician understanding of how the system works are in place.

When this study was conducted, our institution's policy required that medication reconciliation be completed, but no specific roles or standard work documents existed. Since then, we have clarified the role of the physician to be responsible for completing medication reconciliation with ancillary help from nurses, pharmacists, and other clinicians, particularly when obtaining a medication history and preparing the patient for discharge. This role clarity has led to focused training and standard work guide documents as guidance to clinicians in different hospital settings about expectations and how to complete medication reconciliation. Clearly, no single reconciliation workflow process will meet the needs of all hospitals. However, it is crucial that interprofessional teams are established with clearly defined roles and responsibilities, and how these roles and responsibilities may change in various situations or services.[8]

Our study had several limitations. We surveyed 1 academic medical center, thus limiting the generalizability of our findings to other organizations or settings. Our small sample size and low response rate could be susceptible to selection bias. However, our findings are similar to other studies.[7, 10, 11] Finally, we included clinicians practicing on various services throughout our hospital, and the local medication reconciliation process could have contributed to the poor agreement. Nonetheless, differences in perceived roles and attitudes for completing medication reconciliation were observed.

In conclusion, lack of agreement among clinicians about their specific roles and responsibilities in the medication reconciliation process exists, and this may result in incomplete reconciliation, inefficiency, duplication of work, and possibly more confusion about a patient's medication regimen. Clinically meaningful and efficient medication reconciliation requires interprofessional teamwork with clear roles and responsibilities, good communication and better information reporting, and tracking systems to successfully combine the steps of medication reconciliation and ensure patient safety.[8, 12]

Disclosures: Funded by research grant NHLBI R01 HL086473 to Dr. Auerbach, and through UCSF‐ CTSI grant number KL2 RR024130 to Dr. Lee from the National Center for Research Resources, the National Center for Advancing Translational Sciences, and the Office of the Director, National Institutes of Health. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. Dr. Lee had full access to all study data and takes responsibility for data integrity and data analysis accuracy. The authors report no conflicts of interest.

Medication reconciliation, when performed well, effectively identifies discrepancies and reduces medication errors in the hospital setting.[1, 2, 3] This process involves 4 major steps: (1) obtain and document a comprehensive medication history on admission, (2) compare the medication history to medication orders in the hospital and identify and resolve discrepancies, (3) provide the patient with a written list of discharge medications, and (4) educate the patient about their discharge medication regimen.[4, 5, 6]

However, medication reconciliation has been challenging to implement given difficulties with accurate medication information, patients' ability to communicate or remember, and clinician's not having enough time, motivation, or clear roles.[5, 7, 8, 9, 10, 11] Lack of role clarity is generally a barrier to quality improvement; therefore, we studied the perceptions of physicians, nurses, and pharmacists about their roles and responsibilities in completing inpatient medication reconciliation.

METHODS

We independently surveyed attending and resident physicians, nurses, and pharmacists at the University of California San Francisco (UCSF) Medical Center via email who were actively caring for hospitalized patients in April 2010. We collected data on demographics, roles on specific tasks in the medication reconciliation process from admission through discharge, and attitudes and barriers toward medication reconciliation and health information technology systems. Responses to questions used a 4‐point Likert scale. We calculated frequencies and proportions, and used the Fisher exact test to evaluate differences in role agreement for specific medication reconciliation tasks.

RESULTS

Of 256 active clinicians, 78 completed the survey (30.5% overall response rate) providing care in various hospital services (medicine, surgery, cardiology, neurology, pediatrics, obstetrics/gynecology). We received responses from 7 attending physicians (16% response rate), 14 resident physicians (19% response rate), 35 nurses (43% response rate), and 22 pharmacists (43% response rate). Most clinicians worked more than 5 years at UCSF, except residents (14 years).

Overall agreement was poor to fair on whose primary role it was for specific medication reconciliation tasks from admission through discharge (Table 1). Clinicians mainly agreed that it was a physician's responsibility to decide which medications should be continued or discontinued on admission and discharge, although agreement between attending and resident physicians varied. Fisher exact test revealed significant differences in agreement among attending and resident physicians, nurses, and pharmacists to obtain and document a medication history on admission (P=0.001), provide a list of the discharge medications (P<0.001), or educate patients on the postdischarge medication regimen (P<0.001). For these tasks, the physician, nurse, pharmacist or a combination of these clinicians (multiple category) were each identified to be responsible.

Most clinicians believed that maintaining a patient's list of medications improves patient care (94%100% agreement). However, when asked whether clinicians other than yourself should be responsible for an accurate medication list, most nurses (73%) and pharmacists (52%) agreed with this statement compared to resident (50%) and attending physicians (29%). Most clinicians agreed that information technology systems for reconciling medications were complicated, and that patients who do not know their medications, accessing outside medical records, working with inaccurate lists, or nonEnglish‐speaking patients are barriers to reconciliation.

DISCUSSION

We found fair agreement among clinicians that physicians were responsible for reconciling medications on admission and discharge. However, attending and resident physicians each believed it was their primary responsibility, respectively, suggesting the need for better communication between each other. We found poor agreement among clinicians about whose primary role it was to perform the other main steps of medication reconciliation including obtaining and documenting a medication history, and providing a medication list and educating the patient at discharge. For these tasks, there was more confusion among physicians, nurses, and pharmacists. Our findings highlight the need for better role clarity and good communication among team members, particularly at discharge.

Nearly all clinicians agreed that updating patients' medication lists improves patient care. However, most nurses and pharmacists preferred that physicians be responsible for updating information and reconciling medications. They also noted a number of patient‐related and information system barriers to effective reconciliation as others have identified.[7, 8, 9, 10, 11] Although standardizing medication information reporting and implementing technology that can integrate medical records to create, update, and share information between patients and providers can help streamline the medication reconciliation process,[4, 5, 7, 8, 12] these procedures are unlikely to be effective unless good interprofessional communication, role clarity, and clinician understanding of how the system works are in place.

When this study was conducted, our institution's policy required that medication reconciliation be completed, but no specific roles or standard work documents existed. Since then, we have clarified the role of the physician to be responsible for completing medication reconciliation with ancillary help from nurses, pharmacists, and other clinicians, particularly when obtaining a medication history and preparing the patient for discharge. This role clarity has led to focused training and standard work guide documents as guidance to clinicians in different hospital settings about expectations and how to complete medication reconciliation. Clearly, no single reconciliation workflow process will meet the needs of all hospitals. However, it is crucial that interprofessional teams are established with clearly defined roles and responsibilities, and how these roles and responsibilities may change in various situations or services.[8]

Our study had several limitations. We surveyed 1 academic medical center, thus limiting the generalizability of our findings to other organizations or settings. Our small sample size and low response rate could be susceptible to selection bias. However, our findings are similar to other studies.[7, 10, 11] Finally, we included clinicians practicing on various services throughout our hospital, and the local medication reconciliation process could have contributed to the poor agreement. Nonetheless, differences in perceived roles and attitudes for completing medication reconciliation were observed.

In conclusion, lack of agreement among clinicians about their specific roles and responsibilities in the medication reconciliation process exists, and this may result in incomplete reconciliation, inefficiency, duplication of work, and possibly more confusion about a patient's medication regimen. Clinically meaningful and efficient medication reconciliation requires interprofessional teamwork with clear roles and responsibilities, good communication and better information reporting, and tracking systems to successfully combine the steps of medication reconciliation and ensure patient safety.[8, 12]

Disclosures: Funded by research grant NHLBI R01 HL086473 to Dr. Auerbach, and through UCSF‐ CTSI grant number KL2 RR024130 to Dr. Lee from the National Center for Research Resources, the National Center for Advancing Translational Sciences, and the Office of the Director, National Institutes of Health. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. Dr. Lee had full access to all study data and takes responsibility for data integrity and data analysis accuracy. The authors report no conflicts of interest.

Rapid response teams (RRT) have been promoted by numerous patient safety organizations to reduce preventable in‐hospital deaths.14 Initial studies of RRTs were promising,57 but recent literature,811 including systematic reviews and meta‐analyses, has called these findings into question. Nevertheless, RRTs remain popular in academic and community hospitals worldwide, and many have expanded their roles beyond solely responding to the deteriorating patient.12

Some RRTs, for example, proactively round on seriously ill ward patients and patients recently discharged from the intensive care unit (ICU) in an effort to prevent transitions to higher levels of care. Priestley and colleagues demonstrated that institution of such a team, referred to as a critical care outreach team (CCOT), decreased in‐hospital mortality while possibly increasing hospital length of stay (LOS).13 Three additional single‐center studies from the United Kingdom, where CCOTs are common, specifically examined proactive rounding by CCOTs on the ICU readmission rate: 2 observed no improvement,14, 15 while the third, limited by a small sample size, demonstrated a modest reduction in ICU readmissions.16

We sought to determine the impact of proactive rounding by an RRT on patients discharged from intensive care on the ICU readmission rate, ICU LOS, and in‐hospital mortality of patients discharged from the ICU. We hypothesized that proactive rounding by an RRT would decrease the ICU readmission rate, ICU LOS, and the in‐hospital mortality of patients discharged from the ICU.

MATERIALS AND METHODS

RESULTS

ICU Readmission Rate

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in the ICU readmission rate (6.7% preintervention vs 7.3% postintervention, P = 0.24; Table 1). In the adjusted ITS model, the intervention had no net effect on the odds of ICU readmission (adjusted odds ratio [AOR] for net intervention effect 0.98, 95% confidence interval [CI] 0.42, 2.28), with similar secular trends both preintervention (AOR 1.00 per year, 95% CI 0.97, 1.03), and afterwards (AOR 0.99 per year, 95% CI 0.98, 01.00), and a nonsignificant increase at implementation (Table 2). Figure 1 uses solid lines to show the fitted readmission rates, a hatched line to show the projection of the preintervention secular trend into the postintervention period, and circles to represent adjusted monthly means. The lack of a net intervention effect is indicated by the convergence of the solid and hatched lines 24 months postintervention.

mfig001.jpg

Table 2.Adjusted Impact of Proactive Rounding by an RRT on Clinical Outcomes

Outcome: Summary Effect Measure	Value (95% CI)	P Value
NOTE: ICU readmission model adjusted for attending service, age, race/ethnicity, comorbidities (chronic pulmonary disease, weight loss, anemia, neurological disorders, rheumatoid arthritis, and solid tumors without metastasis), and clustering at the attending physician level. Length of stay model adjusted for attending service, age, race/ethnicity, comorbidities (drug abuse, rheumatoid arthritis, anemia, weight loss, paralysis, pulmonary circulation disease, neurological disorders, hypothyroidism, peptic ulcer disease, and solid tumors without metastasis), and clustering at the attending physician level. Mortality model adjusted for attending service, age, race/ethnicity, comorbidities (weight loss, lymphoma, metastatic cancer, chronic pulmonary and pulmonary circulation disease, and paralysis), and clustering at the attending physician level. Abbreviations: CI, confidence interval; ICU, intensive care unit; RRT, rapid response teams.
ICU readmission rateadjusted odds ratio
Pre‐RRT trend	1.00 (0.97, 1.03)	0.98
Change at RRT implementation	1.24 (0.94, 1.63)	0.13
Post‐RRT trend	0.98 (0.97, 1.00)	0.06
Change in trend	0.98 (0.96, 1.02)	0.39
Net intervention effect	0.92 (0.40, 2.12)	0.85
ICU average length of stayadjusted ratio of means
Trend at 9 mo pre‐RRT	0.98 (0.96, 1.00)	0.05
Trend at 3 mo pre‐RRT	1.02 (0.99, 1.04)	0.19
Change in trend at 3 mo pre‐RRT	1.03 (1.00, 1.07)	0.07
Change at RRT implementation	0.92 (0.80, 1.06)	0.27
Post‐RRT trend	1.00 (0.99, 1.00)	0.35
Change in trend at RRT implementation	0.98 (0.96, 1.01)	0.14
Net intervention effect	0.60 (0.31, 1.18)	0.14
In‐hospital mortality of patients discharged from the ICUadjusted odds ratio
Pre‐RRT trend	1.02 (0.99, 1.06)	0.15
Change at RRT implementation	0.74 (0.51, 1.08)	0.12
Post‐RRT trend	1.00 (0.98, 1.01)	0.68
Change in trend	0.97 (0.94, 1.01)	0.14
Net intervention effect	0.39 (0.14, 1.10)	0.08

ICU Average LOS

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in ICU average LOS (5.1 days preintervention vs 4.9 days postintervention, P = 0.24; Table 1). Trends in ICU LOS may have changed in October 2006 (P = 0.07), decreasing in the first half of the study period (adjusted rate ratio [ARR] 0.98 per year, 95% CI 0.961.00), but did not change significantly thereafter. As with the ICU readmission rate, neither the change in estimated secular trend after implementation (ARR 0.98, 95% CI 0.961.01), nor the net effect of the intervention (ARR 0.62, 95% CI 0.321.22) was statistically significant (Table 2); these results are depicted graphically in Figure 2.

mfig002.jpg

In‐Hospital Mortality of Patients Discharged From the ICU

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in the mortality of patients discharged from the ICU (6.0% preintervention vs 5.5% postintervention, P = 0.24; Table 1). Similarly, in the adjusted ITS model, the intervention had no statistically significant net effect on the mortality outcome (Table 2 and Figure 3).

mfig003.jpg

DISCUSSION

In this single center study, introduction of an RRT that proactively rounded on patients discharged from the ICU did not reduce the ICU readmission rate, ICU LOS, or mortality of patients discharged from the ICU, after accounting for secular trends using robust ITS methods and adjusting for patient level factors.

Our study is consistent with 2 smaller studies that assessed the impact of proactive rounding by a CCOT on ICU readmission rate. Leary and Ridley14 found that proactively rounding by a CCOT did not reduce ICU readmissions or shorten the ICU LOS, although this study was limited by a surprisingly low ICU readmission rate and short ICU LOS prior to the intervention. Another study15 also observed no change in the ICU readmission rate following introduction of a proactively rounding CCOT but noted small reductions in both ICU and hospital mortality. The sole study showing an effect16 observed a lower ICU readmission rate and increased survival to hospital discharge (after excluding do not resuscitate [DNR] patients) with implementation of a CCOT, although some of their findings may be explained by their CCOT's use of palliative care services, a function not featured in our model.

Our study adds to the meta‐analyses and systematic reviews810 that have questioned the hypothesis that a trained and proactive team of caregivers should be able to prevent patients from returning to the ICU. Perhaps one reason why this is not true is that proactive rounding by RRTs may have minimal effect in systems where step‐down beds are readily available. At UCSF, nearly every patient transferred out of the ICU is triaged to a step‐down unit, where telemetry and pulse oximetry are continuously monitored. Despite this, however, our institution's 2 step‐down units generate more calls to our RRT than any other units in the hospital.

We were surprised to see that proactive rounding failed to shorten ICU LOS, hypothesizing that clinicians would be more comfortable discharging patients from the ICU knowing that the RRT would be closely monitoring them afterwards. Although we have no data to support this hypothesis, increased use of the RRT may have also increased step‐down bed use, as patients on the general medicalsurgical floors were transferred to a higher level of care upon recommendation of the RRT, thereby delaying transfers out of the ICU. Moreover, the opening of an additional 16‐bed ICU in October 2008 might have encouraged clinicians to transfer patients back to the ICU simply because beds were more easily accessible than before.

Introduction of proactive rounding by the RRT was also not associated with differences in the mortality rate of patients discharged from the ICU. This finding conflicts with the results of Garcea et al,15 Ball et al,16 and Priestley et al13, all of which found that implementation of a CCOT led to small but statistically significant reductions in in‐hospital mortality. All 3 of these studies, however, examined smaller patient populations (1380, 470, and 2903 patients, respectively), and both the Priestley and Ball studies13, 16 had significantly shorter periods of data collection (24 months and 32 weeks, respectively). Our results are based on models with confidence intervals and P values that account for variability in all 3 underlying effect estimates but assume a linear extrapolation of the preintervention trend. This approach allowed us to flexibly deal with changes related to the intervention, while relying on our large sample size to define time trends not dealt with adequately (or at all) in previous research.

The lack of improvement in outcomes cannot be attributed to immaturity of the RRT or failure of the clinical staff to use the RRT adequately. A prespecified secondary data analysis midway through the postintervention time period demonstrated that the RRT failed to gain efficacy with time with respect to all 3 outcomes. The postintervention RRT was also utilized far more frequently than its predecessor (110.6 vs 2.7 calls per 1000 admissions, respectively), and this degree of RRT utilization far surpasses the dose considered to be indicative of a mature RRT system.12

Our study has several limitations. First, we relied on administrative rather than chart‐collected data to determine the reason for ICU admission, and the APR severity of illness and risk of mortality scores. It seems unlikely, however, that coding deficiencies or biases affected the preintervention and postintervention patient populations differently. Even though we adjusted for all available measures, it is possible that we were not able to account for time trends in all potential confounders. Second, we did not have detailed clinical information on reasons for ICU readmission and whether readmissions occurred before or after the RRT proactively rounded on the patient. Therefore, potential readmissions to the ICU that might have been planned or which would have happened regardless of the presence of the RRT, such as for antibiotic desensitization, could not be accounted for. Third, introduction of proactive rounding by the RRT in June 2007 was accompanied by a change in the RRT's composition, from a physician‐led model to a nurse‐led model. Therefore, inherent differences in the way that physicians and nurses might assess and triage patients could not have been adjusted for. Lastly, this was a retrospective study conducted at a single academic medical center with a specific RRT model, and our results may not be directly applicable to nonteaching settings or to different RRT models.

Our findings raise further questions about the benefits of RRTs as they assume additional roles, such as proactive rounding on patients recently discharged from the ICU. The failure of our RRT to reduce the ICU readmission rate, the ICU average LOS, and the mortality of patients discharged from the ICU raises concerns that the benefits of our RRT are not commensurate with its cost. While defining the degree of impact and underlying mechanisms are worthy of prospective study, hospitals seeking to improve their RRT models should consider how to develop systems that achieve the RRT's promise in measurable ways.

Rapid response teams (RRT) have been promoted by numerous patient safety organizations to reduce preventable in‐hospital deaths.14 Initial studies of RRTs were promising,57 but recent literature,811 including systematic reviews and meta‐analyses, has called these findings into question. Nevertheless, RRTs remain popular in academic and community hospitals worldwide, and many have expanded their roles beyond solely responding to the deteriorating patient.12

Some RRTs, for example, proactively round on seriously ill ward patients and patients recently discharged from the intensive care unit (ICU) in an effort to prevent transitions to higher levels of care. Priestley and colleagues demonstrated that institution of such a team, referred to as a critical care outreach team (CCOT), decreased in‐hospital mortality while possibly increasing hospital length of stay (LOS).13 Three additional single‐center studies from the United Kingdom, where CCOTs are common, specifically examined proactive rounding by CCOTs on the ICU readmission rate: 2 observed no improvement,14, 15 while the third, limited by a small sample size, demonstrated a modest reduction in ICU readmissions.16

We sought to determine the impact of proactive rounding by an RRT on patients discharged from intensive care on the ICU readmission rate, ICU LOS, and in‐hospital mortality of patients discharged from the ICU. We hypothesized that proactive rounding by an RRT would decrease the ICU readmission rate, ICU LOS, and the in‐hospital mortality of patients discharged from the ICU.

MATERIALS AND METHODS

RESULTS

ICU Readmission Rate

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in the ICU readmission rate (6.7% preintervention vs 7.3% postintervention, P = 0.24; Table 1). In the adjusted ITS model, the intervention had no net effect on the odds of ICU readmission (adjusted odds ratio [AOR] for net intervention effect 0.98, 95% confidence interval [CI] 0.42, 2.28), with similar secular trends both preintervention (AOR 1.00 per year, 95% CI 0.97, 1.03), and afterwards (AOR 0.99 per year, 95% CI 0.98, 01.00), and a nonsignificant increase at implementation (Table 2). Figure 1 uses solid lines to show the fitted readmission rates, a hatched line to show the projection of the preintervention secular trend into the postintervention period, and circles to represent adjusted monthly means. The lack of a net intervention effect is indicated by the convergence of the solid and hatched lines 24 months postintervention.

mfig001.jpg

Table 2.Adjusted Impact of Proactive Rounding by an RRT on Clinical Outcomes

Outcome: Summary Effect Measure	Value (95% CI)	P Value
NOTE: ICU readmission model adjusted for attending service, age, race/ethnicity, comorbidities (chronic pulmonary disease, weight loss, anemia, neurological disorders, rheumatoid arthritis, and solid tumors without metastasis), and clustering at the attending physician level. Length of stay model adjusted for attending service, age, race/ethnicity, comorbidities (drug abuse, rheumatoid arthritis, anemia, weight loss, paralysis, pulmonary circulation disease, neurological disorders, hypothyroidism, peptic ulcer disease, and solid tumors without metastasis), and clustering at the attending physician level. Mortality model adjusted for attending service, age, race/ethnicity, comorbidities (weight loss, lymphoma, metastatic cancer, chronic pulmonary and pulmonary circulation disease, and paralysis), and clustering at the attending physician level. Abbreviations: CI, confidence interval; ICU, intensive care unit; RRT, rapid response teams.
ICU readmission rateadjusted odds ratio
Pre‐RRT trend	1.00 (0.97, 1.03)	0.98
Change at RRT implementation	1.24 (0.94, 1.63)	0.13
Post‐RRT trend	0.98 (0.97, 1.00)	0.06
Change in trend	0.98 (0.96, 1.02)	0.39
Net intervention effect	0.92 (0.40, 2.12)	0.85
ICU average length of stayadjusted ratio of means
Trend at 9 mo pre‐RRT	0.98 (0.96, 1.00)	0.05
Trend at 3 mo pre‐RRT	1.02 (0.99, 1.04)	0.19
Change in trend at 3 mo pre‐RRT	1.03 (1.00, 1.07)	0.07
Change at RRT implementation	0.92 (0.80, 1.06)	0.27
Post‐RRT trend	1.00 (0.99, 1.00)	0.35
Change in trend at RRT implementation	0.98 (0.96, 1.01)	0.14
Net intervention effect	0.60 (0.31, 1.18)	0.14
In‐hospital mortality of patients discharged from the ICUadjusted odds ratio
Pre‐RRT trend	1.02 (0.99, 1.06)	0.15
Change at RRT implementation	0.74 (0.51, 1.08)	0.12
Post‐RRT trend	1.00 (0.98, 1.01)	0.68
Change in trend	0.97 (0.94, 1.01)	0.14
Net intervention effect	0.39 (0.14, 1.10)	0.08

ICU Average LOS

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in ICU average LOS (5.1 days preintervention vs 4.9 days postintervention, P = 0.24; Table 1). Trends in ICU LOS may have changed in October 2006 (P = 0.07), decreasing in the first half of the study period (adjusted rate ratio [ARR] 0.98 per year, 95% CI 0.961.00), but did not change significantly thereafter. As with the ICU readmission rate, neither the change in estimated secular trend after implementation (ARR 0.98, 95% CI 0.961.01), nor the net effect of the intervention (ARR 0.62, 95% CI 0.321.22) was statistically significant (Table 2); these results are depicted graphically in Figure 2.

mfig002.jpg

In‐Hospital Mortality of Patients Discharged From the ICU

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in the mortality of patients discharged from the ICU (6.0% preintervention vs 5.5% postintervention, P = 0.24; Table 1). Similarly, in the adjusted ITS model, the intervention had no statistically significant net effect on the mortality outcome (Table 2 and Figure 3).

mfig003.jpg

DISCUSSION

In this single center study, introduction of an RRT that proactively rounded on patients discharged from the ICU did not reduce the ICU readmission rate, ICU LOS, or mortality of patients discharged from the ICU, after accounting for secular trends using robust ITS methods and adjusting for patient level factors.

Our study is consistent with 2 smaller studies that assessed the impact of proactive rounding by a CCOT on ICU readmission rate. Leary and Ridley14 found that proactively rounding by a CCOT did not reduce ICU readmissions or shorten the ICU LOS, although this study was limited by a surprisingly low ICU readmission rate and short ICU LOS prior to the intervention. Another study15 also observed no change in the ICU readmission rate following introduction of a proactively rounding CCOT but noted small reductions in both ICU and hospital mortality. The sole study showing an effect16 observed a lower ICU readmission rate and increased survival to hospital discharge (after excluding do not resuscitate [DNR] patients) with implementation of a CCOT, although some of their findings may be explained by their CCOT's use of palliative care services, a function not featured in our model.

Our study adds to the meta‐analyses and systematic reviews810 that have questioned the hypothesis that a trained and proactive team of caregivers should be able to prevent patients from returning to the ICU. Perhaps one reason why this is not true is that proactive rounding by RRTs may have minimal effect in systems where step‐down beds are readily available. At UCSF, nearly every patient transferred out of the ICU is triaged to a step‐down unit, where telemetry and pulse oximetry are continuously monitored. Despite this, however, our institution's 2 step‐down units generate more calls to our RRT than any other units in the hospital.

We were surprised to see that proactive rounding failed to shorten ICU LOS, hypothesizing that clinicians would be more comfortable discharging patients from the ICU knowing that the RRT would be closely monitoring them afterwards. Although we have no data to support this hypothesis, increased use of the RRT may have also increased step‐down bed use, as patients on the general medicalsurgical floors were transferred to a higher level of care upon recommendation of the RRT, thereby delaying transfers out of the ICU. Moreover, the opening of an additional 16‐bed ICU in October 2008 might have encouraged clinicians to transfer patients back to the ICU simply because beds were more easily accessible than before.

Introduction of proactive rounding by the RRT was also not associated with differences in the mortality rate of patients discharged from the ICU. This finding conflicts with the results of Garcea et al,15 Ball et al,16 and Priestley et al13, all of which found that implementation of a CCOT led to small but statistically significant reductions in in‐hospital mortality. All 3 of these studies, however, examined smaller patient populations (1380, 470, and 2903 patients, respectively), and both the Priestley and Ball studies13, 16 had significantly shorter periods of data collection (24 months and 32 weeks, respectively). Our results are based on models with confidence intervals and P values that account for variability in all 3 underlying effect estimates but assume a linear extrapolation of the preintervention trend. This approach allowed us to flexibly deal with changes related to the intervention, while relying on our large sample size to define time trends not dealt with adequately (or at all) in previous research.

The lack of improvement in outcomes cannot be attributed to immaturity of the RRT or failure of the clinical staff to use the RRT adequately. A prespecified secondary data analysis midway through the postintervention time period demonstrated that the RRT failed to gain efficacy with time with respect to all 3 outcomes. The postintervention RRT was also utilized far more frequently than its predecessor (110.6 vs 2.7 calls per 1000 admissions, respectively), and this degree of RRT utilization far surpasses the dose considered to be indicative of a mature RRT system.12

Our study has several limitations. First, we relied on administrative rather than chart‐collected data to determine the reason for ICU admission, and the APR severity of illness and risk of mortality scores. It seems unlikely, however, that coding deficiencies or biases affected the preintervention and postintervention patient populations differently. Even though we adjusted for all available measures, it is possible that we were not able to account for time trends in all potential confounders. Second, we did not have detailed clinical information on reasons for ICU readmission and whether readmissions occurred before or after the RRT proactively rounded on the patient. Therefore, potential readmissions to the ICU that might have been planned or which would have happened regardless of the presence of the RRT, such as for antibiotic desensitization, could not be accounted for. Third, introduction of proactive rounding by the RRT in June 2007 was accompanied by a change in the RRT's composition, from a physician‐led model to a nurse‐led model. Therefore, inherent differences in the way that physicians and nurses might assess and triage patients could not have been adjusted for. Lastly, this was a retrospective study conducted at a single academic medical center with a specific RRT model, and our results may not be directly applicable to nonteaching settings or to different RRT models.

Our findings raise further questions about the benefits of RRTs as they assume additional roles, such as proactive rounding on patients recently discharged from the ICU. The failure of our RRT to reduce the ICU readmission rate, the ICU average LOS, and the mortality of patients discharged from the ICU raises concerns that the benefits of our RRT are not commensurate with its cost. While defining the degree of impact and underlying mechanisms are worthy of prospective study, hospitals seeking to improve their RRT models should consider how to develop systems that achieve the RRT's promise in measurable ways.

Rapid response teams (RRT) have been promoted by numerous patient safety organizations to reduce preventable in‐hospital deaths.14 Initial studies of RRTs were promising,57 but recent literature,811 including systematic reviews and meta‐analyses, has called these findings into question. Nevertheless, RRTs remain popular in academic and community hospitals worldwide, and many have expanded their roles beyond solely responding to the deteriorating patient.12

Some RRTs, for example, proactively round on seriously ill ward patients and patients recently discharged from the intensive care unit (ICU) in an effort to prevent transitions to higher levels of care. Priestley and colleagues demonstrated that institution of such a team, referred to as a critical care outreach team (CCOT), decreased in‐hospital mortality while possibly increasing hospital length of stay (LOS).13 Three additional single‐center studies from the United Kingdom, where CCOTs are common, specifically examined proactive rounding by CCOTs on the ICU readmission rate: 2 observed no improvement,14, 15 while the third, limited by a small sample size, demonstrated a modest reduction in ICU readmissions.16

We sought to determine the impact of proactive rounding by an RRT on patients discharged from intensive care on the ICU readmission rate, ICU LOS, and in‐hospital mortality of patients discharged from the ICU. We hypothesized that proactive rounding by an RRT would decrease the ICU readmission rate, ICU LOS, and the in‐hospital mortality of patients discharged from the ICU.

MATERIALS AND METHODS

RESULTS

ICU Readmission Rate

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in the ICU readmission rate (6.7% preintervention vs 7.3% postintervention, P = 0.24; Table 1). In the adjusted ITS model, the intervention had no net effect on the odds of ICU readmission (adjusted odds ratio [AOR] for net intervention effect 0.98, 95% confidence interval [CI] 0.42, 2.28), with similar secular trends both preintervention (AOR 1.00 per year, 95% CI 0.97, 1.03), and afterwards (AOR 0.99 per year, 95% CI 0.98, 01.00), and a nonsignificant increase at implementation (Table 2). Figure 1 uses solid lines to show the fitted readmission rates, a hatched line to show the projection of the preintervention secular trend into the postintervention period, and circles to represent adjusted monthly means. The lack of a net intervention effect is indicated by the convergence of the solid and hatched lines 24 months postintervention.

mfig001.jpg

Table 2.Adjusted Impact of Proactive Rounding by an RRT on Clinical Outcomes

Outcome: Summary Effect Measure	Value (95% CI)	P Value
NOTE: ICU readmission model adjusted for attending service, age, race/ethnicity, comorbidities (chronic pulmonary disease, weight loss, anemia, neurological disorders, rheumatoid arthritis, and solid tumors without metastasis), and clustering at the attending physician level. Length of stay model adjusted for attending service, age, race/ethnicity, comorbidities (drug abuse, rheumatoid arthritis, anemia, weight loss, paralysis, pulmonary circulation disease, neurological disorders, hypothyroidism, peptic ulcer disease, and solid tumors without metastasis), and clustering at the attending physician level. Mortality model adjusted for attending service, age, race/ethnicity, comorbidities (weight loss, lymphoma, metastatic cancer, chronic pulmonary and pulmonary circulation disease, and paralysis), and clustering at the attending physician level. Abbreviations: CI, confidence interval; ICU, intensive care unit; RRT, rapid response teams.
ICU readmission rateadjusted odds ratio
Pre‐RRT trend	1.00 (0.97, 1.03)	0.98
Change at RRT implementation	1.24 (0.94, 1.63)	0.13
Post‐RRT trend	0.98 (0.97, 1.00)	0.06
Change in trend	0.98 (0.96, 1.02)	0.39
Net intervention effect	0.92 (0.40, 2.12)	0.85
ICU average length of stayadjusted ratio of means
Trend at 9 mo pre‐RRT	0.98 (0.96, 1.00)	0.05
Trend at 3 mo pre‐RRT	1.02 (0.99, 1.04)	0.19
Change in trend at 3 mo pre‐RRT	1.03 (1.00, 1.07)	0.07
Change at RRT implementation	0.92 (0.80, 1.06)	0.27
Post‐RRT trend	1.00 (0.99, 1.00)	0.35
Change in trend at RRT implementation	0.98 (0.96, 1.01)	0.14
Net intervention effect	0.60 (0.31, 1.18)	0.14
In‐hospital mortality of patients discharged from the ICUadjusted odds ratio
Pre‐RRT trend	1.02 (0.99, 1.06)	0.15
Change at RRT implementation	0.74 (0.51, 1.08)	0.12
Post‐RRT trend	1.00 (0.98, 1.01)	0.68
Change in trend	0.97 (0.94, 1.01)	0.14
Net intervention effect	0.39 (0.14, 1.10)	0.08

ICU Average LOS

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in ICU average LOS (5.1 days preintervention vs 4.9 days postintervention, P = 0.24; Table 1). Trends in ICU LOS may have changed in October 2006 (P = 0.07), decreasing in the first half of the study period (adjusted rate ratio [ARR] 0.98 per year, 95% CI 0.961.00), but did not change significantly thereafter. As with the ICU readmission rate, neither the change in estimated secular trend after implementation (ARR 0.98, 95% CI 0.961.01), nor the net effect of the intervention (ARR 0.62, 95% CI 0.321.22) was statistically significant (Table 2); these results are depicted graphically in Figure 2.

mfig002.jpg

In‐Hospital Mortality of Patients Discharged From the ICU

Introduction of proactive rounding by the RRT was not associated with unadjusted differences in the mortality of patients discharged from the ICU (6.0% preintervention vs 5.5% postintervention, P = 0.24; Table 1). Similarly, in the adjusted ITS model, the intervention had no statistically significant net effect on the mortality outcome (Table 2 and Figure 3).

mfig003.jpg

DISCUSSION

In this single center study, introduction of an RRT that proactively rounded on patients discharged from the ICU did not reduce the ICU readmission rate, ICU LOS, or mortality of patients discharged from the ICU, after accounting for secular trends using robust ITS methods and adjusting for patient level factors.

Our study is consistent with 2 smaller studies that assessed the impact of proactive rounding by a CCOT on ICU readmission rate. Leary and Ridley14 found that proactively rounding by a CCOT did not reduce ICU readmissions or shorten the ICU LOS, although this study was limited by a surprisingly low ICU readmission rate and short ICU LOS prior to the intervention. Another study15 also observed no change in the ICU readmission rate following introduction of a proactively rounding CCOT but noted small reductions in both ICU and hospital mortality. The sole study showing an effect16 observed a lower ICU readmission rate and increased survival to hospital discharge (after excluding do not resuscitate [DNR] patients) with implementation of a CCOT, although some of their findings may be explained by their CCOT's use of palliative care services, a function not featured in our model.

Our study adds to the meta‐analyses and systematic reviews810 that have questioned the hypothesis that a trained and proactive team of caregivers should be able to prevent patients from returning to the ICU. Perhaps one reason why this is not true is that proactive rounding by RRTs may have minimal effect in systems where step‐down beds are readily available. At UCSF, nearly every patient transferred out of the ICU is triaged to a step‐down unit, where telemetry and pulse oximetry are continuously monitored. Despite this, however, our institution's 2 step‐down units generate more calls to our RRT than any other units in the hospital.

We were surprised to see that proactive rounding failed to shorten ICU LOS, hypothesizing that clinicians would be more comfortable discharging patients from the ICU knowing that the RRT would be closely monitoring them afterwards. Although we have no data to support this hypothesis, increased use of the RRT may have also increased step‐down bed use, as patients on the general medicalsurgical floors were transferred to a higher level of care upon recommendation of the RRT, thereby delaying transfers out of the ICU. Moreover, the opening of an additional 16‐bed ICU in October 2008 might have encouraged clinicians to transfer patients back to the ICU simply because beds were more easily accessible than before.

Introduction of proactive rounding by the RRT was also not associated with differences in the mortality rate of patients discharged from the ICU. This finding conflicts with the results of Garcea et al,15 Ball et al,16 and Priestley et al13, all of which found that implementation of a CCOT led to small but statistically significant reductions in in‐hospital mortality. All 3 of these studies, however, examined smaller patient populations (1380, 470, and 2903 patients, respectively), and both the Priestley and Ball studies13, 16 had significantly shorter periods of data collection (24 months and 32 weeks, respectively). Our results are based on models with confidence intervals and P values that account for variability in all 3 underlying effect estimates but assume a linear extrapolation of the preintervention trend. This approach allowed us to flexibly deal with changes related to the intervention, while relying on our large sample size to define time trends not dealt with adequately (or at all) in previous research.

The lack of improvement in outcomes cannot be attributed to immaturity of the RRT or failure of the clinical staff to use the RRT adequately. A prespecified secondary data analysis midway through the postintervention time period demonstrated that the RRT failed to gain efficacy with time with respect to all 3 outcomes. The postintervention RRT was also utilized far more frequently than its predecessor (110.6 vs 2.7 calls per 1000 admissions, respectively), and this degree of RRT utilization far surpasses the dose considered to be indicative of a mature RRT system.12

Our study has several limitations. First, we relied on administrative rather than chart‐collected data to determine the reason for ICU admission, and the APR severity of illness and risk of mortality scores. It seems unlikely, however, that coding deficiencies or biases affected the preintervention and postintervention patient populations differently. Even though we adjusted for all available measures, it is possible that we were not able to account for time trends in all potential confounders. Second, we did not have detailed clinical information on reasons for ICU readmission and whether readmissions occurred before or after the RRT proactively rounded on the patient. Therefore, potential readmissions to the ICU that might have been planned or which would have happened regardless of the presence of the RRT, such as for antibiotic desensitization, could not be accounted for. Third, introduction of proactive rounding by the RRT in June 2007 was accompanied by a change in the RRT's composition, from a physician‐led model to a nurse‐led model. Therefore, inherent differences in the way that physicians and nurses might assess and triage patients could not have been adjusted for. Lastly, this was a retrospective study conducted at a single academic medical center with a specific RRT model, and our results may not be directly applicable to nonteaching settings or to different RRT models.

Our findings raise further questions about the benefits of RRTs as they assume additional roles, such as proactive rounding on patients recently discharged from the ICU. The failure of our RRT to reduce the ICU readmission rate, the ICU average LOS, and the mortality of patients discharged from the ICU raises concerns that the benefits of our RRT are not commensurate with its cost. While defining the degree of impact and underlying mechanisms are worthy of prospective study, hospitals seeking to improve their RRT models should consider how to develop systems that achieve the RRT's promise in measurable ways.

The population of patients in the US with limited English proficiency (LEP)those who speak English less than very well1is substantial and continues to grow.1, 2 Patients with LEP are at risk for lower quality health care overall than their English‐speaking counterparts.38 Professional in‐person interpreters greatly improve spoken communication and quality of care for these patients,4, 9 but their assistance is typically based on the clinical encounter. Particularly if interpreting by phone, interpreters are unlikely to be able to help with materials such as discharge instructions or information sheets meant for family members. Professional written translations of patient educational material help to bridge this gap, allowing clinicians to convey detailed written instructions to patients. However, professional translations must be prepared well in advance of any encounter and can only be used for easily anticipated problems.

The need to translate less common, patient‐specific instructions arises spontaneously in clinical practice, and formally prepared written translations are not useful in these situations. Online translation tools such as GoogleTranslate (available at http://translate.google.com/#) and Babelfish (available at http://babelfish.yahoo.com), a subset of machine translation technology, may help supplement professional in‐person interpretation and formal written translations in that they are ubiquitous, inexpensive, and increasingly well‐known and easy to use.10, 11 Machine translation has already been used in situations where in‐person interpretation is limited. For example, after the earthquake in Haiti, Creole interpreters were not widely available and a hand‐held translation application was quickly developed to meet the needs of relief workers and the population.11 However, data on the accuracy of these tools for critical clinical applications such as patient education are limited. A recent study of computer‐translated pharmacy labels suggested computer‐generated translations were frequently erratic, nonsensical, and even dangerous.12

We conducted a pilot evaluation of an online translation tool as it relates to detailed, complex patient educational material. Our primary goal was to compare the accuracy of a Spanish translation generated by the online tool to that done by a professional agency. Our secondary goals were: 1) to assess whether sentence word length or complexity mediated the accuracy of GT; and 2) to lay the foundation for a more comprehensive study of the accuracy of online translation tools, with respect to patient educational material.

Methods

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

mfig001.jpg

mfig002.jpg

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

mfig003.jpg

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Results

Discussion

In this preliminary study comparing the accuracy of GT to professional translation for patient educational material, we found that GT was inferior to the professional translation in grammatical fluency but generally preserved the content and sense of the original text. Out of 30 GT sentences assessed, there was one substantially erroneous translation that was considered potentially dangerous. Evaluators preferred the professionally translated sentences for complex sentences, but when the English source sentence was simplecontaining a single clausethis preference disappeared.

Like Sharif and Tse,12 we found that for information not arranged in sentences, automated translation sometimes produced nonsensical sentences. In our study, these resulted from an English sentence fragment followed by a bulleted list; in their study, the nonsensical translations resulted from pharmacy labels. The difference in frequency of these errors between our studies may have resulted partly from the translation tool evaluated (GT vs programs used by pharmacies in the Bronx), but may have also been due to our use of machine translation for complete sentencesthe purpose for which it is optimally designed. The hypothesis that machine translations of clinical information are most understandable when used for simple, complete sentences concurs with the methodology used by these tools and requires further study.

GT has the potential to be very useful to clinicians, particularly for those instances when the communication required is both spontaneous and routine or noncritical. For example, in the inpatient setting, patients could communicate diet and other nonclinical requests, as well as ask or answer simple, short questions when the interpreter is not available. In such situations, the low cost and ease of using online translations and machine translation more generally may help to circumvent the tendency of clinicians to get by with inadequate language skills or to avoid communication altogether.25 If used wisely, GT and other online tools could supplement the use of standardized translations and professional interpreters in helping clinicians to overcome language barriers and linguistic inertia, though this will require further assessment.

Ours is a pilot study, and while it suggests a more promising way to use online translation tools, significant further evaluation is required regarding accuracy and applicability prior to widespread use of any machine translation tools for patient care. The document we utilized for evaluation was a professionally translated patient educational brochure provided to individuals starting a complex medication. As online translation tools would most likely not be used in this setting, but rather for spontaneous and less critical patient‐specific instructions, further testing of GT as applied to such scenarios should be considered. Second, we only evaluated GT for English translated into Spanish; its usefulness in other languages will need to be evaluated. It also remains to be seen how easily GT translations will be understood by patients, who may have variable medical understanding and educational attainment as compared to our evaluators. Finally, in this evaluation, we only assessed automated written translation, not automated spoken translation services such as those now available on cellular phones and other mobile devices.11 The latter are based upon translation software with an additional speech recognition interface. These applications may prove to be even more useful than online translation, but the speech recognition component will add an additional layer of potential error and these applications will need to be evaluated on their own merits.

The domains chosen for this study had only moderate interrater reliability as assessed by intraclass correlation and repeatability, with meaning and preference scoring particularly poorly. The latter domains in particular will require more thorough assessment before routine use in online translation assessment. The variability in all domains may have resulted partly from the choice of nonclinicians of different ancestral backgrounds as evaluators. However, this variability is likely better representative of the wide range of patient backgrounds. Because our evaluators were not professional translators, we asked a professional interpreter to grade all sentences to assess the quality of their evaluation. While the interpreter noted slightly fewer errors among the professionally translated sentences (13% vs 22%) and slightly more errors among the GT‐translated sentences (50% vs 39%), and preferred the professional translation slightly more (3.8 vs 3.2), his scores for all of the other measures were almost identical, increasing our confidence in our primary findings (Appendix A). Additionally, since statistical translation is conducted sentence by sentence, in our study evaluators only scored translations at the sentence level. The accuracy of GT for whole paragraphs or entire documents will need to be assessed separately. The correlation between METEOR and the manual evaluation scores was less than in prior studies; while inexpensive to assess, METEOR will have to be recalibrated in optimal circumstanceswith several reference translations available rather than just onebefore it can be used to supplement the assessment of new languages, new materials, other translation technologies, and improvements in a given technology over time for patient educational material.

In summary, GT scored worse in grammar but similarly in content and sense to the professional translation, committing one critical error in translating a complex, fragmented sentence as nonsense. We believe that, with further study and judicious use, GT has the potential to substantially improve clinicians' communication with patients with limited English proficiency in the area of brief spontaneous patient‐specific information, supplementing well the role that professional spoken interpretation and standardized written translations already play.

The population of patients in the US with limited English proficiency (LEP)those who speak English less than very well1is substantial and continues to grow.1, 2 Patients with LEP are at risk for lower quality health care overall than their English‐speaking counterparts.38 Professional in‐person interpreters greatly improve spoken communication and quality of care for these patients,4, 9 but their assistance is typically based on the clinical encounter. Particularly if interpreting by phone, interpreters are unlikely to be able to help with materials such as discharge instructions or information sheets meant for family members. Professional written translations of patient educational material help to bridge this gap, allowing clinicians to convey detailed written instructions to patients. However, professional translations must be prepared well in advance of any encounter and can only be used for easily anticipated problems.

The need to translate less common, patient‐specific instructions arises spontaneously in clinical practice, and formally prepared written translations are not useful in these situations. Online translation tools such as GoogleTranslate (available at http://translate.google.com/#) and Babelfish (available at http://babelfish.yahoo.com), a subset of machine translation technology, may help supplement professional in‐person interpretation and formal written translations in that they are ubiquitous, inexpensive, and increasingly well‐known and easy to use.10, 11 Machine translation has already been used in situations where in‐person interpretation is limited. For example, after the earthquake in Haiti, Creole interpreters were not widely available and a hand‐held translation application was quickly developed to meet the needs of relief workers and the population.11 However, data on the accuracy of these tools for critical clinical applications such as patient education are limited. A recent study of computer‐translated pharmacy labels suggested computer‐generated translations were frequently erratic, nonsensical, and even dangerous.12

We conducted a pilot evaluation of an online translation tool as it relates to detailed, complex patient educational material. Our primary goal was to compare the accuracy of a Spanish translation generated by the online tool to that done by a professional agency. Our secondary goals were: 1) to assess whether sentence word length or complexity mediated the accuracy of GT; and 2) to lay the foundation for a more comprehensive study of the accuracy of online translation tools, with respect to patient educational material.

Methods

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

mfig001.jpg

mfig002.jpg

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

mfig003.jpg

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Results

Discussion

In this preliminary study comparing the accuracy of GT to professional translation for patient educational material, we found that GT was inferior to the professional translation in grammatical fluency but generally preserved the content and sense of the original text. Out of 30 GT sentences assessed, there was one substantially erroneous translation that was considered potentially dangerous. Evaluators preferred the professionally translated sentences for complex sentences, but when the English source sentence was simplecontaining a single clausethis preference disappeared.

Like Sharif and Tse,12 we found that for information not arranged in sentences, automated translation sometimes produced nonsensical sentences. In our study, these resulted from an English sentence fragment followed by a bulleted list; in their study, the nonsensical translations resulted from pharmacy labels. The difference in frequency of these errors between our studies may have resulted partly from the translation tool evaluated (GT vs programs used by pharmacies in the Bronx), but may have also been due to our use of machine translation for complete sentencesthe purpose for which it is optimally designed. The hypothesis that machine translations of clinical information are most understandable when used for simple, complete sentences concurs with the methodology used by these tools and requires further study.

GT has the potential to be very useful to clinicians, particularly for those instances when the communication required is both spontaneous and routine or noncritical. For example, in the inpatient setting, patients could communicate diet and other nonclinical requests, as well as ask or answer simple, short questions when the interpreter is not available. In such situations, the low cost and ease of using online translations and machine translation more generally may help to circumvent the tendency of clinicians to get by with inadequate language skills or to avoid communication altogether.25 If used wisely, GT and other online tools could supplement the use of standardized translations and professional interpreters in helping clinicians to overcome language barriers and linguistic inertia, though this will require further assessment.

Ours is a pilot study, and while it suggests a more promising way to use online translation tools, significant further evaluation is required regarding accuracy and applicability prior to widespread use of any machine translation tools for patient care. The document we utilized for evaluation was a professionally translated patient educational brochure provided to individuals starting a complex medication. As online translation tools would most likely not be used in this setting, but rather for spontaneous and less critical patient‐specific instructions, further testing of GT as applied to such scenarios should be considered. Second, we only evaluated GT for English translated into Spanish; its usefulness in other languages will need to be evaluated. It also remains to be seen how easily GT translations will be understood by patients, who may have variable medical understanding and educational attainment as compared to our evaluators. Finally, in this evaluation, we only assessed automated written translation, not automated spoken translation services such as those now available on cellular phones and other mobile devices.11 The latter are based upon translation software with an additional speech recognition interface. These applications may prove to be even more useful than online translation, but the speech recognition component will add an additional layer of potential error and these applications will need to be evaluated on their own merits.

The domains chosen for this study had only moderate interrater reliability as assessed by intraclass correlation and repeatability, with meaning and preference scoring particularly poorly. The latter domains in particular will require more thorough assessment before routine use in online translation assessment. The variability in all domains may have resulted partly from the choice of nonclinicians of different ancestral backgrounds as evaluators. However, this variability is likely better representative of the wide range of patient backgrounds. Because our evaluators were not professional translators, we asked a professional interpreter to grade all sentences to assess the quality of their evaluation. While the interpreter noted slightly fewer errors among the professionally translated sentences (13% vs 22%) and slightly more errors among the GT‐translated sentences (50% vs 39%), and preferred the professional translation slightly more (3.8 vs 3.2), his scores for all of the other measures were almost identical, increasing our confidence in our primary findings (Appendix A). Additionally, since statistical translation is conducted sentence by sentence, in our study evaluators only scored translations at the sentence level. The accuracy of GT for whole paragraphs or entire documents will need to be assessed separately. The correlation between METEOR and the manual evaluation scores was less than in prior studies; while inexpensive to assess, METEOR will have to be recalibrated in optimal circumstanceswith several reference translations available rather than just onebefore it can be used to supplement the assessment of new languages, new materials, other translation technologies, and improvements in a given technology over time for patient educational material.

In summary, GT scored worse in grammar but similarly in content and sense to the professional translation, committing one critical error in translating a complex, fragmented sentence as nonsense. We believe that, with further study and judicious use, GT has the potential to substantially improve clinicians' communication with patients with limited English proficiency in the area of brief spontaneous patient‐specific information, supplementing well the role that professional spoken interpretation and standardized written translations already play.

The population of patients in the US with limited English proficiency (LEP)those who speak English less than very well1is substantial and continues to grow.1, 2 Patients with LEP are at risk for lower quality health care overall than their English‐speaking counterparts.38 Professional in‐person interpreters greatly improve spoken communication and quality of care for these patients,4, 9 but their assistance is typically based on the clinical encounter. Particularly if interpreting by phone, interpreters are unlikely to be able to help with materials such as discharge instructions or information sheets meant for family members. Professional written translations of patient educational material help to bridge this gap, allowing clinicians to convey detailed written instructions to patients. However, professional translations must be prepared well in advance of any encounter and can only be used for easily anticipated problems.

The need to translate less common, patient‐specific instructions arises spontaneously in clinical practice, and formally prepared written translations are not useful in these situations. Online translation tools such as GoogleTranslate (available at http://translate.google.com/#) and Babelfish (available at http://babelfish.yahoo.com), a subset of machine translation technology, may help supplement professional in‐person interpretation and formal written translations in that they are ubiquitous, inexpensive, and increasingly well‐known and easy to use.10, 11 Machine translation has already been used in situations where in‐person interpretation is limited. For example, after the earthquake in Haiti, Creole interpreters were not widely available and a hand‐held translation application was quickly developed to meet the needs of relief workers and the population.11 However, data on the accuracy of these tools for critical clinical applications such as patient education are limited. A recent study of computer‐translated pharmacy labels suggested computer‐generated translations were frequently erratic, nonsensical, and even dangerous.12

We conducted a pilot evaluation of an online translation tool as it relates to detailed, complex patient educational material. Our primary goal was to compare the accuracy of a Spanish translation generated by the online tool to that done by a professional agency. Our secondary goals were: 1) to assess whether sentence word length or complexity mediated the accuracy of GT; and 2) to lay the foundation for a more comprehensive study of the accuracy of online translation tools, with respect to patient educational material.

Methods

Manual Evaluation: Evaluators, Domains, Scoring

We recruited three nonclinician, bilingual, nativeSpanish‐speaking research assistants as evaluators. The evaluators were all college educated with a Bachelor's degree or higher and were of Mexican, Nicaraguan, and Guatemalan ancestry. Each evaluator received a brief orientation regarding the project, as well as an explanation of the scores, and then proceeded to the blinded evaluation independently.

We asked evaluators to score sentences on Likert scales along five primary domains: fluency, adequacy, meaning, severity, and preference. Fluency and adequacy are well accepted components of machine translation evaluation,18 with fluency being an assessment of grammar and readability ranging from 5 (Perfect fluency; like reading a newspaper) to 1 (No fluency; no appreciable grammar, not understandable) and adequacy being an assessment of information preservation ranging from 5 (100% of information conveyed from the original) to 1 (0% of information conveyed from the original). Given that a sentence can be highly adequate but drastically change the connotation and intent of the sentence (eg, a sentence that contains 75% of the correct words but changes a sentence from take this medication twice a day to take this medication once every two days), we asked evaluators to assess meaning, a measure of connotation and intent maintenance, with scores ranging from 5 (Same meaning as original) to 1 (Totally different meaning from the original).19 Evaluators also assessed severity, a new measure of potential harm if a given sentence was assessed as having errors of any kind, ranging from 5 (Error, no effect on patient care) to 1 (Error, dangerous to patient) with an additional option of N/A (Sentence basically accurate). Finally, evaluators rated a blinded preference (also a new measure) for either of two translated sentences, ranging from Strongly prefer translation #1 to Strongly prefer translation #2. The order of the sentences was random (eg, sometimes the professional translation was first and sometimes the GT translation was). We subsequently converted this to preference for the professional translation, ranging from 5 (Strongly prefer the professional translation) to 1 (Strongly prefer the GT translation) in order to standardize the responses (Figures 1 and 2).

mfig001.jpg

mfig002.jpg

The overall flow of the study is given in Figure 3. Each evaluator initially scored 20 sentences translated by GT and 10 sentences translated professionally along the first four domains. All 30 of these sentences were randomly selected from the original, 263‐sentence pamphlet. For fluency, evaluators had access only to the translated sentence to be scored; for adequacy, meaning, and severity, they had access to both the translated sentence and the original English sentence. Ten of the 30 sentences were further selected randomly for scoring on the preference domain. For these 10 sentences, evaluators compared the GT and professional translations of the same sentence (with the original English sentence available as a reference) and indicated a preference, for any reason, for one translation or the other. Evaluators were blinded to the technique of translation (GT or professional) for all scored sentences and domains. We chose twice as many sentences from the GT preparations for the first four domains to maximize measurements for the translation technology we were evaluating, with the smaller number of professional translations serving as controls.

mfig003.jpg

After scoring the first 30 sentences, evaluators met with one of the authors (R.R.K.) to discuss and consolidate their approach to scoring. They then scored an additional 10 GT‐translated sentences and 5 professionally translated sentences for the first four domains, and 9 of these 15 sentences for preference, to see if the meeting changed their scoring approach. These sentences were selected randomly from the original, 263‐sentence pamphlet, excluding the 30 evaluated in the previous step.

Results

Discussion

In this preliminary study comparing the accuracy of GT to professional translation for patient educational material, we found that GT was inferior to the professional translation in grammatical fluency but generally preserved the content and sense of the original text. Out of 30 GT sentences assessed, there was one substantially erroneous translation that was considered potentially dangerous. Evaluators preferred the professionally translated sentences for complex sentences, but when the English source sentence was simplecontaining a single clausethis preference disappeared.

Like Sharif and Tse,12 we found that for information not arranged in sentences, automated translation sometimes produced nonsensical sentences. In our study, these resulted from an English sentence fragment followed by a bulleted list; in their study, the nonsensical translations resulted from pharmacy labels. The difference in frequency of these errors between our studies may have resulted partly from the translation tool evaluated (GT vs programs used by pharmacies in the Bronx), but may have also been due to our use of machine translation for complete sentencesthe purpose for which it is optimally designed. The hypothesis that machine translations of clinical information are most understandable when used for simple, complete sentences concurs with the methodology used by these tools and requires further study.

GT has the potential to be very useful to clinicians, particularly for those instances when the communication required is both spontaneous and routine or noncritical. For example, in the inpatient setting, patients could communicate diet and other nonclinical requests, as well as ask or answer simple, short questions when the interpreter is not available. In such situations, the low cost and ease of using online translations and machine translation more generally may help to circumvent the tendency of clinicians to get by with inadequate language skills or to avoid communication altogether.25 If used wisely, GT and other online tools could supplement the use of standardized translations and professional interpreters in helping clinicians to overcome language barriers and linguistic inertia, though this will require further assessment.

Ours is a pilot study, and while it suggests a more promising way to use online translation tools, significant further evaluation is required regarding accuracy and applicability prior to widespread use of any machine translation tools for patient care. The document we utilized for evaluation was a professionally translated patient educational brochure provided to individuals starting a complex medication. As online translation tools would most likely not be used in this setting, but rather for spontaneous and less critical patient‐specific instructions, further testing of GT as applied to such scenarios should be considered. Second, we only evaluated GT for English translated into Spanish; its usefulness in other languages will need to be evaluated. It also remains to be seen how easily GT translations will be understood by patients, who may have variable medical understanding and educational attainment as compared to our evaluators. Finally, in this evaluation, we only assessed automated written translation, not automated spoken translation services such as those now available on cellular phones and other mobile devices.11 The latter are based upon translation software with an additional speech recognition interface. These applications may prove to be even more useful than online translation, but the speech recognition component will add an additional layer of potential error and these applications will need to be evaluated on their own merits.

The domains chosen for this study had only moderate interrater reliability as assessed by intraclass correlation and repeatability, with meaning and preference scoring particularly poorly. The latter domains in particular will require more thorough assessment before routine use in online translation assessment. The variability in all domains may have resulted partly from the choice of nonclinicians of different ancestral backgrounds as evaluators. However, this variability is likely better representative of the wide range of patient backgrounds. Because our evaluators were not professional translators, we asked a professional interpreter to grade all sentences to assess the quality of their evaluation. While the interpreter noted slightly fewer errors among the professionally translated sentences (13% vs 22%) and slightly more errors among the GT‐translated sentences (50% vs 39%), and preferred the professional translation slightly more (3.8 vs 3.2), his scores for all of the other measures were almost identical, increasing our confidence in our primary findings (Appendix A). Additionally, since statistical translation is conducted sentence by sentence, in our study evaluators only scored translations at the sentence level. The accuracy of GT for whole paragraphs or entire documents will need to be assessed separately. The correlation between METEOR and the manual evaluation scores was less than in prior studies; while inexpensive to assess, METEOR will have to be recalibrated in optimal circumstanceswith several reference translations available rather than just onebefore it can be used to supplement the assessment of new languages, new materials, other translation technologies, and improvements in a given technology over time for patient educational material.

In summary, GT scored worse in grammar but similarly in content and sense to the professional translation, committing one critical error in translating a complex, fragmented sentence as nonsense. We believe that, with further study and judicious use, GT has the potential to substantially improve clinicians' communication with patients with limited English proficiency in the area of brief spontaneous patient‐specific information, supplementing well the role that professional spoken interpretation and standardized written translations already play.