Article Type
Changed
Sun, 05/21/2017 - 13:33
Display Headline
Factors associated with medication warning acceptance for hospitalized adults

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

Setting and Caregivers

Johns Hopkins Bayview Medical Center (JHBMC) is a 400‐bed academic medical center serving southeastern Baltimore, Maryland. Prescribing caregivers include residents and fellows who rotate to both JHBMC and Johns Hopkins Hospital, internal medicine hospitalists, other attending physicians (including teaching attendings for all departments, and hospitalists and clinical associates for departments other than internal medicine), and nurse practitioners and physician assistants from most JHBMC departments. Nearly 100% of patients on the surgery, obstetrics/gynecology, neurology, psychiatry, and chemical dependence services are hospitalized on units dedicated to their respective specialty, and the same is true for approximately 95% of medicine patients.

Order Entry

JHBMC began using a client‐server order entry system by MEDITECH (Westwood, MA) in July 2003. Provider order entry was phased in beginning in October 2003 and completed by the end of 2004. MEDITECH version 5.64 was being used during the study period. Medications may generate duplicate, interaction, allergy, adverse reaction, and dose warnings during a patient ordering session each time they are ordered. Duplicate warnings are generated when the same medication (no matter what route) is ordered that is either on their active medication list, was on the list in the preceding 24 hours, or that is being ordered simultaneously. A drug‐interaction database licensed from First DataBank (South San Francisco, CA) is utilized, and updated monthly, which classifies potential drug‐drug interactions as contraindicated, severe, intermediate, and mild. Those classified as contraindicated by First DataBank are included in the severe category in MEDITECH 5.64. During the study period, JHBMC's version of MEDITECH was configured so that providers were warned of potential severe and intermediate drug‐drug interactions, but not mild. No other customizations had been made. Patients' histories of allergies and other adverse responses to medications can be entered by any credentialed staff member. They are maintained together in an allergies section of the electronic medical record, but are identified as either allergy or adverse reactions at the time they are entered, and each generates its own warnings.

When more than 1 duplicate, interaction, allergy, or adverse reaction warning is generated for a particular medication, all appear listed on a single screen in identical fonts. No visual distinction is made between severe and intermediate drug‐drug interactions; for these, the category of medication ordered is followed by the category of the medication for which there is a potential interaction. A details button can be selected to learn specifically which medications are involved and the severity and nature of the potential interactions identified. In response to the warnings, providers can choose to either override them, erase the order, or replace the order by clicking 1 of 3 buttons at the bottom of the screen. Warnings are not repeated unless the medication is reordered for that patient. Dose warnings appear on a subsequent screen and are not addressed in this article.

Nurses are discouraged from entering verbal orders but do have the capacity to do so, at which time they encounter and must respond to the standard medication warnings, if any. Medical students are able to enter orders, at which time they also encounter and must respond to the standard medication warnings; their orders must then be cosigned by a licensed provider before they can be processed. Warnings encountered by nurses and medical students are not repeated at the time of cosignature by a licensed provider.

Data Collection

We collected data regarding all medication orders placed in our CPOE system from October 1, 2009 to April 20, 2010 for all adult patients. Intensive care unit (ICU) patients were excluded, in anticipation of a separate analysis. Hospitalizations under observation were also excluded. We then ran a report showing all medications that generated any number of warnings of any type (duplicate, interaction, allergy, or adverse reaction) for the same population. Warnings generated during readmissions that occurred at any point during the study period (ranging from 1 to 21 times) were excluded, because these patients likely had many, if not all, of the same medications ordered during their readmissions as during their initial hospitalization, which would unduly influence the analysis if retained.

There was wide variation in the number of warnings generated per medication and in the number of each warning type per medication that generated multiple warnings. Therefore, for ease of analysis and to ensure that we could accurately determine varying response to each individual warning type, we thereafter focused on the medications that generated single warnings during the study period. For each single warning we obtained patient name, account number, event date and time, hospital unit at the time of the event, ordered medication, ordering staff member, warning type, and staff member response to the warning (eg, override warning or erase order [accept the warning]). The response replace was used very infrequently, and therefore warnings that resulted in this response were excluded. Medications available in more than 1 form included the route of administration in their name, and from this they were categorized as parenteral or nonparenteral. All nonparenteral or parenteral forms of a given medication were grouped together as 1 medication (eg, morphine sustained release and morphine elixir were classified as a single‐medication, nonparenteral morphine). Medications were further categorized according to whether or not they were on the Institute for Safe Medication Practice (ISMP) List of High‐Alert Medications.[30]

The study was approved by the Johns Hopkins Institutional Review Board.

Analysis

We collected descriptive data about patients and providers. Age and length of stay (LOS) at the time of the event were determined based on the patients' admit date and date of birth, and grouped into quartiles. Hospital units were grouped according to which service or services they primarily served. Medications were grouped into quartiles according to the total number of warnings they generated during the study period. Warnings were dichotomously categorized according to whether they were overridden or accepted. Unpaired t tests were used to compare continuous variables for the 2 groups, and [2] tests were used to compare categorical variables. A multivariate logistic regression was then performed, using variables with a P value of <0.10 in the univariate analysis, to control for confounders and identify independent predictors of medication warning acceptance. All analyses were performed using Intercooled Stata 12 (StataCorp, College Station, TX).

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and Provider Features
 No. (%)
  • NOTE: Abbreviations: GYN, gynecology; IM, internal medicine; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; PA, physician assistant.

  • Hospital unit at the time of order entry.

  • Total is >100% due to rounding.

Patients (N=6,646) 
Age 
1545 years2,048 (31%)
4657 years1,610 (24%)
5872 years1,520 (23%)
73104 years1,468 (22%)
Gender 
Male2,934 (44%)
Hospital unita 
Medicine2,992 (45%)
Surgery1,836 (28%)
Neuro/psych/chem dep1,337 (20%)
OB/GYN481 (7%)
Caregivers (N=655) 
Resident248 (38%)b
Nurse154 (24%)
Attending or other97 (15%)
NP/PA69 (11%)
IM hospitalist31 (5%)
Fellow27 (4%)
Medical student23 (4%)
Pharmacist6 (1%)

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

Characteristics of Patients, Caregivers, Orders, Medications, and Warnings for Medication Orders Generating Single Warnings, and Association With Warning Acceptance
VariableNo. of Warnings (%)aNo. of Warnings Accepted (%)aP
  • NOTE: Abbreviations: GYN, gynecology; IM, internal medicine; ISMP, Institute for Safe Medication Practices; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; PA, physician assistant.

  • Totals may not equal 100% due to rounding.

  • Total number of medications is >454 because many medications generated more than 1 warning type.

Patient age   
1545 years10,881 (27)602 (5.5%)<0.001
4657 years9,733 (24)382 (3.9%) 
5872 years10,000 (25)308 (3.1%) 
73104 years9,777 (24)262 (2.7%) 
Patient gender   
Female23,395 (58)866 (3.7%)0.074
Male16,996 (42)688 (4.1%) 
Patient length of stay   
<1 day10,721 (27)660 (6.2%)<0.001
1 day10,854 (27)385 (3.5%) 
24 days10,424 (26)277 (2.7%) 
5118 days8,392 (21)232 (2.8%) 
Patient hospital unit   
Medicine20,057 (50)519 (2.6%)<0.001
Surgery10,274 (25)477 (4.6%) 
Neuro/psych/chem dep8,279 (21)417 (5.0%) 
OB/GYN1,781 (4)141 (7.9%) 
Ordering caregiver   
Resident22,523 (56)700 (3.1%)<0.001
NP/PA7,534 (19)369 (4.9%) 
IM hospitalist5,048 (13)155 (3.1%) 
Attending3225 (8)219 (6.8%) 
Fellow910 (2)34 (3.7%) 
Nurse865 (2)58 (6.7%) 
Medical student265 (<1)17 (6.4%) 
Pharmacist21 (<1)2 (9.5%) 
Day ordered   
Weekday31,499 (78%)1276 (4.1%)<0.001
Weekend8,892 (22%)278 (3.1%) 
Time ordered   
000005594,231 (11%)117 (2.8%)<0.001
0600115911,696 (29%)348 (3.0%) 
1200175915,879 (39%)722 (4.6%) 
180023598,585 (21%)367 (4.3%) 
Administration route (no. of meds)  
Nonparenteral (339)27,086 (67%)956 (3.5%)<0.001
Parenteral (115)13,305 (33%)598 (4.5%) 
ISMP List of High‐Alert Medications status (no. of meds)[30]  
Not on ISMP list (394)27,503 (68%)1251 (4.5%)<0.001
On ISMP list (60)12,888 (32%)303 (2.4%) 
No. of warnings per med (no. of meds)  
11062133 (7)9,869 (24%)191 (1.9%)<0.001
4681034 (13)10,014 (25%)331 (3.3%) 
170444 (40)10,182 (25%)314 (3.1%) 
1169 (394)10,326 (26%)718 (7.0%) 
Warning type (no. of meds)b  
Duplicate (369)19,083 (47%)1041 (5.5%)<0.001
Interaction (315)18,894 (47%)254 (1.3%) 
Allergy (138)2,371 (6%)243 (10.0%) 
Adverse reaction (14)43 (0.1%)16 (37%) 
Multivariate Analysis of Factors Associated With Acceptance of Medication Warnings
VariableAdjusted OR95% CI
  • NOTE: Abbreviations: CI, confidence interval; GYN, gynecology; IM, internal medicine; ISMP, Institute for Safe Medication Practices; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; OR, odds ratio; PA, physician assistant.

  • Day ordered and time of order entry were included but were not significant in the multivariate model.

Patient age  
1545 years1.00Reference
4657 years0.890.771.02
5872 years0.850.730.99
73104 years0.910.771.08
Patient gender  
Female1.00Reference
Male1.261.131.41
Patient length of stay 
<1 day1.00Reference
1 day0.650.550.76
24 days0.490.420.58
5118 days0.490.410.58
Patient hospital unit  
Medicine1.00Reference
Surgery1.451.251.68
Neuro/psych/chem dep1.351.151.58
OB/GYN2.431.923.08
Ordering caregiver  
Resident1.00Reference
NP/PA1.631.421.88
IM hospitalist1.241.021.50
Attending1.831.542.18
Fellow1.410.982.03
Nurse1.921.442.57
Medical student1.170.701.95
Pharmacist3.080.6714.03
Medication factors  
Nonparenteral1.00Reference
Parenteral1.791.592.03
HighAlert Medication status (no. of meds)[30]
Not on ISMP list1.00Reference
On ISMP list0.370.320.43
No. of warnings per medication 
110621331.00Reference
46810342.301.902.79
1704442.251.852.73
11694.103.424.92
Warning type  
Duplicate1.00Reference
Interaction0.240.210.28
Allergy2.281.942.68
Adverse reaction9.244.5218.90

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

Top 20 Medications Generating Single Warnings and Warning Type Distribution for Each
MedicationISMP ListbNo. of WarningsDuplicate, No. (%)cInteraction, No. (%)cAllergy, No. (%)cAdverse Reaction, No. (%)c
  • NOTE: Abbreviations: ISMP, Institute for Safe Medication Practices.

  • Medications not noted as injectable should be presumed not parenteral.

  • SMP List of High‐Alert Medications.[30]

  • Total may not add up to 100% due to rounding.

Hydromorphone injectableYes2,1331,584 (74.3)127 (6.0)422 (19.8) 
Metoprolol 1,432550 (38.4)870 (60.8)12 (0.8) 
Aspirin 1,375212 (15.4)1,096 (79.7)67 (4.9) 
OxycodoneYes1,360987 (72.6) 364 (26.8)9 (0.7)
Potassium chloride 1,296379 (29.2)917 (70.8)  
Ondansetron injectable 1,1671,013 (86.8)153 (13.1)1 (0.1) 
Aspart insulin injectableYes1,106643 (58.1)463 (41.9)  
WarfarinYes1,034298 (28.8)736 (71.2)  
Heparin injectableYes1,030205 (19.9)816 (79.2)9 (0.3) 
Furosemide injectable 980438 (45.0)542 (55.3)  
Lisinopril 926225 (24.3)698 (75.4)3 (0.3) 
Acetaminophen 860686 (79.8)118 (13.7)54 (6.3)2 (0.2)
Morphine injectableYes804467 (58.1)100 (12.4)233 (29.0)4 (0.5)
Diazepam 786731 (93.0)41 (5.2)14 (1.8) 
Glargine insulin injectableYes746268 (35.9)478 (64.1)  
Ibuprofen 713125 (17.5)529 (74.2)54 (7.6)5 (0.7)
HydromorphoneYes594372 (62.6)31 (5.2)187 (31.5)4 (0.7)
Furosemide 586273 (46.6)312 (53.2)1 (0.2) 
Ketorolac injectable 48739 (8.0)423 (86.9)23 (4.7)2 (0.4)
Prednisone 468166 (35.5)297 (63.5)5 (1.1) 

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Acknowledgements

The authors thank, in particular, Scott Carey, Research Informatics Manager, for assistance with data collection. Additional thanks go to Olga Sherman and Kathleen Ancinich for assistance with data collection and management.

Disclosures: This research was supported in part by the Johns Hopkins Institute for Clinical and Translational Research. All listed authors contributed substantially to the study conception and design, analysis and interpretation of data, drafting the article or revising it critically for important intellectual content, and final approval of the version to be published. No one who fulfills these criteria has been excluded from authorship. This research received no specific grant from any funding agency in the public, commercial, or not‐for‐profit sectors. The authors have no competing interests to declare.

Files
References
  1. Bates DW, Leape L, Cullen DJ, et al., Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:13111316.
  2. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized provider order entry on prescribing practices. Arch Intern Med. 2000;160:27412747.
  3. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinician decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:12231238.
  4. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23:451458.
  5. Eslami S, Keizer NF, Abu‐Hanna A. The impact of computerized physician medication order entry in hospitalized patients—a systematic review. Int J Med Inform. 2008;77:365376.
  6. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc. 2009;16:531538.
  7. Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO. Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc. 2009;16:613623.
  8. Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13:138147.
  9. Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc. 2008;15:620626.
  10. Magnus D, Rodger S, Avery AJ. GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther. 2002;27:377382.
  11. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians' assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med. 2009;169:16271632.
  12. Lapane KL, Waring ME, Schneider KL, Dube C, Quilliam BJ. A mixed method study of the merits of e‐prescribing drug alerts in primary care. J Gen Intern Med. 2008;23:442446.
  13. Bates DW. CPOE and clinical decision support in hospitals: getting the benefits: comment on “Unintended effects of a computerized physician order entry nearly hard‐stop alert to prevent a drug interaction.” Arch Intern Med. 2010;170:15831584.
  14. Classen DC, Phansalkar S, Bates DW. Critical drug‐drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf. 2011;7:6165.
  15. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood). 2011;30:23102317.
  16. Hines LE, Murphy JE, Grizzle AJ, Malone DC. Critical issues associated with drug‐drug interactions: highlights of a multistakeholder conference. Am J Health Syst Pharm. 2011;68:941946.
  17. Riedmann D, Jung M, Hackl WO, Stuhlinger W, der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak. 2011;11:35.
  18. Seidling HM, Phansalkar S, Seger DL, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011;18:479484.
  19. Riedmann D, Jung M, Hackl WO, Ammenwerth E. How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc. 2011;18:760766.
  20. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163:26252631.
  21. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006;13:511.
  22. Stutman HR, Fineman R, Meyer K, Jones D. Optimizing the acceptance of medication‐based alerts by physicians during CPOE implementation in a community hospital environment. AMIA Annu Symp Proc. 2007:701705.
  23. Long AJ, Chang P, Li YC, Chiu WT. The use of a CPOE log for the analysis of physicians' behavior when responding to drug‐duplication reminders. Int J Med Inform. 2008;77:499506.
  24. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med. 2009;169:305311.
  25. der Sijs H, Mulder A, Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf. 2009;18:941947.
  26. Seidling HM, Schmitt SP, Bruckner T, et al. Patient‐specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care. 2010;19:e15.
  27. Peberdy MA, Ornato JP, Larkin GL, et al. Survival from in‐hospital cardiac arrest during nights and weekends. JAMA. 2008;299:785792.
  28. Steinman MA, Hanlon JT. Managing medications in clinically complex elders: “There's got to be a happy medium.” JAMA. 2010;304:15921601.
  29. Agency for Healthcare Research and Quality. Safety culture. Available at: http://psnet.ahrq.gov/primer.aspx?primerID=5. Accessed October 29, 2013.
  30. Institute for Safe Medication Practice. List of High‐Alert Medications. Available at: http://www.ismp.org/Tools/highalertmedications.pdf. Accessed June 18, 2013.
  31. First Databank. FDB AlertSpace. Available at: http://www.fdbhealth.com/solutions/fdb‐alertspace. Accessed July 3, 2014.
  32. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp. 2000:26.
  33. Boussadi A, Caruba T, Zapletal E, Sabatier B, Durieux P, Degoulet P. A clinical data warehouse‐based process for refining medication orders alerts. J Am Med Inform Assoc. 2012;19:782785.
  34. Phansalkar S, der Sijs H, Tucker AD, et al. Drug‐drug interactions that should be non‐interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20:489493.
  35. Phansalkar S, Desai AA, Bell D, et al. High‐priority drug‐drug interactions for use in electronic health records. J Am Med Inform Assoc. 2012;19:735743.
  36. Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform. 2013;82:492503.
  37. Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of on‐demand versus computer‐triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15:430438.
  38. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug‐drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009;16:4046.
  39. Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc. 2010;17:493501.
  40. Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard‐stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170:15781583.
  41. Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E. Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co‐prescribing as a test case. J Am Med Inform Assoc. 2010;17:411415.
  42. Scott GP, Shah P, Wyatt JC, Makubate B, Cross FW. Making electronic prescribing alerts more effective: scenario‐based experimental study in junior doctors. J Am Med Inform Assoc. 2011;18:789798.
  43. Zachariah M, Phansalkar S, Seidling HM, et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human‐factors principles in medication‐related decision‐support systems—I‐MeDeSA. J Am Med Inform Assoc. 2011;18(suppl 1):i62i72.
  44. Kuperman GJ, Bobb A, Payne TH, et al. Medication‐related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14:2940.
  45. Jung M, Riedmann D, Hackl WO, et al. Physicians' perceptions on the usefulness of contextual information for prioritizing and presenting alerts in Computerized Physician Order Entry systems. BMC Med Inform Decis Mak. 2012;12:111.
  46. Hemens BJ, Holbrook A, Tonkin M, et al. Computerized clinical decision support systems for drug prescribing and management: a decision‐maker‐researcher partnership systematic review. Implement Sci. 2011;6:89.
  47. Duke JD, Bolchini D. A successful model and visual design for creating context‐aware drug‐drug interaction alerts. AMIA Annu Symp Proc. 2011;2011:339348.
Article PDF
Issue
Journal of Hospital Medicine - 10(1)
Publications
Page Number
19-25
Sections
Files
Files
Article PDF
Article PDF

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

Setting and Caregivers

Johns Hopkins Bayview Medical Center (JHBMC) is a 400‐bed academic medical center serving southeastern Baltimore, Maryland. Prescribing caregivers include residents and fellows who rotate to both JHBMC and Johns Hopkins Hospital, internal medicine hospitalists, other attending physicians (including teaching attendings for all departments, and hospitalists and clinical associates for departments other than internal medicine), and nurse practitioners and physician assistants from most JHBMC departments. Nearly 100% of patients on the surgery, obstetrics/gynecology, neurology, psychiatry, and chemical dependence services are hospitalized on units dedicated to their respective specialty, and the same is true for approximately 95% of medicine patients.

Order Entry

JHBMC began using a client‐server order entry system by MEDITECH (Westwood, MA) in July 2003. Provider order entry was phased in beginning in October 2003 and completed by the end of 2004. MEDITECH version 5.64 was being used during the study period. Medications may generate duplicate, interaction, allergy, adverse reaction, and dose warnings during a patient ordering session each time they are ordered. Duplicate warnings are generated when the same medication (no matter what route) is ordered that is either on their active medication list, was on the list in the preceding 24 hours, or that is being ordered simultaneously. A drug‐interaction database licensed from First DataBank (South San Francisco, CA) is utilized, and updated monthly, which classifies potential drug‐drug interactions as contraindicated, severe, intermediate, and mild. Those classified as contraindicated by First DataBank are included in the severe category in MEDITECH 5.64. During the study period, JHBMC's version of MEDITECH was configured so that providers were warned of potential severe and intermediate drug‐drug interactions, but not mild. No other customizations had been made. Patients' histories of allergies and other adverse responses to medications can be entered by any credentialed staff member. They are maintained together in an allergies section of the electronic medical record, but are identified as either allergy or adverse reactions at the time they are entered, and each generates its own warnings.

When more than 1 duplicate, interaction, allergy, or adverse reaction warning is generated for a particular medication, all appear listed on a single screen in identical fonts. No visual distinction is made between severe and intermediate drug‐drug interactions; for these, the category of medication ordered is followed by the category of the medication for which there is a potential interaction. A details button can be selected to learn specifically which medications are involved and the severity and nature of the potential interactions identified. In response to the warnings, providers can choose to either override them, erase the order, or replace the order by clicking 1 of 3 buttons at the bottom of the screen. Warnings are not repeated unless the medication is reordered for that patient. Dose warnings appear on a subsequent screen and are not addressed in this article.

Nurses are discouraged from entering verbal orders but do have the capacity to do so, at which time they encounter and must respond to the standard medication warnings, if any. Medical students are able to enter orders, at which time they also encounter and must respond to the standard medication warnings; their orders must then be cosigned by a licensed provider before they can be processed. Warnings encountered by nurses and medical students are not repeated at the time of cosignature by a licensed provider.

Data Collection

We collected data regarding all medication orders placed in our CPOE system from October 1, 2009 to April 20, 2010 for all adult patients. Intensive care unit (ICU) patients were excluded, in anticipation of a separate analysis. Hospitalizations under observation were also excluded. We then ran a report showing all medications that generated any number of warnings of any type (duplicate, interaction, allergy, or adverse reaction) for the same population. Warnings generated during readmissions that occurred at any point during the study period (ranging from 1 to 21 times) were excluded, because these patients likely had many, if not all, of the same medications ordered during their readmissions as during their initial hospitalization, which would unduly influence the analysis if retained.

There was wide variation in the number of warnings generated per medication and in the number of each warning type per medication that generated multiple warnings. Therefore, for ease of analysis and to ensure that we could accurately determine varying response to each individual warning type, we thereafter focused on the medications that generated single warnings during the study period. For each single warning we obtained patient name, account number, event date and time, hospital unit at the time of the event, ordered medication, ordering staff member, warning type, and staff member response to the warning (eg, override warning or erase order [accept the warning]). The response replace was used very infrequently, and therefore warnings that resulted in this response were excluded. Medications available in more than 1 form included the route of administration in their name, and from this they were categorized as parenteral or nonparenteral. All nonparenteral or parenteral forms of a given medication were grouped together as 1 medication (eg, morphine sustained release and morphine elixir were classified as a single‐medication, nonparenteral morphine). Medications were further categorized according to whether or not they were on the Institute for Safe Medication Practice (ISMP) List of High‐Alert Medications.[30]

The study was approved by the Johns Hopkins Institutional Review Board.

Analysis

We collected descriptive data about patients and providers. Age and length of stay (LOS) at the time of the event were determined based on the patients' admit date and date of birth, and grouped into quartiles. Hospital units were grouped according to which service or services they primarily served. Medications were grouped into quartiles according to the total number of warnings they generated during the study period. Warnings were dichotomously categorized according to whether they were overridden or accepted. Unpaired t tests were used to compare continuous variables for the 2 groups, and [2] tests were used to compare categorical variables. A multivariate logistic regression was then performed, using variables with a P value of <0.10 in the univariate analysis, to control for confounders and identify independent predictors of medication warning acceptance. All analyses were performed using Intercooled Stata 12 (StataCorp, College Station, TX).

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and Provider Features
 No. (%)
  • NOTE: Abbreviations: GYN, gynecology; IM, internal medicine; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; PA, physician assistant.

  • Hospital unit at the time of order entry.

  • Total is >100% due to rounding.

Patients (N=6,646) 
Age 
1545 years2,048 (31%)
4657 years1,610 (24%)
5872 years1,520 (23%)
73104 years1,468 (22%)
Gender 
Male2,934 (44%)
Hospital unita 
Medicine2,992 (45%)
Surgery1,836 (28%)
Neuro/psych/chem dep1,337 (20%)
OB/GYN481 (7%)
Caregivers (N=655) 
Resident248 (38%)b
Nurse154 (24%)
Attending or other97 (15%)
NP/PA69 (11%)
IM hospitalist31 (5%)
Fellow27 (4%)
Medical student23 (4%)
Pharmacist6 (1%)

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

Characteristics of Patients, Caregivers, Orders, Medications, and Warnings for Medication Orders Generating Single Warnings, and Association With Warning Acceptance
VariableNo. of Warnings (%)aNo. of Warnings Accepted (%)aP
  • NOTE: Abbreviations: GYN, gynecology; IM, internal medicine; ISMP, Institute for Safe Medication Practices; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; PA, physician assistant.

  • Totals may not equal 100% due to rounding.

  • Total number of medications is >454 because many medications generated more than 1 warning type.

Patient age   
1545 years10,881 (27)602 (5.5%)<0.001
4657 years9,733 (24)382 (3.9%) 
5872 years10,000 (25)308 (3.1%) 
73104 years9,777 (24)262 (2.7%) 
Patient gender   
Female23,395 (58)866 (3.7%)0.074
Male16,996 (42)688 (4.1%) 
Patient length of stay   
<1 day10,721 (27)660 (6.2%)<0.001
1 day10,854 (27)385 (3.5%) 
24 days10,424 (26)277 (2.7%) 
5118 days8,392 (21)232 (2.8%) 
Patient hospital unit   
Medicine20,057 (50)519 (2.6%)<0.001
Surgery10,274 (25)477 (4.6%) 
Neuro/psych/chem dep8,279 (21)417 (5.0%) 
OB/GYN1,781 (4)141 (7.9%) 
Ordering caregiver   
Resident22,523 (56)700 (3.1%)<0.001
NP/PA7,534 (19)369 (4.9%) 
IM hospitalist5,048 (13)155 (3.1%) 
Attending3225 (8)219 (6.8%) 
Fellow910 (2)34 (3.7%) 
Nurse865 (2)58 (6.7%) 
Medical student265 (<1)17 (6.4%) 
Pharmacist21 (<1)2 (9.5%) 
Day ordered   
Weekday31,499 (78%)1276 (4.1%)<0.001
Weekend8,892 (22%)278 (3.1%) 
Time ordered   
000005594,231 (11%)117 (2.8%)<0.001
0600115911,696 (29%)348 (3.0%) 
1200175915,879 (39%)722 (4.6%) 
180023598,585 (21%)367 (4.3%) 
Administration route (no. of meds)  
Nonparenteral (339)27,086 (67%)956 (3.5%)<0.001
Parenteral (115)13,305 (33%)598 (4.5%) 
ISMP List of High‐Alert Medications status (no. of meds)[30]  
Not on ISMP list (394)27,503 (68%)1251 (4.5%)<0.001
On ISMP list (60)12,888 (32%)303 (2.4%) 
No. of warnings per med (no. of meds)  
11062133 (7)9,869 (24%)191 (1.9%)<0.001
4681034 (13)10,014 (25%)331 (3.3%) 
170444 (40)10,182 (25%)314 (3.1%) 
1169 (394)10,326 (26%)718 (7.0%) 
Warning type (no. of meds)b  
Duplicate (369)19,083 (47%)1041 (5.5%)<0.001
Interaction (315)18,894 (47%)254 (1.3%) 
Allergy (138)2,371 (6%)243 (10.0%) 
Adverse reaction (14)43 (0.1%)16 (37%) 
Multivariate Analysis of Factors Associated With Acceptance of Medication Warnings
VariableAdjusted OR95% CI
  • NOTE: Abbreviations: CI, confidence interval; GYN, gynecology; IM, internal medicine; ISMP, Institute for Safe Medication Practices; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; OR, odds ratio; PA, physician assistant.

  • Day ordered and time of order entry were included but were not significant in the multivariate model.

Patient age  
1545 years1.00Reference
4657 years0.890.771.02
5872 years0.850.730.99
73104 years0.910.771.08
Patient gender  
Female1.00Reference
Male1.261.131.41
Patient length of stay 
<1 day1.00Reference
1 day0.650.550.76
24 days0.490.420.58
5118 days0.490.410.58
Patient hospital unit  
Medicine1.00Reference
Surgery1.451.251.68
Neuro/psych/chem dep1.351.151.58
OB/GYN2.431.923.08
Ordering caregiver  
Resident1.00Reference
NP/PA1.631.421.88
IM hospitalist1.241.021.50
Attending1.831.542.18
Fellow1.410.982.03
Nurse1.921.442.57
Medical student1.170.701.95
Pharmacist3.080.6714.03
Medication factors  
Nonparenteral1.00Reference
Parenteral1.791.592.03
HighAlert Medication status (no. of meds)[30]
Not on ISMP list1.00Reference
On ISMP list0.370.320.43
No. of warnings per medication 
110621331.00Reference
46810342.301.902.79
1704442.251.852.73
11694.103.424.92
Warning type  
Duplicate1.00Reference
Interaction0.240.210.28
Allergy2.281.942.68
Adverse reaction9.244.5218.90

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

Top 20 Medications Generating Single Warnings and Warning Type Distribution for Each
MedicationISMP ListbNo. of WarningsDuplicate, No. (%)cInteraction, No. (%)cAllergy, No. (%)cAdverse Reaction, No. (%)c
  • NOTE: Abbreviations: ISMP, Institute for Safe Medication Practices.

  • Medications not noted as injectable should be presumed not parenteral.

  • SMP List of High‐Alert Medications.[30]

  • Total may not add up to 100% due to rounding.

Hydromorphone injectableYes2,1331,584 (74.3)127 (6.0)422 (19.8) 
Metoprolol 1,432550 (38.4)870 (60.8)12 (0.8) 
Aspirin 1,375212 (15.4)1,096 (79.7)67 (4.9) 
OxycodoneYes1,360987 (72.6) 364 (26.8)9 (0.7)
Potassium chloride 1,296379 (29.2)917 (70.8)  
Ondansetron injectable 1,1671,013 (86.8)153 (13.1)1 (0.1) 
Aspart insulin injectableYes1,106643 (58.1)463 (41.9)  
WarfarinYes1,034298 (28.8)736 (71.2)  
Heparin injectableYes1,030205 (19.9)816 (79.2)9 (0.3) 
Furosemide injectable 980438 (45.0)542 (55.3)  
Lisinopril 926225 (24.3)698 (75.4)3 (0.3) 
Acetaminophen 860686 (79.8)118 (13.7)54 (6.3)2 (0.2)
Morphine injectableYes804467 (58.1)100 (12.4)233 (29.0)4 (0.5)
Diazepam 786731 (93.0)41 (5.2)14 (1.8) 
Glargine insulin injectableYes746268 (35.9)478 (64.1)  
Ibuprofen 713125 (17.5)529 (74.2)54 (7.6)5 (0.7)
HydromorphoneYes594372 (62.6)31 (5.2)187 (31.5)4 (0.7)
Furosemide 586273 (46.6)312 (53.2)1 (0.2) 
Ketorolac injectable 48739 (8.0)423 (86.9)23 (4.7)2 (0.4)
Prednisone 468166 (35.5)297 (63.5)5 (1.1) 

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Acknowledgements

The authors thank, in particular, Scott Carey, Research Informatics Manager, for assistance with data collection. Additional thanks go to Olga Sherman and Kathleen Ancinich for assistance with data collection and management.

Disclosures: This research was supported in part by the Johns Hopkins Institute for Clinical and Translational Research. All listed authors contributed substantially to the study conception and design, analysis and interpretation of data, drafting the article or revising it critically for important intellectual content, and final approval of the version to be published. No one who fulfills these criteria has been excluded from authorship. This research received no specific grant from any funding agency in the public, commercial, or not‐for‐profit sectors. The authors have no competing interests to declare.

Many computerized provider order entry (CPOE) systems suffer from having too much of a good thing. Few would question the beneficial effect of CPOE on medication order clarity, completeness, and transmission.[1, 2] When mechanisms for basic decision support have been added, however, such as allergy, interaction, and duplicate warnings, reductions in medication errors and adverse events have not been consistently achieved.[3, 4, 5, 6, 7] This is likely due in part to the fact that ordering providers override medication warnings at staggeringly high rates.[8, 9] Clinicians acknowledge that they are ignoring potentially valuable warnings,[10, 11] but suffer from alert fatigue due to the sheer number of messages, many of them judged by clinicians to be of low‐value.[11, 12]

Redesign of medication alert systems to increase their signal‐to‐noise ratio is badly needed,[13, 14, 15, 16] and will need to consider the clinical significance of alerts, their presentation, and context‐specific factors that potentially contribute to warning effectiveness.[17, 18, 19] Relatively few studies, however, have objectively looked at context factors such as the characteristics of providers, patients, medications, and warnings that are associated with provider responses to warnings,[9, 20, 21, 22, 23, 24, 25] and only 2 have studied how warning acceptance is associated with medication risk.[18, 26] We wished to explore these factors further. Warning acceptance has been shown to be higher, at least in the outpatient setting, when orders are entered by low‐volume prescribers for infrequently encountered warnings,[24] and there is some evidence that patients receive higher‐quality care during the day.[27] Significant attention has been placed in recent years on inappropriate prescribing in older patients,[28] and on creating a culture of safety in healthcare.[29] We therefore hypothesized that our providers would be more cautious, and medication warning acceptance rates would be higher, when orders were entered for patients who were older or with more complex medical problems, when they were entered during the day by caregivers who entered few orders, when the medications ordered were potentially associated with greater risk, and when the warnings themselves were infrequently encountered.

METHODS

Setting and Caregivers

Johns Hopkins Bayview Medical Center (JHBMC) is a 400‐bed academic medical center serving southeastern Baltimore, Maryland. Prescribing caregivers include residents and fellows who rotate to both JHBMC and Johns Hopkins Hospital, internal medicine hospitalists, other attending physicians (including teaching attendings for all departments, and hospitalists and clinical associates for departments other than internal medicine), and nurse practitioners and physician assistants from most JHBMC departments. Nearly 100% of patients on the surgery, obstetrics/gynecology, neurology, psychiatry, and chemical dependence services are hospitalized on units dedicated to their respective specialty, and the same is true for approximately 95% of medicine patients.

Order Entry

JHBMC began using a client‐server order entry system by MEDITECH (Westwood, MA) in July 2003. Provider order entry was phased in beginning in October 2003 and completed by the end of 2004. MEDITECH version 5.64 was being used during the study period. Medications may generate duplicate, interaction, allergy, adverse reaction, and dose warnings during a patient ordering session each time they are ordered. Duplicate warnings are generated when the same medication (no matter what route) is ordered that is either on their active medication list, was on the list in the preceding 24 hours, or that is being ordered simultaneously. A drug‐interaction database licensed from First DataBank (South San Francisco, CA) is utilized, and updated monthly, which classifies potential drug‐drug interactions as contraindicated, severe, intermediate, and mild. Those classified as contraindicated by First DataBank are included in the severe category in MEDITECH 5.64. During the study period, JHBMC's version of MEDITECH was configured so that providers were warned of potential severe and intermediate drug‐drug interactions, but not mild. No other customizations had been made. Patients' histories of allergies and other adverse responses to medications can be entered by any credentialed staff member. They are maintained together in an allergies section of the electronic medical record, but are identified as either allergy or adverse reactions at the time they are entered, and each generates its own warnings.

When more than 1 duplicate, interaction, allergy, or adverse reaction warning is generated for a particular medication, all appear listed on a single screen in identical fonts. No visual distinction is made between severe and intermediate drug‐drug interactions; for these, the category of medication ordered is followed by the category of the medication for which there is a potential interaction. A details button can be selected to learn specifically which medications are involved and the severity and nature of the potential interactions identified. In response to the warnings, providers can choose to either override them, erase the order, or replace the order by clicking 1 of 3 buttons at the bottom of the screen. Warnings are not repeated unless the medication is reordered for that patient. Dose warnings appear on a subsequent screen and are not addressed in this article.

Nurses are discouraged from entering verbal orders but do have the capacity to do so, at which time they encounter and must respond to the standard medication warnings, if any. Medical students are able to enter orders, at which time they also encounter and must respond to the standard medication warnings; their orders must then be cosigned by a licensed provider before they can be processed. Warnings encountered by nurses and medical students are not repeated at the time of cosignature by a licensed provider.

Data Collection

We collected data regarding all medication orders placed in our CPOE system from October 1, 2009 to April 20, 2010 for all adult patients. Intensive care unit (ICU) patients were excluded, in anticipation of a separate analysis. Hospitalizations under observation were also excluded. We then ran a report showing all medications that generated any number of warnings of any type (duplicate, interaction, allergy, or adverse reaction) for the same population. Warnings generated during readmissions that occurred at any point during the study period (ranging from 1 to 21 times) were excluded, because these patients likely had many, if not all, of the same medications ordered during their readmissions as during their initial hospitalization, which would unduly influence the analysis if retained.

There was wide variation in the number of warnings generated per medication and in the number of each warning type per medication that generated multiple warnings. Therefore, for ease of analysis and to ensure that we could accurately determine varying response to each individual warning type, we thereafter focused on the medications that generated single warnings during the study period. For each single warning we obtained patient name, account number, event date and time, hospital unit at the time of the event, ordered medication, ordering staff member, warning type, and staff member response to the warning (eg, override warning or erase order [accept the warning]). The response replace was used very infrequently, and therefore warnings that resulted in this response were excluded. Medications available in more than 1 form included the route of administration in their name, and from this they were categorized as parenteral or nonparenteral. All nonparenteral or parenteral forms of a given medication were grouped together as 1 medication (eg, morphine sustained release and morphine elixir were classified as a single‐medication, nonparenteral morphine). Medications were further categorized according to whether or not they were on the Institute for Safe Medication Practice (ISMP) List of High‐Alert Medications.[30]

The study was approved by the Johns Hopkins Institutional Review Board.

Analysis

We collected descriptive data about patients and providers. Age and length of stay (LOS) at the time of the event were determined based on the patients' admit date and date of birth, and grouped into quartiles. Hospital units were grouped according to which service or services they primarily served. Medications were grouped into quartiles according to the total number of warnings they generated during the study period. Warnings were dichotomously categorized according to whether they were overridden or accepted. Unpaired t tests were used to compare continuous variables for the 2 groups, and [2] tests were used to compare categorical variables. A multivariate logistic regression was then performed, using variables with a P value of <0.10 in the univariate analysis, to control for confounders and identify independent predictors of medication warning acceptance. All analyses were performed using Intercooled Stata 12 (StataCorp, College Station, TX).

RESULTS

A total of 259,656 medication orders were placed for adult non‐ICU patients during the 7‐month study period. Of those orders, 45,835 generated some number of medication warnings.[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] The median number of warnings per patient was 4 (interquartile range [IQR]=28; mean=5.9, standard deviation [SD]=6.2), with a range from 1 to 84. The median number of warnings generated per provider during the study period was 36 (IQR=6106, mean=87.4, SD=133.7), with a range of 1 to 1096.

There were 40,391 orders placed for 454 medications for adult non‐ICU patients, which generated a single‐medication warning (excluding those with the response replace, which was used 20 times) during the 7‐month study period. Data regarding the patients and providers associated with the orders generating single warnings are shown in Table 1. Most patients were on medicine units, and most orders were entered by residents. Patients' LOS at the time the orders were placed ranged from 0 to 118 days (median=1, IQR=04; mean=4.0, SD=7.2). The median number of single warnings per patient was 4 (IQR=28; mean=6.1, SD=6.5), with a range from 1 to 84. The median number of single warnings generated per provider during the study period was 15 (IQR=373; mean=61.7, SD=109.6), with a range of 1 to 1057.

Patient and Provider Features
 No. (%)
  • NOTE: Abbreviations: GYN, gynecology; IM, internal medicine; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; PA, physician assistant.

  • Hospital unit at the time of order entry.

  • Total is >100% due to rounding.

Patients (N=6,646) 
Age 
1545 years2,048 (31%)
4657 years1,610 (24%)
5872 years1,520 (23%)
73104 years1,468 (22%)
Gender 
Male2,934 (44%)
Hospital unita 
Medicine2,992 (45%)
Surgery1,836 (28%)
Neuro/psych/chem dep1,337 (20%)
OB/GYN481 (7%)
Caregivers (N=655) 
Resident248 (38%)b
Nurse154 (24%)
Attending or other97 (15%)
NP/PA69 (11%)
IM hospitalist31 (5%)
Fellow27 (4%)
Medical student23 (4%)
Pharmacist6 (1%)

Patient and caregiver characteristics for the medication orders that generated single warnings are shown in Table 2. The majority of medications were nonparenteral and not on the ISMP list (Table 3). Most warnings generated were either duplicate (47%) or interaction warnings (47%). Warnings of a particular type were repeated 14.5% of the time for a particular medication and patient (from 2 to 24 times, median=2, IQR=22, mean=2.7, SD=1.4), and 9.8% of the time for a particular caregiver, medication, and patient (from 2 to 18 times, median=2, IQR=22, mean=2.4, SD=1.1).

Characteristics of Patients, Caregivers, Orders, Medications, and Warnings for Medication Orders Generating Single Warnings, and Association With Warning Acceptance
VariableNo. of Warnings (%)aNo. of Warnings Accepted (%)aP
  • NOTE: Abbreviations: GYN, gynecology; IM, internal medicine; ISMP, Institute for Safe Medication Practices; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; PA, physician assistant.

  • Totals may not equal 100% due to rounding.

  • Total number of medications is >454 because many medications generated more than 1 warning type.

Patient age   
1545 years10,881 (27)602 (5.5%)<0.001
4657 years9,733 (24)382 (3.9%) 
5872 years10,000 (25)308 (3.1%) 
73104 years9,777 (24)262 (2.7%) 
Patient gender   
Female23,395 (58)866 (3.7%)0.074
Male16,996 (42)688 (4.1%) 
Patient length of stay   
<1 day10,721 (27)660 (6.2%)<0.001
1 day10,854 (27)385 (3.5%) 
24 days10,424 (26)277 (2.7%) 
5118 days8,392 (21)232 (2.8%) 
Patient hospital unit   
Medicine20,057 (50)519 (2.6%)<0.001
Surgery10,274 (25)477 (4.6%) 
Neuro/psych/chem dep8,279 (21)417 (5.0%) 
OB/GYN1,781 (4)141 (7.9%) 
Ordering caregiver   
Resident22,523 (56)700 (3.1%)<0.001
NP/PA7,534 (19)369 (4.9%) 
IM hospitalist5,048 (13)155 (3.1%) 
Attending3225 (8)219 (6.8%) 
Fellow910 (2)34 (3.7%) 
Nurse865 (2)58 (6.7%) 
Medical student265 (<1)17 (6.4%) 
Pharmacist21 (<1)2 (9.5%) 
Day ordered   
Weekday31,499 (78%)1276 (4.1%)<0.001
Weekend8,892 (22%)278 (3.1%) 
Time ordered   
000005594,231 (11%)117 (2.8%)<0.001
0600115911,696 (29%)348 (3.0%) 
1200175915,879 (39%)722 (4.6%) 
180023598,585 (21%)367 (4.3%) 
Administration route (no. of meds)  
Nonparenteral (339)27,086 (67%)956 (3.5%)<0.001
Parenteral (115)13,305 (33%)598 (4.5%) 
ISMP List of High‐Alert Medications status (no. of meds)[30]  
Not on ISMP list (394)27,503 (68%)1251 (4.5%)<0.001
On ISMP list (60)12,888 (32%)303 (2.4%) 
No. of warnings per med (no. of meds)  
11062133 (7)9,869 (24%)191 (1.9%)<0.001
4681034 (13)10,014 (25%)331 (3.3%) 
170444 (40)10,182 (25%)314 (3.1%) 
1169 (394)10,326 (26%)718 (7.0%) 
Warning type (no. of meds)b  
Duplicate (369)19,083 (47%)1041 (5.5%)<0.001
Interaction (315)18,894 (47%)254 (1.3%) 
Allergy (138)2,371 (6%)243 (10.0%) 
Adverse reaction (14)43 (0.1%)16 (37%) 
Multivariate Analysis of Factors Associated With Acceptance of Medication Warnings
VariableAdjusted OR95% CI
  • NOTE: Abbreviations: CI, confidence interval; GYN, gynecology; IM, internal medicine; ISMP, Institute for Safe Medication Practices; Neuro/psych/chem dep, neurology/psychiatry/chemical dependence; NP, nurse practitioner; OB, obstetrics; OR, odds ratio; PA, physician assistant.

  • Day ordered and time of order entry were included but were not significant in the multivariate model.

Patient age  
1545 years1.00Reference
4657 years0.890.771.02
5872 years0.850.730.99
73104 years0.910.771.08
Patient gender  
Female1.00Reference
Male1.261.131.41
Patient length of stay 
<1 day1.00Reference
1 day0.650.550.76
24 days0.490.420.58
5118 days0.490.410.58
Patient hospital unit  
Medicine1.00Reference
Surgery1.451.251.68
Neuro/psych/chem dep1.351.151.58
OB/GYN2.431.923.08
Ordering caregiver  
Resident1.00Reference
NP/PA1.631.421.88
IM hospitalist1.241.021.50
Attending1.831.542.18
Fellow1.410.982.03
Nurse1.921.442.57
Medical student1.170.701.95
Pharmacist3.080.6714.03
Medication factors  
Nonparenteral1.00Reference
Parenteral1.791.592.03
HighAlert Medication status (no. of meds)[30]
Not on ISMP list1.00Reference
On ISMP list0.370.320.43
No. of warnings per medication 
110621331.00Reference
46810342.301.902.79
1704442.251.852.73
11694.103.424.92
Warning type  
Duplicate1.00Reference
Interaction0.240.210.28
Allergy2.281.942.68
Adverse reaction9.244.5218.90

One thousand five hundred fifty‐four warnings were erased (ie, accepted by clinicians [4%]). In univariate analysis, only patient gender was not associated with warning acceptance. Patient age, LOS, hospital unit at the time of order entry, ordering caregiver type, day and time the medication was ordered, administration route, presence on the ISMP list, warning frequency, and warning type were all significantly associated with warning acceptance (Table 2).

Older patient age, longer LOS, presence of the medication on the ISMP list, and interaction warning type were all negatively associated with warning acceptance in multivariable analysis. Warning acceptance was positively associated with male patient gender, being on a service other than medicine, being a caregiver other than a resident, parenteral medications, lower warning frequency, and allergy or adverse reaction warning types (Table 3).

The 20 medications that generated the most single warnings are shown in Table 4. Medications on the ISMP list accounted for 8 of these top 20 medications. For most of them, duplicate and interaction warnings accounted for most of the warnings generated, except for parenteral hydromorphone, oral oxycodone, parenteral morphine, and oral hydromorphone, which each had more allergy than interaction warnings.

Top 20 Medications Generating Single Warnings and Warning Type Distribution for Each
MedicationISMP ListbNo. of WarningsDuplicate, No. (%)cInteraction, No. (%)cAllergy, No. (%)cAdverse Reaction, No. (%)c
  • NOTE: Abbreviations: ISMP, Institute for Safe Medication Practices.

  • Medications not noted as injectable should be presumed not parenteral.

  • SMP List of High‐Alert Medications.[30]

  • Total may not add up to 100% due to rounding.

Hydromorphone injectableYes2,1331,584 (74.3)127 (6.0)422 (19.8) 
Metoprolol 1,432550 (38.4)870 (60.8)12 (0.8) 
Aspirin 1,375212 (15.4)1,096 (79.7)67 (4.9) 
OxycodoneYes1,360987 (72.6) 364 (26.8)9 (0.7)
Potassium chloride 1,296379 (29.2)917 (70.8)  
Ondansetron injectable 1,1671,013 (86.8)153 (13.1)1 (0.1) 
Aspart insulin injectableYes1,106643 (58.1)463 (41.9)  
WarfarinYes1,034298 (28.8)736 (71.2)  
Heparin injectableYes1,030205 (19.9)816 (79.2)9 (0.3) 
Furosemide injectable 980438 (45.0)542 (55.3)  
Lisinopril 926225 (24.3)698 (75.4)3 (0.3) 
Acetaminophen 860686 (79.8)118 (13.7)54 (6.3)2 (0.2)
Morphine injectableYes804467 (58.1)100 (12.4)233 (29.0)4 (0.5)
Diazepam 786731 (93.0)41 (5.2)14 (1.8) 
Glargine insulin injectableYes746268 (35.9)478 (64.1)  
Ibuprofen 713125 (17.5)529 (74.2)54 (7.6)5 (0.7)
HydromorphoneYes594372 (62.6)31 (5.2)187 (31.5)4 (0.7)
Furosemide 586273 (46.6)312 (53.2)1 (0.2) 
Ketorolac injectable 48739 (8.0)423 (86.9)23 (4.7)2 (0.4)
Prednisone 468166 (35.5)297 (63.5)5 (1.1) 

DISCUSSION

Medication warnings in our study were frequently overridden, particularly when encountered by residents, for patients with a long LOS and on the internal medicine service, and for medications generating the most warnings and on the ISMP list. Disturbingly, this means that potentially important warnings for medications with the highest potential for causing harm, for possibly the sickest and most complex patients, were those that were most often ignored by young physicians in training who should have had the most to gain from them. Of course, this is not entirely surprising. Despite our hope that a culture of safety would influence young physicians' actions when caring for these patients and prescribing these medications, these patients and medications are those for whom the most warnings are generated, and these physicians are the ones entering the most orders. Only 13% of the medications studied were on the ISMP list, but they generated 32% of the warnings. We controlled for number of warnings and ISMP list status, but not for warning validity. Most likely, high‐risk medications have been set up with more warnings, many of them of lower quality, in an errant but well‐intentioned effort to make them safer. If developers of CPOE systems want to gain serious traction in using decision support to promote prescribing safe medications, they must take substantial action to increase attention to important warnings and decrease the number of clinically insignificant, low‐value warnings encountered by active caregivers on a daily basis.

Only 2 prior studies, both by Seidling et al., have specifically looked at provider response to warnings for high risk medications. Interaction warnings were rarely accepted in 1,[18] as in our study; however, in contrast to our findings, warning acceptance in both studies was higher for drugs with dose‐dependent toxicity.[18, 26] The effect of physician experience on warning acceptance has been addressed in 2 prior studies. In Weingart et al., residents were more likely than staff physicians to erase medication orders when presented with allergy and interaction warnings in a primary care setting.[20] Long et al. found that physicians younger than 40 years were less likely than older physicians to accept duplicate warnings, but those who had been at the study hospital for a longer period of time were more likely to accept them.[23] The influence of patient LOS and service on warning acceptance has not previously been described. Further study is needed looking at each of these factors.

Individual hospitals tend to avoid making modifications to order entry warning systems, because monitoring and maintaining these changes is labor intensive. Some institutions may make the decision to turn off certain categories of alerts, such as intermediate interaction warnings, to minimize the noise their providers encounter. There are even tools for disabling individual alerts or groups of alerts, such as that available for purchase from our interaction database vendor.[31] However, institutions may fear litigation should an adverse event be attributed to a disabled warning.[15, 16] Clearly, a comprehensive, health system‐wide approach is warranted.[13, 15] To date, published efforts describing ways to improve the effectiveness of medication warning systems have focused on either heightening the clinical significance of alerts[14, 21, 22, 32, 33, 34, 35, 36] or altering their presentation and how providers experience them.[21, 36, 37, 38, 39, 40, 41, 42, 43] The single medication warnings our providers receive are all presented in an identical font, and presumably response to each would be different if they were better distinguished from each other. We also found that a small but significant number of warnings were repeated for a given patient and even a given provider. If the providers knew they would only be presented with warnings the first time they occurred for a given patient and medication, they might be more attuned to the remaining warnings. Previous studies describe context‐specific decision support for medication ordering[44, 45, 46]; however, only 1 has described the use of patient context factors to modify when or how warnings are presented to providers.[47] None have described tailoring allergy, duplicate, and interaction warnings according to medication or provider types. If further study confirms our findings, modulating basic warning systems according to severity of illness, provider experience, and medication risk could powerfully increase their effectiveness. Of course, this would be extremely challenging to achieve, and is likely outside the capabilities of most, if not all, CPOE systems, at least for now.

Our study has some limitations. First, it was limited to medications that generated a single warning. We did this for ease of analysis and so that we could ensure understanding of provider response to each warning type without bias from simultaneously occurring warnings; however, caregiver response to multiple warnings appearing simultaneously for a particular medication order might be quite different. Second, we did not include any assessment of the number of medications ordered by each provider type or for each patient, either of which could significantly affect provider response to warnings. Third, as previously noted, we did not include any assessment of the validity of the warnings, beyond the 4 main categories described, which could also significantly affect provider response. However, it should be noted that although the validity of interaction warnings varies significantly from 1 medication to another, the validity of duplicate, allergy, and adverse reaction warnings in the described system are essentially the same for all medications. Fourth, it is possible that providers did modify or even erase their orders even after selecting override in response to the warning; it is also possible that providers reentered the same order after choosing erase. Unfortunately auditing for actions such as these would be extremely laborious. Finally, the study was conducted at a single medical center using a single order‐entry system. The system in use at our medical center is in use at one‐third of the 6000 hospitals in the United States, though certainly not all are using our version. Even if a hospital was using the same CPOE version and interaction database as our institution, variations in patient population and local decisions modifying how the database interacts with the warning presentation system might affect reproducibility at that institution.

Commonly encountered medication warnings are overridden at extremely high rates, and in our study this was particularly so for medications on the ISMP list, when ordered by physicians in training. Warnings of little clinical significance must be identified and eliminated, the most important warnings need to be visually distinct to increase user attention, and further research should be done into the patient, provider, setting, and medication factors that affect user responses to warnings, so that they may be customized accordingly and their significance increased. Doing so will enable us to reap the maximum possible potential from our CPOE systems, and increase the CPOE's power to protect our most vulnerable patients from our most dangerous medications, particularly when cared for by our most inexperienced physicians.

Acknowledgements

The authors thank, in particular, Scott Carey, Research Informatics Manager, for assistance with data collection. Additional thanks go to Olga Sherman and Kathleen Ancinich for assistance with data collection and management.

Disclosures: This research was supported in part by the Johns Hopkins Institute for Clinical and Translational Research. All listed authors contributed substantially to the study conception and design, analysis and interpretation of data, drafting the article or revising it critically for important intellectual content, and final approval of the version to be published. No one who fulfills these criteria has been excluded from authorship. This research received no specific grant from any funding agency in the public, commercial, or not‐for‐profit sectors. The authors have no competing interests to declare.

References
  1. Bates DW, Leape L, Cullen DJ, et al., Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:13111316.
  2. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized provider order entry on prescribing practices. Arch Intern Med. 2000;160:27412747.
  3. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinician decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:12231238.
  4. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23:451458.
  5. Eslami S, Keizer NF, Abu‐Hanna A. The impact of computerized physician medication order entry in hospitalized patients—a systematic review. Int J Med Inform. 2008;77:365376.
  6. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc. 2009;16:531538.
  7. Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO. Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc. 2009;16:613623.
  8. Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13:138147.
  9. Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc. 2008;15:620626.
  10. Magnus D, Rodger S, Avery AJ. GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther. 2002;27:377382.
  11. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians' assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med. 2009;169:16271632.
  12. Lapane KL, Waring ME, Schneider KL, Dube C, Quilliam BJ. A mixed method study of the merits of e‐prescribing drug alerts in primary care. J Gen Intern Med. 2008;23:442446.
  13. Bates DW. CPOE and clinical decision support in hospitals: getting the benefits: comment on “Unintended effects of a computerized physician order entry nearly hard‐stop alert to prevent a drug interaction.” Arch Intern Med. 2010;170:15831584.
  14. Classen DC, Phansalkar S, Bates DW. Critical drug‐drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf. 2011;7:6165.
  15. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood). 2011;30:23102317.
  16. Hines LE, Murphy JE, Grizzle AJ, Malone DC. Critical issues associated with drug‐drug interactions: highlights of a multistakeholder conference. Am J Health Syst Pharm. 2011;68:941946.
  17. Riedmann D, Jung M, Hackl WO, Stuhlinger W, der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak. 2011;11:35.
  18. Seidling HM, Phansalkar S, Seger DL, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011;18:479484.
  19. Riedmann D, Jung M, Hackl WO, Ammenwerth E. How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc. 2011;18:760766.
  20. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163:26252631.
  21. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006;13:511.
  22. Stutman HR, Fineman R, Meyer K, Jones D. Optimizing the acceptance of medication‐based alerts by physicians during CPOE implementation in a community hospital environment. AMIA Annu Symp Proc. 2007:701705.
  23. Long AJ, Chang P, Li YC, Chiu WT. The use of a CPOE log for the analysis of physicians' behavior when responding to drug‐duplication reminders. Int J Med Inform. 2008;77:499506.
  24. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med. 2009;169:305311.
  25. der Sijs H, Mulder A, Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf. 2009;18:941947.
  26. Seidling HM, Schmitt SP, Bruckner T, et al. Patient‐specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care. 2010;19:e15.
  27. Peberdy MA, Ornato JP, Larkin GL, et al. Survival from in‐hospital cardiac arrest during nights and weekends. JAMA. 2008;299:785792.
  28. Steinman MA, Hanlon JT. Managing medications in clinically complex elders: “There's got to be a happy medium.” JAMA. 2010;304:15921601.
  29. Agency for Healthcare Research and Quality. Safety culture. Available at: http://psnet.ahrq.gov/primer.aspx?primerID=5. Accessed October 29, 2013.
  30. Institute for Safe Medication Practice. List of High‐Alert Medications. Available at: http://www.ismp.org/Tools/highalertmedications.pdf. Accessed June 18, 2013.
  31. First Databank. FDB AlertSpace. Available at: http://www.fdbhealth.com/solutions/fdb‐alertspace. Accessed July 3, 2014.
  32. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp. 2000:26.
  33. Boussadi A, Caruba T, Zapletal E, Sabatier B, Durieux P, Degoulet P. A clinical data warehouse‐based process for refining medication orders alerts. J Am Med Inform Assoc. 2012;19:782785.
  34. Phansalkar S, der Sijs H, Tucker AD, et al. Drug‐drug interactions that should be non‐interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20:489493.
  35. Phansalkar S, Desai AA, Bell D, et al. High‐priority drug‐drug interactions for use in electronic health records. J Am Med Inform Assoc. 2012;19:735743.
  36. Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform. 2013;82:492503.
  37. Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of on‐demand versus computer‐triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15:430438.
  38. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug‐drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009;16:4046.
  39. Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc. 2010;17:493501.
  40. Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard‐stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170:15781583.
  41. Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E. Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co‐prescribing as a test case. J Am Med Inform Assoc. 2010;17:411415.
  42. Scott GP, Shah P, Wyatt JC, Makubate B, Cross FW. Making electronic prescribing alerts more effective: scenario‐based experimental study in junior doctors. J Am Med Inform Assoc. 2011;18:789798.
  43. Zachariah M, Phansalkar S, Seidling HM, et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human‐factors principles in medication‐related decision‐support systems—I‐MeDeSA. J Am Med Inform Assoc. 2011;18(suppl 1):i62i72.
  44. Kuperman GJ, Bobb A, Payne TH, et al. Medication‐related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14:2940.
  45. Jung M, Riedmann D, Hackl WO, et al. Physicians' perceptions on the usefulness of contextual information for prioritizing and presenting alerts in Computerized Physician Order Entry systems. BMC Med Inform Decis Mak. 2012;12:111.
  46. Hemens BJ, Holbrook A, Tonkin M, et al. Computerized clinical decision support systems for drug prescribing and management: a decision‐maker‐researcher partnership systematic review. Implement Sci. 2011;6:89.
  47. Duke JD, Bolchini D. A successful model and visual design for creating context‐aware drug‐drug interaction alerts. AMIA Annu Symp Proc. 2011;2011:339348.
References
  1. Bates DW, Leape L, Cullen DJ, et al., Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:13111316.
  2. Teich JM, Merchia PR, Schmiz JL, Kuperman GJ, Spurr CD, Bates DW. Effects of computerized provider order entry on prescribing practices. Arch Intern Med. 2000;160:27412747.
  3. Garg AX, Adhikari NKJ, McDonald H, et al. Effects of computerized clinician decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293:12231238.
  4. Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23:451458.
  5. Eslami S, Keizer NF, Abu‐Hanna A. The impact of computerized physician medication order entry in hospitalized patients—a systematic review. Int J Med Inform. 2008;77:365376.
  6. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc. 2009;16:531538.
  7. Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO. Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc. 2009;16:613623.
  8. Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13:138147.
  9. Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc. 2008;15:620626.
  10. Magnus D, Rodger S, Avery AJ. GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther. 2002;27:377382.
  11. Weingart SN, Simchowitz B, Shiman L, et al. Clinicians' assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med. 2009;169:16271632.
  12. Lapane KL, Waring ME, Schneider KL, Dube C, Quilliam BJ. A mixed method study of the merits of e‐prescribing drug alerts in primary care. J Gen Intern Med. 2008;23:442446.
  13. Bates DW. CPOE and clinical decision support in hospitals: getting the benefits: comment on “Unintended effects of a computerized physician order entry nearly hard‐stop alert to prevent a drug interaction.” Arch Intern Med. 2010;170:15831584.
  14. Classen DC, Phansalkar S, Bates DW. Critical drug‐drug interactions for use in electronic health records systems with computerized physician order entry: review of leading approaches. J Patient Saf. 2011;7:6165.
  15. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood). 2011;30:23102317.
  16. Hines LE, Murphy JE, Grizzle AJ, Malone DC. Critical issues associated with drug‐drug interactions: highlights of a multistakeholder conference. Am J Health Syst Pharm. 2011;68:941946.
  17. Riedmann D, Jung M, Hackl WO, Stuhlinger W, der Sijs H, Ammenwerth E. Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak. 2011;11:35.
  18. Seidling HM, Phansalkar S, Seger DL, et al. Factors influencing alert acceptance: a novel approach for predicting the success of clinical decision support. J Am Med Inform Assoc. 2011;18:479484.
  19. Riedmann D, Jung M, Hackl WO, Ammenwerth E. How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc. 2011;18:760766.
  20. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163:26252631.
  21. Shah NR, Seger AC, Seger DL, et al. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006;13:511.
  22. Stutman HR, Fineman R, Meyer K, Jones D. Optimizing the acceptance of medication‐based alerts by physicians during CPOE implementation in a community hospital environment. AMIA Annu Symp Proc. 2007:701705.
  23. Long AJ, Chang P, Li YC, Chiu WT. The use of a CPOE log for the analysis of physicians' behavior when responding to drug‐duplication reminders. Int J Med Inform. 2008;77:499506.
  24. Isaac T, Weissman JS, Davis RB, et al. Overrides of medication alerts in ambulatory care. Arch Intern Med. 2009;169:305311.
  25. der Sijs H, Mulder A, Gelder T, Aarts J, Berg M, Vulto A. Drug safety alert generation and overriding in a large Dutch university medical centre. Pharmacoepidemiol Drug Saf. 2009;18:941947.
  26. Seidling HM, Schmitt SP, Bruckner T, et al. Patient‐specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care. 2010;19:e15.
  27. Peberdy MA, Ornato JP, Larkin GL, et al. Survival from in‐hospital cardiac arrest during nights and weekends. JAMA. 2008;299:785792.
  28. Steinman MA, Hanlon JT. Managing medications in clinically complex elders: “There's got to be a happy medium.” JAMA. 2010;304:15921601.
  29. Agency for Healthcare Research and Quality. Safety culture. Available at: http://psnet.ahrq.gov/primer.aspx?primerID=5. Accessed October 29, 2013.
  30. Institute for Safe Medication Practice. List of High‐Alert Medications. Available at: http://www.ismp.org/Tools/highalertmedications.pdf. Accessed June 18, 2013.
  31. First Databank. FDB AlertSpace. Available at: http://www.fdbhealth.com/solutions/fdb‐alertspace. Accessed July 3, 2014.
  32. Abookire SA, Teich JM, Sandige H, et al. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp. 2000:26.
  33. Boussadi A, Caruba T, Zapletal E, Sabatier B, Durieux P, Degoulet P. A clinical data warehouse‐based process for refining medication orders alerts. J Am Med Inform Assoc. 2012;19:782785.
  34. Phansalkar S, der Sijs H, Tucker AD, et al. Drug‐drug interactions that should be non‐interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20:489493.
  35. Phansalkar S, Desai AA, Bell D, et al. High‐priority drug‐drug interactions for use in electronic health records. J Am Med Inform Assoc. 2012;19:735743.
  36. Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decision support interventions for medication prescribing. Int J Med Inform. 2013;82:492503.
  37. Tamblyn R, Huang A, Taylor L, et al. A randomized trial of the effectiveness of on‐demand versus computer‐triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15:430438.
  38. Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug‐drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009;16:4046.
  39. Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc. 2010;17:493501.
  40. Strom BL, Schinnar R, Aberra F, et al. Unintended effects of a computerized physician order entry nearly hard‐stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med. 2010;170:15781583.
  41. Strom BL, Schinnar R, Bilker W, Hennessy S, Leonard CE, Pifer E. Randomized clinical trial of a customized electronic alert requiring an affirmative response compared to a control group receiving a commercial passive CPOE alert: NSAID—warfarin co‐prescribing as a test case. J Am Med Inform Assoc. 2010;17:411415.
  42. Scott GP, Shah P, Wyatt JC, Makubate B, Cross FW. Making electronic prescribing alerts more effective: scenario‐based experimental study in junior doctors. J Am Med Inform Assoc. 2011;18:789798.
  43. Zachariah M, Phansalkar S, Seidling HM, et al. Development and preliminary evidence for the validity of an instrument assessing implementation of human‐factors principles in medication‐related decision‐support systems—I‐MeDeSA. J Am Med Inform Assoc. 2011;18(suppl 1):i62i72.
  44. Kuperman GJ, Bobb A, Payne TH, et al. Medication‐related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14:2940.
  45. Jung M, Riedmann D, Hackl WO, et al. Physicians' perceptions on the usefulness of contextual information for prioritizing and presenting alerts in Computerized Physician Order Entry systems. BMC Med Inform Decis Mak. 2012;12:111.
  46. Hemens BJ, Holbrook A, Tonkin M, et al. Computerized clinical decision support systems for drug prescribing and management: a decision‐maker‐researcher partnership systematic review. Implement Sci. 2011;6:89.
  47. Duke JD, Bolchini D. A successful model and visual design for creating context‐aware drug‐drug interaction alerts. AMIA Annu Symp Proc. 2011;2011:339348.
Issue
Journal of Hospital Medicine - 10(1)
Issue
Journal of Hospital Medicine - 10(1)
Page Number
19-25
Page Number
19-25
Publications
Publications
Article Type
Display Headline
Factors associated with medication warning acceptance for hospitalized adults
Display Headline
Factors associated with medication warning acceptance for hospitalized adults
Sections
Article Source

© 2015 Society of Hospital Medicine

Disallow All Ads
Correspondence Location
Address for correspondence and reprint requests: Amy M. Knight, MD, Division of Hospital Medicine, Johns Hopkins Bayview Medical Center, 5200 Eastern Ave., Mason F. Lord West Tower, 6th Floor, Baltimore, MD 21224; Telephone: 410‐550‐5018; Fax: 410‐550‐2972; E‐mail: aknight@jhmi.edu
Content Gating
No Gating (article Unlocked/Free)
Alternative CME
Article PDF Media
Media Files