Philip C. Lee, MD

Although the prevalence of coronary heart disease and death from acute myocardial infarction (AMI) have declined steadily, about 935,000 heart attacks still occur annually in the United States, with approximately one‐third of these being fatal.[1, 2, 3] Studies have demonstrated decreased 30‐day and longer‐term mortality in AMI patients who receive evidence‐based treatment, including aspirin, ‐blockers, angiotensin‐converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs), anticoagulation therapy, and statins.[4, 5, 6, 7] Despite clinical practice guidelines (CPGs) outlining evidence‐based care and considerable efforts to implement processes that improve patient outcomes, delivery of effective therapy remains suboptimal.[8] For example, the Hospital Quality Alliance Program[9] found that in AMI patients, use of aspirin on admission was only 81% to 92%, ‐blocker on admission 75% to 85%, and ACE inhibitors for left ventricular dysfunction 71% to 74%.

Efforts to increase adherence to CPGs and improve patient outcomes in AMI have resulted in variable degrees of success. They include promotion of CPGs,[4, 5, 6, 7] physician education with feedback, report cards, care paths, registries,[10] Joint Commission standardized measures,[11] and paper checklists or order sets (OS).[12, 13]

In this report, we describe the association between use of an evidence‐based, electronic OS for AMI (AMI‐OS) and better adherence to CPGs. This AMI‐OS was implemented in the inpatient electronic medical records (EMRs) of a large integrated healthcare delivery system, Kaiser Permanente Northern California (KPNC). The purpose of our investigation was to determine (1) whether use of the AMI‐OS was associated with improved AMI processes and patient outcomes, and (2) whether these associations persisted after risk adjustment using a comprehensive severity of illness scoring system.

MATERIALS AND METHODS

This project was approved by the KPNC institutional review board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group, Inc., care for 3.4 million Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All KPNC facilities employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere.[14] Our setting consisted of 21 KPNC hospitals described in previous reports,[15, 16, 17, 18] using the same commercially available EMR system that includes computerized physician order entry (CPOE). Deployment of the customized inpatient Epic EMR (www.epicsystems.com), known internally as KP HealthConnect (KPHC), began in 2006 and was completed in 2010.

In this EMR's CPOE, physicians have options to select individual orders (a la carte) or they can utilize an OS, which is a collection of the most appropriate orders associated with specific diagnoses, procedures, or treatments. The evidence‐based AMI‐OS studied in this project was developed by a multidisciplinary team (for detailed components see Supporting Appendix 1Appendix 5 in the online version of this article).

Our study focused on the first set of hospital admission orders for patients with AMI. The study sample consisted of patients meeting these criteria: (1) age 18 years at admission; (2) admitted to a KPNC hospital for an overnight stay between September 28, 2008 and December 31, 2010; (3) principal diagnosis was AMI (International Classification of Diseases, 9th Revision [ICD‐9][19] codes 410.00, 01, 10, 11, 20, 21, 30, 31, 40, 41, 50, 51, 60, 61, 70, 71, 80, 90, and 91); and (4) KPHC had been operational at the hospital for at least 3 months to be included (for assembly descriptions see Supporting Appendices 15 in the online version of this article). At the study hospitals, troponin I was measured using the Beckman Access AccuTnI assay (Beckman Coulter, Inc., Brea, CA), whose upper reference limit (99th percentile) is 0.04 ng/mL. We excluded patients initially hospitalized for AMI at a non‐KPNC site and transferred into a study hospital.

The data processing methods we employed have been detailed elsewhere.[14, 15, 17, 20, 21, 22] The dependent outcome variables were total hospital length of stay, inpatient mortality, 30‐day mortality, and all‐cause rehospitalization within 30 days of discharge. Linked state mortality data were unavailable for the entire study period, so we ascertained 30‐day mortality based on the combination of KPNC patient demographic data and publicly available Social Security Administration decedent files. We ascertained rehospitalization by scanning KPNC hospitalization databases, which also track out‐of‐plan use.

The dependent process variables were use of aspirin within 24 hours of admission, ‐blockers, anticoagulation, ACE inhibitors or ARBs, and statins. The primary independent variable of interest was whether or not the admitting physician employed the AMI‐OS when admission orders were entered. Consequently, this variable is dichotomous (AMI‐OS vs a la carte).

We controlled for acute illness severity and chronic illness burden using a recent modification[22] of an externally validated risk‐adjustment system applicable to all hospitalized patients.[15, 16, 23, 24, 25] Our methodology included vital signs, neurological status checks, and laboratory test results obtained in the 72 hours preceding hospital admission; comorbidities were captured longitudinally using data from the year preceding hospitalization (for comparison purposes, we also assigned a Charlson Comorbidity Index score[26]).

End‐of‐life care directives are mandatory on admission at KPNC hospitals. Physicians have 4 options: full code, partial code, do not resuscitate, and comfort care only. Because of small numbers in some categories, we collapsed these 4 categories into full code and not full code. Because patients' care directives may change, we elected to capture the care directive in effect when a patient first entered a hospital unit other than the emergency department (ED).

Two authors (M.B., P.C.L.), one of whom is a board‐certified cardiologist, reviewed all admission electrocardiograms and made a consensus determination as to whether or not criteria for ST‐segment elevation myocardial infarction (STEMI) were present (ie, new ST‐segment elevation or left bundle branch block); we also reviewed the records of all patients with missing troponin I data to confirm the AMI diagnosis.

RESULTS

Table 1 summarizes the characteristics of the 5879 patients. It shows that AMI‐OS patients were more likely to receive evidence‐based therapies for AMI (aspirin, ‐blockers, ACE inhibitors or ARBs, anticoagulation, and statins) and had a 46% lower mortality rate in hospital (3.51 % vs 6.52%) and 33% lower rate at 30 days (5.66% vs 8.48%). AMI‐OS patients were also found to be at lower risk for an adverse outcome than nonAMI‐OS patients. The AMI‐OS patients had lower peak troponin I values, severity of illness (lower Laboratory‐Based Acute Physiology Score, version 2 [LAPS2] scores), comorbidity burdens (lower Comorbidity Point Score, version 2 [COPS2] and Charlson scores), and global predicted mortality risk. AMI‐OS patients were also less likely to have required intensive care. AMI‐OS patients were at higher risk of death than nonAMI‐OS patients with respect to only 1 variable (being full code at the time of admission), but although this difference was statistically significant, it was of minor clinical impact (86% vs 88%).

Table 2 shows the result of a logistic regression model in which the dependent variable was inpatient mortality and either the 5 evidence‐based therapies or the total number of evidence‐based therapies are the dependent variables. ‐blocker, statin, and ACE inhibitor or ARB therapies all had a protective effect on mortality, with odds ratios ranging from 0.48 (95% confidence interval [CI]: 0.36‐0.64), 0.63 (95% CI: 0.45‐0.89), and 0.40 (95% CI: 0.30‐0.53), respectively. An increased number of therapies also had a beneficial effect on inpatient mortality, with patients having 3 or more of the evidence‐based therapies showing an adjusted odds ratio (AOR) of 0.49 (95% CI: 0.33‐0.73), 4 or more therapies an AOR of 0.29 (95% CI: 0.20‐0.42), and 0.17 (95% CI: 0.11‐0.25) for 5 or more therapies.

Table 3 shows that the use of the AMI‐OS is protective, with an AOR of 0.59 and a 95% CI of 0.45‐0.76. Table 3 also shows that the most potent predictors were comorbidity burden (AOR: 1.07, 95% CI: 1.03‐1.10 per 10 COPS2 points), severity of illness (AOR: 1.09, 95% CI: 1.07‐1.12 per 10 LAPS2 points), STEMI (AOR: 3.86, 95% CI: 2.68‐5.58), and timing of cardiac catheterization referral occurring immediately prior to or during the admission (AOR: 0.37, 95% CI: 0.27‐0.51). The statistical significance of the AMI‐OS effect disappears when both AMI‐OS and the individual therapies are included in the same model (see Supporting Information, Appendices 15, in the online version of this article).

Table 4 shows separately the average treatment effect (ATE) and average treatment effect for the treated (ATT) of AMI‐OS and of increasing number of therapies on other outcomes (30‐day mortality, LOS, and readmission). Both the ATE and ATT show that the use of the AMI‐OS was significantly protective with respect to mortality and total hospital LOS but not significant with respect to readmission. The effect of the number of therapies on mortality is significantly higher with increasing number of therapies. For example, patients who received 5 therapies had an average treatment effect on 30‐day inpatient mortality of 0.23 (95% CI: 0.15‐0.35) compared to 0.64 (95% CI: 0.43‐0.96) for 3 therapies, almost a 3‐fold difference. The effects of increasing number of therapies were not significant for LOS or readmission. A sensitivity analysis in which the 535 STEMI patients were removed showed essentially the same results, so it is not reported here.

To further elucidate possible reasons why physicians did not use the AMI‐OS, the lead author reviewed 105 randomly selected records where the AMI‐OS was not used, 5 records from each of the 21 study hospitals. This review found that in 36% of patients, the AMI‐OS was not used because emergent catheterization or transfer to a facility with percutaneous coronary intervention capability occurred. Presence of other significant medical conditions, including critical illness, was the reason in 17% of these cases, patient or family refusal of treatments in 8%, issues around end‐of‐life care in 3%, and specific medical contraindications in 1%. In the remaining 34%, no reason for not using the AMI‐OS could be identified.

DISCUSSION

We evaluated the use of an evidence‐based electronic AMI‐OS embedded in a comprehensive EMR and found that it was beneficial. Its use was associated with increased adherence to evidence‐based therapies, which in turn were associated with improved outcomes. Using data from a large cohort of hospitalized AMI patients in 21 community hospitals, we were able to use risk adjustment that included physiologic illness severity to adjust for baseline mortality risk. Patients in whom the AMI‐OS was employed tended to be at lower risk; nonetheless, after controlling for confounding variables and adjusting for bias using propensity scores, the AMI‐OS was associated with increased use of evidence‐based therapies and decreased mortality. Most importantly, it appears that the benefits of the OS were not just due to increased receipt of individual recommended therapies, but to increased concurrent receipt of multiple recommended therapies.

Modern EMRs have great potential for significant improvements in the quality, efficiency, and safety of care provided,[36] and our study highlights this potential. However, a number of important limitations to our study must be considered. Although we had access to a very rich dataset, we could not control for all possible confounders, and our risk adjustment cannot match the level of information available to clinicians. In particular, the measurements available to us with respect to cardiac risk are limited. Thus, we have to recognize that the strength of our findings does not approximate that of a randomized trial, and one would expect that the magnitude of the beneficial association would fall under more controlled conditions. Resource limitations also did not permit us to gather more time course data (eg, sequential measurements of patient instability, cardiac damage, or use of recommended therapies), which could provide a better delineation of differences in both processes and outcomes.

Limitations also exist to the generalizability of the use of order sets in other settings that go beyond the availability of a comprehensive EMR. Our study population was cared for in a setting with an unusually high level of integration.[1] For example, KPNC has an elaborate administrative infrastructure for training in the use of the EMR as well as ensuring that order sets are not just evidence‐based, but that they are perceived by clinicians to be of significant value. This infrastructure, established to ensure physician buy‐in, may not be easy to replicate in smaller or less‐integrated settings. Thus, it is conceivable that factors other than the degree of support during the EMR deployments can affect rates of order set use.

Although our use of counterfactual methods included illness severity (LAPS2) and longitudinal comorbidity burden (COPS2), which are not yet available outside highly integrated delivery services employing comprehensive EMRs, it is possible they are insufficient. We cannot exclude the possibility that other biases or patient characteristics were present that led clinicians to preferentially employ the electronic order set in some patients but not in others. One could also argue that future studies should consider using overall adherence to recommended AMI treatment guidelines as a risk adjustment tool that would permit one to analyze what other factors may be playing a role in residual differences in patient outcomes. Last, one could object to our inclusion of STEMI patients; however, this was not a study on optimum treatment strategies for STEMI patients. Rather, it was a study on the impact on AMI outcomes of a specific component of computerized order entry outside the research setting.

Despite these limitations, we believe that our findings provide strong support for the continued use of electronic evidence‐based order sets in the inpatient medical setting. Once the initial implementation of a comprehensive EMR has occurred, deployment of these electronic order sets is a relatively inexpensive but effective method to foster compliance with evidence‐based care.

Future research in healthcare information technology can take a number of directions. One important area, of course, revolves around ways to promote enhanced physician adoption of EMRs. Our audit of records where the AMI‐OS was not used found that specific reasons for not using the order set (eg, treatment refusals, emergent intervention) were present in two‐thirds of the cases. This suggests that future analyses of adherence involving EMRs and CPOE implementation should take a more nuanced look at how order entry is actually enabled. It may be that understanding how order sets affect care enhances clinician acceptance and thus could serve as an incentive to EMR adoption. However, once an EMR is adopted, a need exists to continue evaluations such as this because, ultimately, the gold standard should be improved patient care processes and better outcomes for patients.

Although the prevalence of coronary heart disease and death from acute myocardial infarction (AMI) have declined steadily, about 935,000 heart attacks still occur annually in the United States, with approximately one‐third of these being fatal.[1, 2, 3] Studies have demonstrated decreased 30‐day and longer‐term mortality in AMI patients who receive evidence‐based treatment, including aspirin, ‐blockers, angiotensin‐converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs), anticoagulation therapy, and statins.[4, 5, 6, 7] Despite clinical practice guidelines (CPGs) outlining evidence‐based care and considerable efforts to implement processes that improve patient outcomes, delivery of effective therapy remains suboptimal.[8] For example, the Hospital Quality Alliance Program[9] found that in AMI patients, use of aspirin on admission was only 81% to 92%, ‐blocker on admission 75% to 85%, and ACE inhibitors for left ventricular dysfunction 71% to 74%.

Efforts to increase adherence to CPGs and improve patient outcomes in AMI have resulted in variable degrees of success. They include promotion of CPGs,[4, 5, 6, 7] physician education with feedback, report cards, care paths, registries,[10] Joint Commission standardized measures,[11] and paper checklists or order sets (OS).[12, 13]

In this report, we describe the association between use of an evidence‐based, electronic OS for AMI (AMI‐OS) and better adherence to CPGs. This AMI‐OS was implemented in the inpatient electronic medical records (EMRs) of a large integrated healthcare delivery system, Kaiser Permanente Northern California (KPNC). The purpose of our investigation was to determine (1) whether use of the AMI‐OS was associated with improved AMI processes and patient outcomes, and (2) whether these associations persisted after risk adjustment using a comprehensive severity of illness scoring system.

MATERIALS AND METHODS

This project was approved by the KPNC institutional review board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group, Inc., care for 3.4 million Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All KPNC facilities employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere.[14] Our setting consisted of 21 KPNC hospitals described in previous reports,[15, 16, 17, 18] using the same commercially available EMR system that includes computerized physician order entry (CPOE). Deployment of the customized inpatient Epic EMR (www.epicsystems.com), known internally as KP HealthConnect (KPHC), began in 2006 and was completed in 2010.

In this EMR's CPOE, physicians have options to select individual orders (a la carte) or they can utilize an OS, which is a collection of the most appropriate orders associated with specific diagnoses, procedures, or treatments. The evidence‐based AMI‐OS studied in this project was developed by a multidisciplinary team (for detailed components see Supporting Appendix 1Appendix 5 in the online version of this article).

Our study focused on the first set of hospital admission orders for patients with AMI. The study sample consisted of patients meeting these criteria: (1) age 18 years at admission; (2) admitted to a KPNC hospital for an overnight stay between September 28, 2008 and December 31, 2010; (3) principal diagnosis was AMI (International Classification of Diseases, 9th Revision [ICD‐9][19] codes 410.00, 01, 10, 11, 20, 21, 30, 31, 40, 41, 50, 51, 60, 61, 70, 71, 80, 90, and 91); and (4) KPHC had been operational at the hospital for at least 3 months to be included (for assembly descriptions see Supporting Appendices 15 in the online version of this article). At the study hospitals, troponin I was measured using the Beckman Access AccuTnI assay (Beckman Coulter, Inc., Brea, CA), whose upper reference limit (99th percentile) is 0.04 ng/mL. We excluded patients initially hospitalized for AMI at a non‐KPNC site and transferred into a study hospital.

The data processing methods we employed have been detailed elsewhere.[14, 15, 17, 20, 21, 22] The dependent outcome variables were total hospital length of stay, inpatient mortality, 30‐day mortality, and all‐cause rehospitalization within 30 days of discharge. Linked state mortality data were unavailable for the entire study period, so we ascertained 30‐day mortality based on the combination of KPNC patient demographic data and publicly available Social Security Administration decedent files. We ascertained rehospitalization by scanning KPNC hospitalization databases, which also track out‐of‐plan use.

The dependent process variables were use of aspirin within 24 hours of admission, ‐blockers, anticoagulation, ACE inhibitors or ARBs, and statins. The primary independent variable of interest was whether or not the admitting physician employed the AMI‐OS when admission orders were entered. Consequently, this variable is dichotomous (AMI‐OS vs a la carte).

We controlled for acute illness severity and chronic illness burden using a recent modification[22] of an externally validated risk‐adjustment system applicable to all hospitalized patients.[15, 16, 23, 24, 25] Our methodology included vital signs, neurological status checks, and laboratory test results obtained in the 72 hours preceding hospital admission; comorbidities were captured longitudinally using data from the year preceding hospitalization (for comparison purposes, we also assigned a Charlson Comorbidity Index score[26]).

End‐of‐life care directives are mandatory on admission at KPNC hospitals. Physicians have 4 options: full code, partial code, do not resuscitate, and comfort care only. Because of small numbers in some categories, we collapsed these 4 categories into full code and not full code. Because patients' care directives may change, we elected to capture the care directive in effect when a patient first entered a hospital unit other than the emergency department (ED).

Two authors (M.B., P.C.L.), one of whom is a board‐certified cardiologist, reviewed all admission electrocardiograms and made a consensus determination as to whether or not criteria for ST‐segment elevation myocardial infarction (STEMI) were present (ie, new ST‐segment elevation or left bundle branch block); we also reviewed the records of all patients with missing troponin I data to confirm the AMI diagnosis.

RESULTS

Table 1 summarizes the characteristics of the 5879 patients. It shows that AMI‐OS patients were more likely to receive evidence‐based therapies for AMI (aspirin, ‐blockers, ACE inhibitors or ARBs, anticoagulation, and statins) and had a 46% lower mortality rate in hospital (3.51 % vs 6.52%) and 33% lower rate at 30 days (5.66% vs 8.48%). AMI‐OS patients were also found to be at lower risk for an adverse outcome than nonAMI‐OS patients. The AMI‐OS patients had lower peak troponin I values, severity of illness (lower Laboratory‐Based Acute Physiology Score, version 2 [LAPS2] scores), comorbidity burdens (lower Comorbidity Point Score, version 2 [COPS2] and Charlson scores), and global predicted mortality risk. AMI‐OS patients were also less likely to have required intensive care. AMI‐OS patients were at higher risk of death than nonAMI‐OS patients with respect to only 1 variable (being full code at the time of admission), but although this difference was statistically significant, it was of minor clinical impact (86% vs 88%).

Table 2 shows the result of a logistic regression model in which the dependent variable was inpatient mortality and either the 5 evidence‐based therapies or the total number of evidence‐based therapies are the dependent variables. ‐blocker, statin, and ACE inhibitor or ARB therapies all had a protective effect on mortality, with odds ratios ranging from 0.48 (95% confidence interval [CI]: 0.36‐0.64), 0.63 (95% CI: 0.45‐0.89), and 0.40 (95% CI: 0.30‐0.53), respectively. An increased number of therapies also had a beneficial effect on inpatient mortality, with patients having 3 or more of the evidence‐based therapies showing an adjusted odds ratio (AOR) of 0.49 (95% CI: 0.33‐0.73), 4 or more therapies an AOR of 0.29 (95% CI: 0.20‐0.42), and 0.17 (95% CI: 0.11‐0.25) for 5 or more therapies.

Table 3 shows that the use of the AMI‐OS is protective, with an AOR of 0.59 and a 95% CI of 0.45‐0.76. Table 3 also shows that the most potent predictors were comorbidity burden (AOR: 1.07, 95% CI: 1.03‐1.10 per 10 COPS2 points), severity of illness (AOR: 1.09, 95% CI: 1.07‐1.12 per 10 LAPS2 points), STEMI (AOR: 3.86, 95% CI: 2.68‐5.58), and timing of cardiac catheterization referral occurring immediately prior to or during the admission (AOR: 0.37, 95% CI: 0.27‐0.51). The statistical significance of the AMI‐OS effect disappears when both AMI‐OS and the individual therapies are included in the same model (see Supporting Information, Appendices 15, in the online version of this article).

Table 4 shows separately the average treatment effect (ATE) and average treatment effect for the treated (ATT) of AMI‐OS and of increasing number of therapies on other outcomes (30‐day mortality, LOS, and readmission). Both the ATE and ATT show that the use of the AMI‐OS was significantly protective with respect to mortality and total hospital LOS but not significant with respect to readmission. The effect of the number of therapies on mortality is significantly higher with increasing number of therapies. For example, patients who received 5 therapies had an average treatment effect on 30‐day inpatient mortality of 0.23 (95% CI: 0.15‐0.35) compared to 0.64 (95% CI: 0.43‐0.96) for 3 therapies, almost a 3‐fold difference. The effects of increasing number of therapies were not significant for LOS or readmission. A sensitivity analysis in which the 535 STEMI patients were removed showed essentially the same results, so it is not reported here.

To further elucidate possible reasons why physicians did not use the AMI‐OS, the lead author reviewed 105 randomly selected records where the AMI‐OS was not used, 5 records from each of the 21 study hospitals. This review found that in 36% of patients, the AMI‐OS was not used because emergent catheterization or transfer to a facility with percutaneous coronary intervention capability occurred. Presence of other significant medical conditions, including critical illness, was the reason in 17% of these cases, patient or family refusal of treatments in 8%, issues around end‐of‐life care in 3%, and specific medical contraindications in 1%. In the remaining 34%, no reason for not using the AMI‐OS could be identified.

DISCUSSION

We evaluated the use of an evidence‐based electronic AMI‐OS embedded in a comprehensive EMR and found that it was beneficial. Its use was associated with increased adherence to evidence‐based therapies, which in turn were associated with improved outcomes. Using data from a large cohort of hospitalized AMI patients in 21 community hospitals, we were able to use risk adjustment that included physiologic illness severity to adjust for baseline mortality risk. Patients in whom the AMI‐OS was employed tended to be at lower risk; nonetheless, after controlling for confounding variables and adjusting for bias using propensity scores, the AMI‐OS was associated with increased use of evidence‐based therapies and decreased mortality. Most importantly, it appears that the benefits of the OS were not just due to increased receipt of individual recommended therapies, but to increased concurrent receipt of multiple recommended therapies.

Modern EMRs have great potential for significant improvements in the quality, efficiency, and safety of care provided,[36] and our study highlights this potential. However, a number of important limitations to our study must be considered. Although we had access to a very rich dataset, we could not control for all possible confounders, and our risk adjustment cannot match the level of information available to clinicians. In particular, the measurements available to us with respect to cardiac risk are limited. Thus, we have to recognize that the strength of our findings does not approximate that of a randomized trial, and one would expect that the magnitude of the beneficial association would fall under more controlled conditions. Resource limitations also did not permit us to gather more time course data (eg, sequential measurements of patient instability, cardiac damage, or use of recommended therapies), which could provide a better delineation of differences in both processes and outcomes.

Limitations also exist to the generalizability of the use of order sets in other settings that go beyond the availability of a comprehensive EMR. Our study population was cared for in a setting with an unusually high level of integration.[1] For example, KPNC has an elaborate administrative infrastructure for training in the use of the EMR as well as ensuring that order sets are not just evidence‐based, but that they are perceived by clinicians to be of significant value. This infrastructure, established to ensure physician buy‐in, may not be easy to replicate in smaller or less‐integrated settings. Thus, it is conceivable that factors other than the degree of support during the EMR deployments can affect rates of order set use.

Although our use of counterfactual methods included illness severity (LAPS2) and longitudinal comorbidity burden (COPS2), which are not yet available outside highly integrated delivery services employing comprehensive EMRs, it is possible they are insufficient. We cannot exclude the possibility that other biases or patient characteristics were present that led clinicians to preferentially employ the electronic order set in some patients but not in others. One could also argue that future studies should consider using overall adherence to recommended AMI treatment guidelines as a risk adjustment tool that would permit one to analyze what other factors may be playing a role in residual differences in patient outcomes. Last, one could object to our inclusion of STEMI patients; however, this was not a study on optimum treatment strategies for STEMI patients. Rather, it was a study on the impact on AMI outcomes of a specific component of computerized order entry outside the research setting.

Despite these limitations, we believe that our findings provide strong support for the continued use of electronic evidence‐based order sets in the inpatient medical setting. Once the initial implementation of a comprehensive EMR has occurred, deployment of these electronic order sets is a relatively inexpensive but effective method to foster compliance with evidence‐based care.

Future research in healthcare information technology can take a number of directions. One important area, of course, revolves around ways to promote enhanced physician adoption of EMRs. Our audit of records where the AMI‐OS was not used found that specific reasons for not using the order set (eg, treatment refusals, emergent intervention) were present in two‐thirds of the cases. This suggests that future analyses of adherence involving EMRs and CPOE implementation should take a more nuanced look at how order entry is actually enabled. It may be that understanding how order sets affect care enhances clinician acceptance and thus could serve as an incentive to EMR adoption. However, once an EMR is adopted, a need exists to continue evaluations such as this because, ultimately, the gold standard should be improved patient care processes and better outcomes for patients.

Although the prevalence of coronary heart disease and death from acute myocardial infarction (AMI) have declined steadily, about 935,000 heart attacks still occur annually in the United States, with approximately one‐third of these being fatal.[1, 2, 3] Studies have demonstrated decreased 30‐day and longer‐term mortality in AMI patients who receive evidence‐based treatment, including aspirin, ‐blockers, angiotensin‐converting enzyme (ACE) inhibitors or angiotensin receptor blockers (ARBs), anticoagulation therapy, and statins.[4, 5, 6, 7] Despite clinical practice guidelines (CPGs) outlining evidence‐based care and considerable efforts to implement processes that improve patient outcomes, delivery of effective therapy remains suboptimal.[8] For example, the Hospital Quality Alliance Program[9] found that in AMI patients, use of aspirin on admission was only 81% to 92%, ‐blocker on admission 75% to 85%, and ACE inhibitors for left ventricular dysfunction 71% to 74%.

Efforts to increase adherence to CPGs and improve patient outcomes in AMI have resulted in variable degrees of success. They include promotion of CPGs,[4, 5, 6, 7] physician education with feedback, report cards, care paths, registries,[10] Joint Commission standardized measures,[11] and paper checklists or order sets (OS).[12, 13]

In this report, we describe the association between use of an evidence‐based, electronic OS for AMI (AMI‐OS) and better adherence to CPGs. This AMI‐OS was implemented in the inpatient electronic medical records (EMRs) of a large integrated healthcare delivery system, Kaiser Permanente Northern California (KPNC). The purpose of our investigation was to determine (1) whether use of the AMI‐OS was associated with improved AMI processes and patient outcomes, and (2) whether these associations persisted after risk adjustment using a comprehensive severity of illness scoring system.

MATERIALS AND METHODS

This project was approved by the KPNC institutional review board.

Under a mutual exclusivity arrangement, salaried physicians of The Permanente Medical Group, Inc., care for 3.4 million Kaiser Foundation Health Plan, Inc. members at facilities owned by Kaiser Foundation Hospitals, Inc. All KPNC facilities employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere.[14] Our setting consisted of 21 KPNC hospitals described in previous reports,[15, 16, 17, 18] using the same commercially available EMR system that includes computerized physician order entry (CPOE). Deployment of the customized inpatient Epic EMR (www.epicsystems.com), known internally as KP HealthConnect (KPHC), began in 2006 and was completed in 2010.

In this EMR's CPOE, physicians have options to select individual orders (a la carte) or they can utilize an OS, which is a collection of the most appropriate orders associated with specific diagnoses, procedures, or treatments. The evidence‐based AMI‐OS studied in this project was developed by a multidisciplinary team (for detailed components see Supporting Appendix 1Appendix 5 in the online version of this article).

Our study focused on the first set of hospital admission orders for patients with AMI. The study sample consisted of patients meeting these criteria: (1) age 18 years at admission; (2) admitted to a KPNC hospital for an overnight stay between September 28, 2008 and December 31, 2010; (3) principal diagnosis was AMI (International Classification of Diseases, 9th Revision [ICD‐9][19] codes 410.00, 01, 10, 11, 20, 21, 30, 31, 40, 41, 50, 51, 60, 61, 70, 71, 80, 90, and 91); and (4) KPHC had been operational at the hospital for at least 3 months to be included (for assembly descriptions see Supporting Appendices 15 in the online version of this article). At the study hospitals, troponin I was measured using the Beckman Access AccuTnI assay (Beckman Coulter, Inc., Brea, CA), whose upper reference limit (99th percentile) is 0.04 ng/mL. We excluded patients initially hospitalized for AMI at a non‐KPNC site and transferred into a study hospital.

The data processing methods we employed have been detailed elsewhere.[14, 15, 17, 20, 21, 22] The dependent outcome variables were total hospital length of stay, inpatient mortality, 30‐day mortality, and all‐cause rehospitalization within 30 days of discharge. Linked state mortality data were unavailable for the entire study period, so we ascertained 30‐day mortality based on the combination of KPNC patient demographic data and publicly available Social Security Administration decedent files. We ascertained rehospitalization by scanning KPNC hospitalization databases, which also track out‐of‐plan use.

The dependent process variables were use of aspirin within 24 hours of admission, ‐blockers, anticoagulation, ACE inhibitors or ARBs, and statins. The primary independent variable of interest was whether or not the admitting physician employed the AMI‐OS when admission orders were entered. Consequently, this variable is dichotomous (AMI‐OS vs a la carte).

We controlled for acute illness severity and chronic illness burden using a recent modification[22] of an externally validated risk‐adjustment system applicable to all hospitalized patients.[15, 16, 23, 24, 25] Our methodology included vital signs, neurological status checks, and laboratory test results obtained in the 72 hours preceding hospital admission; comorbidities were captured longitudinally using data from the year preceding hospitalization (for comparison purposes, we also assigned a Charlson Comorbidity Index score[26]).

End‐of‐life care directives are mandatory on admission at KPNC hospitals. Physicians have 4 options: full code, partial code, do not resuscitate, and comfort care only. Because of small numbers in some categories, we collapsed these 4 categories into full code and not full code. Because patients' care directives may change, we elected to capture the care directive in effect when a patient first entered a hospital unit other than the emergency department (ED).

Two authors (M.B., P.C.L.), one of whom is a board‐certified cardiologist, reviewed all admission electrocardiograms and made a consensus determination as to whether or not criteria for ST‐segment elevation myocardial infarction (STEMI) were present (ie, new ST‐segment elevation or left bundle branch block); we also reviewed the records of all patients with missing troponin I data to confirm the AMI diagnosis.

RESULTS

Table 1 summarizes the characteristics of the 5879 patients. It shows that AMI‐OS patients were more likely to receive evidence‐based therapies for AMI (aspirin, ‐blockers, ACE inhibitors or ARBs, anticoagulation, and statins) and had a 46% lower mortality rate in hospital (3.51 % vs 6.52%) and 33% lower rate at 30 days (5.66% vs 8.48%). AMI‐OS patients were also found to be at lower risk for an adverse outcome than nonAMI‐OS patients. The AMI‐OS patients had lower peak troponin I values, severity of illness (lower Laboratory‐Based Acute Physiology Score, version 2 [LAPS2] scores), comorbidity burdens (lower Comorbidity Point Score, version 2 [COPS2] and Charlson scores), and global predicted mortality risk. AMI‐OS patients were also less likely to have required intensive care. AMI‐OS patients were at higher risk of death than nonAMI‐OS patients with respect to only 1 variable (being full code at the time of admission), but although this difference was statistically significant, it was of minor clinical impact (86% vs 88%).

Table 2 shows the result of a logistic regression model in which the dependent variable was inpatient mortality and either the 5 evidence‐based therapies or the total number of evidence‐based therapies are the dependent variables. ‐blocker, statin, and ACE inhibitor or ARB therapies all had a protective effect on mortality, with odds ratios ranging from 0.48 (95% confidence interval [CI]: 0.36‐0.64), 0.63 (95% CI: 0.45‐0.89), and 0.40 (95% CI: 0.30‐0.53), respectively. An increased number of therapies also had a beneficial effect on inpatient mortality, with patients having 3 or more of the evidence‐based therapies showing an adjusted odds ratio (AOR) of 0.49 (95% CI: 0.33‐0.73), 4 or more therapies an AOR of 0.29 (95% CI: 0.20‐0.42), and 0.17 (95% CI: 0.11‐0.25) for 5 or more therapies.

Table 3 shows that the use of the AMI‐OS is protective, with an AOR of 0.59 and a 95% CI of 0.45‐0.76. Table 3 also shows that the most potent predictors were comorbidity burden (AOR: 1.07, 95% CI: 1.03‐1.10 per 10 COPS2 points), severity of illness (AOR: 1.09, 95% CI: 1.07‐1.12 per 10 LAPS2 points), STEMI (AOR: 3.86, 95% CI: 2.68‐5.58), and timing of cardiac catheterization referral occurring immediately prior to or during the admission (AOR: 0.37, 95% CI: 0.27‐0.51). The statistical significance of the AMI‐OS effect disappears when both AMI‐OS and the individual therapies are included in the same model (see Supporting Information, Appendices 15, in the online version of this article).

Table 4 shows separately the average treatment effect (ATE) and average treatment effect for the treated (ATT) of AMI‐OS and of increasing number of therapies on other outcomes (30‐day mortality, LOS, and readmission). Both the ATE and ATT show that the use of the AMI‐OS was significantly protective with respect to mortality and total hospital LOS but not significant with respect to readmission. The effect of the number of therapies on mortality is significantly higher with increasing number of therapies. For example, patients who received 5 therapies had an average treatment effect on 30‐day inpatient mortality of 0.23 (95% CI: 0.15‐0.35) compared to 0.64 (95% CI: 0.43‐0.96) for 3 therapies, almost a 3‐fold difference. The effects of increasing number of therapies were not significant for LOS or readmission. A sensitivity analysis in which the 535 STEMI patients were removed showed essentially the same results, so it is not reported here.

To further elucidate possible reasons why physicians did not use the AMI‐OS, the lead author reviewed 105 randomly selected records where the AMI‐OS was not used, 5 records from each of the 21 study hospitals. This review found that in 36% of patients, the AMI‐OS was not used because emergent catheterization or transfer to a facility with percutaneous coronary intervention capability occurred. Presence of other significant medical conditions, including critical illness, was the reason in 17% of these cases, patient or family refusal of treatments in 8%, issues around end‐of‐life care in 3%, and specific medical contraindications in 1%. In the remaining 34%, no reason for not using the AMI‐OS could be identified.

DISCUSSION

We evaluated the use of an evidence‐based electronic AMI‐OS embedded in a comprehensive EMR and found that it was beneficial. Its use was associated with increased adherence to evidence‐based therapies, which in turn were associated with improved outcomes. Using data from a large cohort of hospitalized AMI patients in 21 community hospitals, we were able to use risk adjustment that included physiologic illness severity to adjust for baseline mortality risk. Patients in whom the AMI‐OS was employed tended to be at lower risk; nonetheless, after controlling for confounding variables and adjusting for bias using propensity scores, the AMI‐OS was associated with increased use of evidence‐based therapies and decreased mortality. Most importantly, it appears that the benefits of the OS were not just due to increased receipt of individual recommended therapies, but to increased concurrent receipt of multiple recommended therapies.

Modern EMRs have great potential for significant improvements in the quality, efficiency, and safety of care provided,[36] and our study highlights this potential. However, a number of important limitations to our study must be considered. Although we had access to a very rich dataset, we could not control for all possible confounders, and our risk adjustment cannot match the level of information available to clinicians. In particular, the measurements available to us with respect to cardiac risk are limited. Thus, we have to recognize that the strength of our findings does not approximate that of a randomized trial, and one would expect that the magnitude of the beneficial association would fall under more controlled conditions. Resource limitations also did not permit us to gather more time course data (eg, sequential measurements of patient instability, cardiac damage, or use of recommended therapies), which could provide a better delineation of differences in both processes and outcomes.

Limitations also exist to the generalizability of the use of order sets in other settings that go beyond the availability of a comprehensive EMR. Our study population was cared for in a setting with an unusually high level of integration.[1] For example, KPNC has an elaborate administrative infrastructure for training in the use of the EMR as well as ensuring that order sets are not just evidence‐based, but that they are perceived by clinicians to be of significant value. This infrastructure, established to ensure physician buy‐in, may not be easy to replicate in smaller or less‐integrated settings. Thus, it is conceivable that factors other than the degree of support during the EMR deployments can affect rates of order set use.

Although our use of counterfactual methods included illness severity (LAPS2) and longitudinal comorbidity burden (COPS2), which are not yet available outside highly integrated delivery services employing comprehensive EMRs, it is possible they are insufficient. We cannot exclude the possibility that other biases or patient characteristics were present that led clinicians to preferentially employ the electronic order set in some patients but not in others. One could also argue that future studies should consider using overall adherence to recommended AMI treatment guidelines as a risk adjustment tool that would permit one to analyze what other factors may be playing a role in residual differences in patient outcomes. Last, one could object to our inclusion of STEMI patients; however, this was not a study on optimum treatment strategies for STEMI patients. Rather, it was a study on the impact on AMI outcomes of a specific component of computerized order entry outside the research setting.

Despite these limitations, we believe that our findings provide strong support for the continued use of electronic evidence‐based order sets in the inpatient medical setting. Once the initial implementation of a comprehensive EMR has occurred, deployment of these electronic order sets is a relatively inexpensive but effective method to foster compliance with evidence‐based care.

Future research in healthcare information technology can take a number of directions. One important area, of course, revolves around ways to promote enhanced physician adoption of EMRs. Our audit of records where the AMI‐OS was not used found that specific reasons for not using the order set (eg, treatment refusals, emergent intervention) were present in two‐thirds of the cases. This suggests that future analyses of adherence involving EMRs and CPOE implementation should take a more nuanced look at how order entry is actually enabled. It may be that understanding how order sets affect care enhances clinician acceptance and thus could serve as an incentive to EMR adoption. However, once an EMR is adopted, a need exists to continue evaluations such as this because, ultimately, the gold standard should be improved patient care processes and better outcomes for patients.