Lawrence Tim Goodnough, MD

Transfusion of blood products has multiple clinical applications, yet when used outside of recommended guidelines it is associated with increased cost, waste, morbidity, and mortality.[1] Studies reviewing restrictive versus liberal blood transfusion strategies have demonstrated no benefit to liberal strategies over restrictive strategies[2, 3, 4, 5, 6, 7, 8, 9, 10] and possibly even harm.[11] Given the risks and real costs of blood transfusions occurring outside of recommended guidelines, professional societies in hematology, anesthesiology, and hospital medicine each include excessive blood transfusions among their top 5 questionable clinical practices in the American Board of Internal Medicine Foundation's Choosing Wisely initiative.[12] To modify behaviors around blood transfusion practices, hospitals and blood banks may need to provide clinical decision support (CDS) for physicians.

The conventional approach to CDS is direct education and training campaigns, but there is a unique opportunity for intelligent decision support at the point of care through electronic medical record (EMR) systems directly integrated with computerized physician order entry (CPOE).[13, 14, 15] Prior work at Stanford toward reducing unnecessary blood transfusions started with hospital‐wide education campaigns, which brought down the percentage of transfusions ordered for patients with hemoglobin (Hgb) >8 g/dL from 57% to 52%. Further reduction to <30% was achieved after the introduction of an interruptive best practice alert (BPA) integrated into the CPOE transfusion ordering process.[16, 17] Specifically, providers attempting to order a blood transfusion for patients with an EMR‐determined Hgb >7 are presented with a BPA popup reminding them of best practice guidelines and a prompt to either abort the transfusion or provide a reason to override the BPA and proceed.

It remains uncertain why up to 30% of transfusions continue to occur outside of recommended guidelines in spite of interruptive prompts. This study demonstrates a general approach to secondary use of clinical data from the EMR toward understanding provider behavior, specifically by analyzing free‐text comments linked to transfusion override behavior and identifying the type of providers interacting with the BPAs.

MATERIALS AND METHODS

At Stanford University Hospital, a 447‐bed academic tertiary care center servicing adult patients, clinicians order blood transfusions through an EMR+CPOE system. When an order for red blood cell transfusion is attempted, the EMR evaluates the patient chart for specific criteria based on previously published guidelines.[18] Specifically, the BPA will review the last recorded Hgb value and trigger if the Hgb is >8, or if the Hgb is >7 and there is no concurrent EMR problem‐list entry related to acute coronary syndrome or acute hemorrhage. Once the blood transfusion BPA triggers, the ordering provider is presented with an interruptive prompt reminding them of best practice guidelines and the 3 most recent Hgb values for the patient.[16] From there, the provider may either abort the transfusion or override the BPA and proceed. Overrides require the provider to select a reason from a predefined list of institutionally accepted transfusion indications including acute bleeding, acute coronary syndrome and Hgb <8, and postoperative cardiothoracic surgery and Hgb <8. If none of the predefined override reasons are selected, the provider simply selects other, with the option of a free‐text comment to elaborate their rationale.

Data from provider interactions with the BPA were collected from the EMR across all inpatient wards for 8 months after the implementation period. Data collected from each interaction included the patient identifier, alert description, action taken, ordering provider identifier, ordering provider type (job title), optional comments for overriding the BPA, and date and time.

The free‐text override comments were independently reviewed by 2 licensed physicians, tagging them into a set of general categories by iterative inductive analysis of the comment content. Individual comments were allowed multiple possible category tags (eg, coronary artery disease and symptomatic). The initial tagging process was expedited by isolating common keywords in the override comments and assigning likely category tags to each, as in Table 2. The 2 physician reviewers then manually inspected all comments with the option to revise the initial category tagging to ensure validity. Many category tags occurred sporadically and were aggregated into more general categories, such as hematologic disorder (includes myelodysplastic syndrome, myelofibrosis, multiple myeloma, sickle cell, thalassemia, Waldenstrom's), symptomatic (includes fatigue, lightheaded, short of breath), and per other medical doctor [MD] (includes any other physician direction, primarily specialty consultation services).

To assess the inter‐rater agreement of this category tagging between the 2 reviewers, a confusion matrix similar to the example in Table 1 was setup for each tag used. Several agreement statistics are calculated based on the confusion matrix, including the positive agreement rate (P_a+) and Cohen's kappa statistic (). Kappa statistic values range from 1 to +1, with values <0 indicating no agreement and values >0.8 indicating near perfect agreement.[19] To reject the null hypothesis that the 2 reviewers could have independently arrived at their similar tagging assignments by chance, a ² test was applied for each confusion matrix, with Yates' correction to avoid overestimating statistical significance given the low rates of inter‐rater disagreement.[20]

RESULTS

During the data collection period from January 1, 2011 to August 31, 2012, the blood transfusion BPA fired in 11,791 instances, of which 10,642 recorded sufficient data for analysis (Figure 1). The ordering provider proceeded to override the BPA and continued with transfusion in 98% of cases (10,442/10,642). Acute bleeding" was the most common structured response (34%). The majority of BPA overrides used the general purpose other structured response option accounting for 56% (5886/10,442) of override responses, of which 37% (2185/5886) entered a free‐text comment elaborating the override reason. With 3701 nonresponders (other overrides with no free‐text comment), the overall response rate was 65% (6941/10,642).

Figure 1

With a handful of free‐text comments included with structured override responses (eg, 28 acute bleeding overrides included additional comment from the provider), a total of 2216 override comments were available for analysis. Using an initial selection of keyword‐tag associations, as in Table 2, 95% (2104/2216) of the override comments had a preliminary category tagging assigned. After manual review and revision by the first physician reviewer, 74% (1633/2216) of the comments retained their automated tags, whereas 26% (583/2216) were updated based on the reviewer's assessment of validity. This included 112 comments lacking automated tags the reviewer manually added, as well as 471 comments with automated tags revised by the reviewer.

The most common override category tags are presented in Figure 2 and Figure 3 for BPAs triggered in response to blood transfusions ordered for patients with Hgb 78 and Hgb >8, respectively. The agreed+ percentages correspond to the number of comments where the 2 physician reviewers both assigned the respective category tag (P_a+), whereas the disagreed percentages correspond to comments where the reviewers differed (P_d=1‐P_a). By Yates' corrected ² analysis, P was <10¹⁰ for independence between reviewer tag assignments for all tags assessed. Kappa statistics are reported in the figures to describe inter‐rater agreement.

Figure 2

Figure 3

Resident physicians were the primary ordering provider group, accounting for approximately 55% (5863/10,642) of BPA interactions, followed by registered nurses, fellows, and attending physicians.

DISCUSSION

Prior work has established that an interruptive BPA to provide decision support against inappropriate blood transfusions can drive a significant and sustained reduction in unnecessary transfusions,[16] especially when satisfying the primary features of effective decision support.[14] The quantity of transfusions directly aborted by the BPA (only 2% in this case) does not nearly account for the total observed reductions in transfusions, suggesting that the BPA yields an indirect teaching effect over time beyond direct interruption. In other words, once a user has been interrupted by the BPA several times, they will be discouraged from even attempting to order inappropriate blood transfusions in the future.

Despite the improvements above, there remains a substantial fraction (up to 30%) of blood transfusions occurring outside of recommended guidelines where providers specifically override the BPA prompt.[21] This work is the first structured analysis of such BPA override behavior by generalizable methods of EMR data analysis to guide further improvements.

Analysis of the override behavior focused primarily on the free‐text comments explaining provider rationales for overriding the BPA. These comments were categorized by 2 physician reviewers, with P<10¹⁰ for all category tags, rejecting the null hypothesis that the 2 reviewers could have independently arrived at their similar category tags by chance. Further assessment of inter‐rater agreement was generally excellent, with >0.8 in the majority of cases. The remaining cases (ie, Hgb 78, per other MD, anemia, cardiac) still had substantial agreement, with >0.6. These disagreements almost universally occurred when a reviewer assigned a subset of the category tags assigned by the other reviewer. For example, 1 reviewer may tag a comment bone marrow transplant (BMT) patient with Hgb 7.2 with BMT and Hgb 78, whereas the other would only cite BMT.

Figure 2 and Figure 3 highlight the varied categories of responses, with most categories comprising <4% of the available responses. Among the most common override reasons are structured protocols for specialty services, as in the stem cell transplant service, whose protocol recommends transfusing blood when Hgb <8 rather than Hgb <7. In these cases, the BPA is unlikely to change protocolized behavior, thus overall workflow would likely be improved by adding a structured BPA override option for these protocols.

Analysis of the override comments did expose some relatively common and questionable transfusion practices, including perioperative and periprocedural transfusions, as well as anticipation of imminent discharge. Prior studies indicate that transfusions in anticipation of surgical or procedural intervention provide no benefit compared to responding to blood requirements intraoperatively as needed,[3, 22, 23] and imminent discharge of a patient is not a well‐recognized reason to transfuse outside of guidelines. The identification of these questionable and relatively common practices identifies opportunities for targeted education and training campaigns.

Symptomatic anemia was 1 of the more commonly cited BPA override reasons with a fraction providing a specific symptom such as fatigue, lightheadedness, or shortness of breath. Although the pervasiveness of this rationale may speak to adding it as a structured BPA override option, the nonspecific, subjective, and nonevidence‐based nature of symptom‐driven blood transfusions suggests that ongoing interruptive BPA prompts can still be useful to remind providers of the risks and guideline‐based approaches to such cases.

Limitations of the analysis are revealed as a fraction of BPA events did not fully record all relevant data, preventing proper analysis. Override comments suggest the BPA was triggering inappropriately for patients appropriately below the recommended transfusion threshold of Hgb <7, assuming provider free‐textentered values were accurate. Review of these cases showed some variability, such as when providers based their transfusion decision on a hematocrit of 20 rather than a Hgb of 7. Many comments also stated nonthreshold Hgb values, such as Hgb 7.2, seeming to imply that the value was close enough to the recommended threshold to justify overriding the transfusion prompt.

The most significant limitation of this study is the substantial fraction of nonresponder BPA transfusion overrides with a nonspecific other reason and no text commentary, comprising 34% (3670/10,642) of all BPA interactions and 62% (3670/5886) of all other overrides. Although the BPA is easily overridden by design to avoid workflow disruption that could compromise the priority of patient care, the nonresponses raise concern for skewed interpretation of the override data. General studies in survey responses provide reassurance that lower response rates do not necessarily indicate response bias,[24] with response rates as low as 25% yielding results statistically indistinguishable from more rigorous methods achieving >50% response rates.[25] In this specific case, response bias is better characterized by comparing ordering provider characteristics for the other overrides with and without free‐text comments. Specifically, Figure 4 shows the distribution of other overrides by provider type (job title) and provider home department (where available from physician department registries), separated by whether a free‐text comment was left. For each sub‐group, a ² analysis compared the observed versus expected proportion of providers leaving comments based on the null hypothesis that leaving a comment was independent of membership in the subgroup. Similar proportions with nonsignificant P values suggest against significant response bias for most subgroups, but the data do indicate that this work likely under‐represents the opinion of fellows, physician assistants, and neurosurgeons, while slightly over‐representing the opinion of medicine, general surgery, and obstetrics/gynecology practitioners. Given that the under‐represented groups overall constitute a small minority of the total BPA interactions, this work should still be generalizable to the majority of transfusion behavior.

Figure 4

Further review of the ordering provider type (job title) distribution indicates that the vast majority of providers who actually interact with BPAs in this academic hospital are resident physicians. Similarly notable is that 25% of BPAs trigger for registered nurses. Upon review of hospital practices, we confirmed that nurses can enter transfusion orders on behalf of MD cosigners, generally in the context of a verbal order or routine treatment protocol, most notably on the stem cell transplant service. The prevalence of resident, nurse, nurse practitioner, and physician assistant order entry overall indicates that the providers encountering the BPA often do not have the negotiating power to accept or override the prompts, specifically defying the meaningful use goal of decision‐making providers directly interacting with CPOE and decision support prompts.[26]

The theoretical intelligence of the BPA not to trigger in well‐accepted cases of transfusion, where acute bleeding is occurring based on the presence of bleeding related items in the patient's problem list, is demonstrated to be limited. Acute bleeding represented the most common structured override response. This is likely multifactorial, but is largely the result of inconsistent usage of the patient problem lists that the BPA would depend upon. Another commonly cited override explanation was for a dropping Hgb without a specific bleeding source. This could point toward the BPA triggering an algorithm incorporating the last several Hgb values to assess the trajectory. Even then, however, guidelines would advocate holding transfusion and proceeding with serial monitoring of Hgb levels until an accepted transfusion threshold was actually breached.

This analysis demonstrates a general approach to secondary analysis of EMR data, in this case to provide insight into the specific question of why providers continue to order costly, unnecessary, and potentially harmful blood transfusions in spite of an interruptive BPA CDS tool. Limitations of the BPA are now better understood, including technical aspects of accounting for all decision‐making data, the frequency of nonresponse for override rationale, and that the providers interacting with the BPAs often do not have the negotiating power to change ordering behavior. The analysis recognizes protocolized behaviors that should be integrated into the CDS algorithms, and specification of symptomatic anemia as a common albeit subjective indication for blood transfusion. A wide swath of individually uncommon rationales drive transfusion override behavior, motivating further EMR‐based analysis to more efficiently recognize common scenarios (eg, perioperative/procedural, imminent discharge) for directed education and training interventions than can be achieved by conventional chart review.

Transfusion of blood products has multiple clinical applications, yet when used outside of recommended guidelines it is associated with increased cost, waste, morbidity, and mortality.[1] Studies reviewing restrictive versus liberal blood transfusion strategies have demonstrated no benefit to liberal strategies over restrictive strategies[2, 3, 4, 5, 6, 7, 8, 9, 10] and possibly even harm.[11] Given the risks and real costs of blood transfusions occurring outside of recommended guidelines, professional societies in hematology, anesthesiology, and hospital medicine each include excessive blood transfusions among their top 5 questionable clinical practices in the American Board of Internal Medicine Foundation's Choosing Wisely initiative.[12] To modify behaviors around blood transfusion practices, hospitals and blood banks may need to provide clinical decision support (CDS) for physicians.

The conventional approach to CDS is direct education and training campaigns, but there is a unique opportunity for intelligent decision support at the point of care through electronic medical record (EMR) systems directly integrated with computerized physician order entry (CPOE).[13, 14, 15] Prior work at Stanford toward reducing unnecessary blood transfusions started with hospital‐wide education campaigns, which brought down the percentage of transfusions ordered for patients with hemoglobin (Hgb) >8 g/dL from 57% to 52%. Further reduction to <30% was achieved after the introduction of an interruptive best practice alert (BPA) integrated into the CPOE transfusion ordering process.[16, 17] Specifically, providers attempting to order a blood transfusion for patients with an EMR‐determined Hgb >7 are presented with a BPA popup reminding them of best practice guidelines and a prompt to either abort the transfusion or provide a reason to override the BPA and proceed.

It remains uncertain why up to 30% of transfusions continue to occur outside of recommended guidelines in spite of interruptive prompts. This study demonstrates a general approach to secondary use of clinical data from the EMR toward understanding provider behavior, specifically by analyzing free‐text comments linked to transfusion override behavior and identifying the type of providers interacting with the BPAs.

MATERIALS AND METHODS

At Stanford University Hospital, a 447‐bed academic tertiary care center servicing adult patients, clinicians order blood transfusions through an EMR+CPOE system. When an order for red blood cell transfusion is attempted, the EMR evaluates the patient chart for specific criteria based on previously published guidelines.[18] Specifically, the BPA will review the last recorded Hgb value and trigger if the Hgb is >8, or if the Hgb is >7 and there is no concurrent EMR problem‐list entry related to acute coronary syndrome or acute hemorrhage. Once the blood transfusion BPA triggers, the ordering provider is presented with an interruptive prompt reminding them of best practice guidelines and the 3 most recent Hgb values for the patient.[16] From there, the provider may either abort the transfusion or override the BPA and proceed. Overrides require the provider to select a reason from a predefined list of institutionally accepted transfusion indications including acute bleeding, acute coronary syndrome and Hgb <8, and postoperative cardiothoracic surgery and Hgb <8. If none of the predefined override reasons are selected, the provider simply selects other, with the option of a free‐text comment to elaborate their rationale.

Data from provider interactions with the BPA were collected from the EMR across all inpatient wards for 8 months after the implementation period. Data collected from each interaction included the patient identifier, alert description, action taken, ordering provider identifier, ordering provider type (job title), optional comments for overriding the BPA, and date and time.

The free‐text override comments were independently reviewed by 2 licensed physicians, tagging them into a set of general categories by iterative inductive analysis of the comment content. Individual comments were allowed multiple possible category tags (eg, coronary artery disease and symptomatic). The initial tagging process was expedited by isolating common keywords in the override comments and assigning likely category tags to each, as in Table 2. The 2 physician reviewers then manually inspected all comments with the option to revise the initial category tagging to ensure validity. Many category tags occurred sporadically and were aggregated into more general categories, such as hematologic disorder (includes myelodysplastic syndrome, myelofibrosis, multiple myeloma, sickle cell, thalassemia, Waldenstrom's), symptomatic (includes fatigue, lightheaded, short of breath), and per other medical doctor [MD] (includes any other physician direction, primarily specialty consultation services).

To assess the inter‐rater agreement of this category tagging between the 2 reviewers, a confusion matrix similar to the example in Table 1 was setup for each tag used. Several agreement statistics are calculated based on the confusion matrix, including the positive agreement rate (P_a+) and Cohen's kappa statistic (). Kappa statistic values range from 1 to +1, with values <0 indicating no agreement and values >0.8 indicating near perfect agreement.[19] To reject the null hypothesis that the 2 reviewers could have independently arrived at their similar tagging assignments by chance, a ² test was applied for each confusion matrix, with Yates' correction to avoid overestimating statistical significance given the low rates of inter‐rater disagreement.[20]

RESULTS

During the data collection period from January 1, 2011 to August 31, 2012, the blood transfusion BPA fired in 11,791 instances, of which 10,642 recorded sufficient data for analysis (Figure 1). The ordering provider proceeded to override the BPA and continued with transfusion in 98% of cases (10,442/10,642). Acute bleeding" was the most common structured response (34%). The majority of BPA overrides used the general purpose other structured response option accounting for 56% (5886/10,442) of override responses, of which 37% (2185/5886) entered a free‐text comment elaborating the override reason. With 3701 nonresponders (other overrides with no free‐text comment), the overall response rate was 65% (6941/10,642).

Figure 1

With a handful of free‐text comments included with structured override responses (eg, 28 acute bleeding overrides included additional comment from the provider), a total of 2216 override comments were available for analysis. Using an initial selection of keyword‐tag associations, as in Table 2, 95% (2104/2216) of the override comments had a preliminary category tagging assigned. After manual review and revision by the first physician reviewer, 74% (1633/2216) of the comments retained their automated tags, whereas 26% (583/2216) were updated based on the reviewer's assessment of validity. This included 112 comments lacking automated tags the reviewer manually added, as well as 471 comments with automated tags revised by the reviewer.

The most common override category tags are presented in Figure 2 and Figure 3 for BPAs triggered in response to blood transfusions ordered for patients with Hgb 78 and Hgb >8, respectively. The agreed+ percentages correspond to the number of comments where the 2 physician reviewers both assigned the respective category tag (P_a+), whereas the disagreed percentages correspond to comments where the reviewers differed (P_d=1‐P_a). By Yates' corrected ² analysis, P was <10¹⁰ for independence between reviewer tag assignments for all tags assessed. Kappa statistics are reported in the figures to describe inter‐rater agreement.

Figure 2

Figure 3

Resident physicians were the primary ordering provider group, accounting for approximately 55% (5863/10,642) of BPA interactions, followed by registered nurses, fellows, and attending physicians.

DISCUSSION

Prior work has established that an interruptive BPA to provide decision support against inappropriate blood transfusions can drive a significant and sustained reduction in unnecessary transfusions,[16] especially when satisfying the primary features of effective decision support.[14] The quantity of transfusions directly aborted by the BPA (only 2% in this case) does not nearly account for the total observed reductions in transfusions, suggesting that the BPA yields an indirect teaching effect over time beyond direct interruption. In other words, once a user has been interrupted by the BPA several times, they will be discouraged from even attempting to order inappropriate blood transfusions in the future.

Despite the improvements above, there remains a substantial fraction (up to 30%) of blood transfusions occurring outside of recommended guidelines where providers specifically override the BPA prompt.[21] This work is the first structured analysis of such BPA override behavior by generalizable methods of EMR data analysis to guide further improvements.

Analysis of the override behavior focused primarily on the free‐text comments explaining provider rationales for overriding the BPA. These comments were categorized by 2 physician reviewers, with P<10¹⁰ for all category tags, rejecting the null hypothesis that the 2 reviewers could have independently arrived at their similar category tags by chance. Further assessment of inter‐rater agreement was generally excellent, with >0.8 in the majority of cases. The remaining cases (ie, Hgb 78, per other MD, anemia, cardiac) still had substantial agreement, with >0.6. These disagreements almost universally occurred when a reviewer assigned a subset of the category tags assigned by the other reviewer. For example, 1 reviewer may tag a comment bone marrow transplant (BMT) patient with Hgb 7.2 with BMT and Hgb 78, whereas the other would only cite BMT.

Figure 2 and Figure 3 highlight the varied categories of responses, with most categories comprising <4% of the available responses. Among the most common override reasons are structured protocols for specialty services, as in the stem cell transplant service, whose protocol recommends transfusing blood when Hgb <8 rather than Hgb <7. In these cases, the BPA is unlikely to change protocolized behavior, thus overall workflow would likely be improved by adding a structured BPA override option for these protocols.

Analysis of the override comments did expose some relatively common and questionable transfusion practices, including perioperative and periprocedural transfusions, as well as anticipation of imminent discharge. Prior studies indicate that transfusions in anticipation of surgical or procedural intervention provide no benefit compared to responding to blood requirements intraoperatively as needed,[3, 22, 23] and imminent discharge of a patient is not a well‐recognized reason to transfuse outside of guidelines. The identification of these questionable and relatively common practices identifies opportunities for targeted education and training campaigns.

Symptomatic anemia was 1 of the more commonly cited BPA override reasons with a fraction providing a specific symptom such as fatigue, lightheadedness, or shortness of breath. Although the pervasiveness of this rationale may speak to adding it as a structured BPA override option, the nonspecific, subjective, and nonevidence‐based nature of symptom‐driven blood transfusions suggests that ongoing interruptive BPA prompts can still be useful to remind providers of the risks and guideline‐based approaches to such cases.

Limitations of the analysis are revealed as a fraction of BPA events did not fully record all relevant data, preventing proper analysis. Override comments suggest the BPA was triggering inappropriately for patients appropriately below the recommended transfusion threshold of Hgb <7, assuming provider free‐textentered values were accurate. Review of these cases showed some variability, such as when providers based their transfusion decision on a hematocrit of 20 rather than a Hgb of 7. Many comments also stated nonthreshold Hgb values, such as Hgb 7.2, seeming to imply that the value was close enough to the recommended threshold to justify overriding the transfusion prompt.

The most significant limitation of this study is the substantial fraction of nonresponder BPA transfusion overrides with a nonspecific other reason and no text commentary, comprising 34% (3670/10,642) of all BPA interactions and 62% (3670/5886) of all other overrides. Although the BPA is easily overridden by design to avoid workflow disruption that could compromise the priority of patient care, the nonresponses raise concern for skewed interpretation of the override data. General studies in survey responses provide reassurance that lower response rates do not necessarily indicate response bias,[24] with response rates as low as 25% yielding results statistically indistinguishable from more rigorous methods achieving >50% response rates.[25] In this specific case, response bias is better characterized by comparing ordering provider characteristics for the other overrides with and without free‐text comments. Specifically, Figure 4 shows the distribution of other overrides by provider type (job title) and provider home department (where available from physician department registries), separated by whether a free‐text comment was left. For each sub‐group, a ² analysis compared the observed versus expected proportion of providers leaving comments based on the null hypothesis that leaving a comment was independent of membership in the subgroup. Similar proportions with nonsignificant P values suggest against significant response bias for most subgroups, but the data do indicate that this work likely under‐represents the opinion of fellows, physician assistants, and neurosurgeons, while slightly over‐representing the opinion of medicine, general surgery, and obstetrics/gynecology practitioners. Given that the under‐represented groups overall constitute a small minority of the total BPA interactions, this work should still be generalizable to the majority of transfusion behavior.

Figure 4

Further review of the ordering provider type (job title) distribution indicates that the vast majority of providers who actually interact with BPAs in this academic hospital are resident physicians. Similarly notable is that 25% of BPAs trigger for registered nurses. Upon review of hospital practices, we confirmed that nurses can enter transfusion orders on behalf of MD cosigners, generally in the context of a verbal order or routine treatment protocol, most notably on the stem cell transplant service. The prevalence of resident, nurse, nurse practitioner, and physician assistant order entry overall indicates that the providers encountering the BPA often do not have the negotiating power to accept or override the prompts, specifically defying the meaningful use goal of decision‐making providers directly interacting with CPOE and decision support prompts.[26]

The theoretical intelligence of the BPA not to trigger in well‐accepted cases of transfusion, where acute bleeding is occurring based on the presence of bleeding related items in the patient's problem list, is demonstrated to be limited. Acute bleeding represented the most common structured override response. This is likely multifactorial, but is largely the result of inconsistent usage of the patient problem lists that the BPA would depend upon. Another commonly cited override explanation was for a dropping Hgb without a specific bleeding source. This could point toward the BPA triggering an algorithm incorporating the last several Hgb values to assess the trajectory. Even then, however, guidelines would advocate holding transfusion and proceeding with serial monitoring of Hgb levels until an accepted transfusion threshold was actually breached.

This analysis demonstrates a general approach to secondary analysis of EMR data, in this case to provide insight into the specific question of why providers continue to order costly, unnecessary, and potentially harmful blood transfusions in spite of an interruptive BPA CDS tool. Limitations of the BPA are now better understood, including technical aspects of accounting for all decision‐making data, the frequency of nonresponse for override rationale, and that the providers interacting with the BPAs often do not have the negotiating power to change ordering behavior. The analysis recognizes protocolized behaviors that should be integrated into the CDS algorithms, and specification of symptomatic anemia as a common albeit subjective indication for blood transfusion. A wide swath of individually uncommon rationales drive transfusion override behavior, motivating further EMR‐based analysis to more efficiently recognize common scenarios (eg, perioperative/procedural, imminent discharge) for directed education and training interventions than can be achieved by conventional chart review.

Transfusion of blood products has multiple clinical applications, yet when used outside of recommended guidelines it is associated with increased cost, waste, morbidity, and mortality.[1] Studies reviewing restrictive versus liberal blood transfusion strategies have demonstrated no benefit to liberal strategies over restrictive strategies[2, 3, 4, 5, 6, 7, 8, 9, 10] and possibly even harm.[11] Given the risks and real costs of blood transfusions occurring outside of recommended guidelines, professional societies in hematology, anesthesiology, and hospital medicine each include excessive blood transfusions among their top 5 questionable clinical practices in the American Board of Internal Medicine Foundation's Choosing Wisely initiative.[12] To modify behaviors around blood transfusion practices, hospitals and blood banks may need to provide clinical decision support (CDS) for physicians.

The conventional approach to CDS is direct education and training campaigns, but there is a unique opportunity for intelligent decision support at the point of care through electronic medical record (EMR) systems directly integrated with computerized physician order entry (CPOE).[13, 14, 15] Prior work at Stanford toward reducing unnecessary blood transfusions started with hospital‐wide education campaigns, which brought down the percentage of transfusions ordered for patients with hemoglobin (Hgb) >8 g/dL from 57% to 52%. Further reduction to <30% was achieved after the introduction of an interruptive best practice alert (BPA) integrated into the CPOE transfusion ordering process.[16, 17] Specifically, providers attempting to order a blood transfusion for patients with an EMR‐determined Hgb >7 are presented with a BPA popup reminding them of best practice guidelines and a prompt to either abort the transfusion or provide a reason to override the BPA and proceed.

It remains uncertain why up to 30% of transfusions continue to occur outside of recommended guidelines in spite of interruptive prompts. This study demonstrates a general approach to secondary use of clinical data from the EMR toward understanding provider behavior, specifically by analyzing free‐text comments linked to transfusion override behavior and identifying the type of providers interacting with the BPAs.

MATERIALS AND METHODS

At Stanford University Hospital, a 447‐bed academic tertiary care center servicing adult patients, clinicians order blood transfusions through an EMR+CPOE system. When an order for red blood cell transfusion is attempted, the EMR evaluates the patient chart for specific criteria based on previously published guidelines.[18] Specifically, the BPA will review the last recorded Hgb value and trigger if the Hgb is >8, or if the Hgb is >7 and there is no concurrent EMR problem‐list entry related to acute coronary syndrome or acute hemorrhage. Once the blood transfusion BPA triggers, the ordering provider is presented with an interruptive prompt reminding them of best practice guidelines and the 3 most recent Hgb values for the patient.[16] From there, the provider may either abort the transfusion or override the BPA and proceed. Overrides require the provider to select a reason from a predefined list of institutionally accepted transfusion indications including acute bleeding, acute coronary syndrome and Hgb <8, and postoperative cardiothoracic surgery and Hgb <8. If none of the predefined override reasons are selected, the provider simply selects other, with the option of a free‐text comment to elaborate their rationale.

Data from provider interactions with the BPA were collected from the EMR across all inpatient wards for 8 months after the implementation period. Data collected from each interaction included the patient identifier, alert description, action taken, ordering provider identifier, ordering provider type (job title), optional comments for overriding the BPA, and date and time.

The free‐text override comments were independently reviewed by 2 licensed physicians, tagging them into a set of general categories by iterative inductive analysis of the comment content. Individual comments were allowed multiple possible category tags (eg, coronary artery disease and symptomatic). The initial tagging process was expedited by isolating common keywords in the override comments and assigning likely category tags to each, as in Table 2. The 2 physician reviewers then manually inspected all comments with the option to revise the initial category tagging to ensure validity. Many category tags occurred sporadically and were aggregated into more general categories, such as hematologic disorder (includes myelodysplastic syndrome, myelofibrosis, multiple myeloma, sickle cell, thalassemia, Waldenstrom's), symptomatic (includes fatigue, lightheaded, short of breath), and per other medical doctor [MD] (includes any other physician direction, primarily specialty consultation services).

To assess the inter‐rater agreement of this category tagging between the 2 reviewers, a confusion matrix similar to the example in Table 1 was setup for each tag used. Several agreement statistics are calculated based on the confusion matrix, including the positive agreement rate (P_a+) and Cohen's kappa statistic (). Kappa statistic values range from 1 to +1, with values <0 indicating no agreement and values >0.8 indicating near perfect agreement.[19] To reject the null hypothesis that the 2 reviewers could have independently arrived at their similar tagging assignments by chance, a ² test was applied for each confusion matrix, with Yates' correction to avoid overestimating statistical significance given the low rates of inter‐rater disagreement.[20]

RESULTS

During the data collection period from January 1, 2011 to August 31, 2012, the blood transfusion BPA fired in 11,791 instances, of which 10,642 recorded sufficient data for analysis (Figure 1). The ordering provider proceeded to override the BPA and continued with transfusion in 98% of cases (10,442/10,642). Acute bleeding" was the most common structured response (34%). The majority of BPA overrides used the general purpose other structured response option accounting for 56% (5886/10,442) of override responses, of which 37% (2185/5886) entered a free‐text comment elaborating the override reason. With 3701 nonresponders (other overrides with no free‐text comment), the overall response rate was 65% (6941/10,642).

Figure 1

With a handful of free‐text comments included with structured override responses (eg, 28 acute bleeding overrides included additional comment from the provider), a total of 2216 override comments were available for analysis. Using an initial selection of keyword‐tag associations, as in Table 2, 95% (2104/2216) of the override comments had a preliminary category tagging assigned. After manual review and revision by the first physician reviewer, 74% (1633/2216) of the comments retained their automated tags, whereas 26% (583/2216) were updated based on the reviewer's assessment of validity. This included 112 comments lacking automated tags the reviewer manually added, as well as 471 comments with automated tags revised by the reviewer.

The most common override category tags are presented in Figure 2 and Figure 3 for BPAs triggered in response to blood transfusions ordered for patients with Hgb 78 and Hgb >8, respectively. The agreed+ percentages correspond to the number of comments where the 2 physician reviewers both assigned the respective category tag (P_a+), whereas the disagreed percentages correspond to comments where the reviewers differed (P_d=1‐P_a). By Yates' corrected ² analysis, P was <10¹⁰ for independence between reviewer tag assignments for all tags assessed. Kappa statistics are reported in the figures to describe inter‐rater agreement.

Figure 2

Figure 3

Resident physicians were the primary ordering provider group, accounting for approximately 55% (5863/10,642) of BPA interactions, followed by registered nurses, fellows, and attending physicians.

DISCUSSION

Prior work has established that an interruptive BPA to provide decision support against inappropriate blood transfusions can drive a significant and sustained reduction in unnecessary transfusions,[16] especially when satisfying the primary features of effective decision support.[14] The quantity of transfusions directly aborted by the BPA (only 2% in this case) does not nearly account for the total observed reductions in transfusions, suggesting that the BPA yields an indirect teaching effect over time beyond direct interruption. In other words, once a user has been interrupted by the BPA several times, they will be discouraged from even attempting to order inappropriate blood transfusions in the future.

Despite the improvements above, there remains a substantial fraction (up to 30%) of blood transfusions occurring outside of recommended guidelines where providers specifically override the BPA prompt.[21] This work is the first structured analysis of such BPA override behavior by generalizable methods of EMR data analysis to guide further improvements.

Analysis of the override behavior focused primarily on the free‐text comments explaining provider rationales for overriding the BPA. These comments were categorized by 2 physician reviewers, with P<10¹⁰ for all category tags, rejecting the null hypothesis that the 2 reviewers could have independently arrived at their similar category tags by chance. Further assessment of inter‐rater agreement was generally excellent, with >0.8 in the majority of cases. The remaining cases (ie, Hgb 78, per other MD, anemia, cardiac) still had substantial agreement, with >0.6. These disagreements almost universally occurred when a reviewer assigned a subset of the category tags assigned by the other reviewer. For example, 1 reviewer may tag a comment bone marrow transplant (BMT) patient with Hgb 7.2 with BMT and Hgb 78, whereas the other would only cite BMT.

Figure 2 and Figure 3 highlight the varied categories of responses, with most categories comprising <4% of the available responses. Among the most common override reasons are structured protocols for specialty services, as in the stem cell transplant service, whose protocol recommends transfusing blood when Hgb <8 rather than Hgb <7. In these cases, the BPA is unlikely to change protocolized behavior, thus overall workflow would likely be improved by adding a structured BPA override option for these protocols.

Analysis of the override comments did expose some relatively common and questionable transfusion practices, including perioperative and periprocedural transfusions, as well as anticipation of imminent discharge. Prior studies indicate that transfusions in anticipation of surgical or procedural intervention provide no benefit compared to responding to blood requirements intraoperatively as needed,[3, 22, 23] and imminent discharge of a patient is not a well‐recognized reason to transfuse outside of guidelines. The identification of these questionable and relatively common practices identifies opportunities for targeted education and training campaigns.

Symptomatic anemia was 1 of the more commonly cited BPA override reasons with a fraction providing a specific symptom such as fatigue, lightheadedness, or shortness of breath. Although the pervasiveness of this rationale may speak to adding it as a structured BPA override option, the nonspecific, subjective, and nonevidence‐based nature of symptom‐driven blood transfusions suggests that ongoing interruptive BPA prompts can still be useful to remind providers of the risks and guideline‐based approaches to such cases.

Limitations of the analysis are revealed as a fraction of BPA events did not fully record all relevant data, preventing proper analysis. Override comments suggest the BPA was triggering inappropriately for patients appropriately below the recommended transfusion threshold of Hgb <7, assuming provider free‐textentered values were accurate. Review of these cases showed some variability, such as when providers based their transfusion decision on a hematocrit of 20 rather than a Hgb of 7. Many comments also stated nonthreshold Hgb values, such as Hgb 7.2, seeming to imply that the value was close enough to the recommended threshold to justify overriding the transfusion prompt.

The most significant limitation of this study is the substantial fraction of nonresponder BPA transfusion overrides with a nonspecific other reason and no text commentary, comprising 34% (3670/10,642) of all BPA interactions and 62% (3670/5886) of all other overrides. Although the BPA is easily overridden by design to avoid workflow disruption that could compromise the priority of patient care, the nonresponses raise concern for skewed interpretation of the override data. General studies in survey responses provide reassurance that lower response rates do not necessarily indicate response bias,[24] with response rates as low as 25% yielding results statistically indistinguishable from more rigorous methods achieving >50% response rates.[25] In this specific case, response bias is better characterized by comparing ordering provider characteristics for the other overrides with and without free‐text comments. Specifically, Figure 4 shows the distribution of other overrides by provider type (job title) and provider home department (where available from physician department registries), separated by whether a free‐text comment was left. For each sub‐group, a ² analysis compared the observed versus expected proportion of providers leaving comments based on the null hypothesis that leaving a comment was independent of membership in the subgroup. Similar proportions with nonsignificant P values suggest against significant response bias for most subgroups, but the data do indicate that this work likely under‐represents the opinion of fellows, physician assistants, and neurosurgeons, while slightly over‐representing the opinion of medicine, general surgery, and obstetrics/gynecology practitioners. Given that the under‐represented groups overall constitute a small minority of the total BPA interactions, this work should still be generalizable to the majority of transfusion behavior.

Figure 4

Further review of the ordering provider type (job title) distribution indicates that the vast majority of providers who actually interact with BPAs in this academic hospital are resident physicians. Similarly notable is that 25% of BPAs trigger for registered nurses. Upon review of hospital practices, we confirmed that nurses can enter transfusion orders on behalf of MD cosigners, generally in the context of a verbal order or routine treatment protocol, most notably on the stem cell transplant service. The prevalence of resident, nurse, nurse practitioner, and physician assistant order entry overall indicates that the providers encountering the BPA often do not have the negotiating power to accept or override the prompts, specifically defying the meaningful use goal of decision‐making providers directly interacting with CPOE and decision support prompts.[26]

The theoretical intelligence of the BPA not to trigger in well‐accepted cases of transfusion, where acute bleeding is occurring based on the presence of bleeding related items in the patient's problem list, is demonstrated to be limited. Acute bleeding represented the most common structured override response. This is likely multifactorial, but is largely the result of inconsistent usage of the patient problem lists that the BPA would depend upon. Another commonly cited override explanation was for a dropping Hgb without a specific bleeding source. This could point toward the BPA triggering an algorithm incorporating the last several Hgb values to assess the trajectory. Even then, however, guidelines would advocate holding transfusion and proceeding with serial monitoring of Hgb levels until an accepted transfusion threshold was actually breached.

This analysis demonstrates a general approach to secondary analysis of EMR data, in this case to provide insight into the specific question of why providers continue to order costly, unnecessary, and potentially harmful blood transfusions in spite of an interruptive BPA CDS tool. Limitations of the BPA are now better understood, including technical aspects of accounting for all decision‐making data, the frequency of nonresponse for override rationale, and that the providers interacting with the BPAs often do not have the negotiating power to change ordering behavior. The analysis recognizes protocolized behaviors that should be integrated into the CDS algorithms, and specification of symptomatic anemia as a common albeit subjective indication for blood transfusion. A wide swath of individually uncommon rationales drive transfusion override behavior, motivating further EMR‐based analysis to more efficiently recognize common scenarios (eg, perioperative/procedural, imminent discharge) for directed education and training interventions than can be achieved by conventional chart review.