Craig A. Umscheid, MD, MSCE

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

Hospital evidence‐based practice centers (EPCs) are structures with the potential to facilitate the integration of evidence into institutional decision making to close knowing‐doing gaps[1, 2, 3, 4, 5, 6]; in the process, they can support the evolution of their parent institutions into learning healthcare systems.[7] The potential of hospital EPCs stems from their ability to identify and adapt national evidence‐based guidelines and systematic reviews for the local setting,[8] create local evidence‐based guidelines in the absence of national guidelines, use local data to help define problems and assess the impact of solutions,[9] and implement evidence into practice through computerized clinical decision support (CDS) interventions and other quality‐improvement (QI) initiatives.[9, 10] As such, hospital EPCs have the potential to strengthen relationships and understanding between clinicians and administrators[11]; foster a culture of evidence‐based practice; and improve the quality, safety, and value of care provided.[10]

Formal hospital EPCs remain uncommon in the United States,[10, 11, 12] though their numbers have expanded worldwide.[13, 14] This growth is due not to any reduced role for national EPCs, such as the National Institute for Health and Clinical Excellence[15] in the United Kingdom, or the 13 EPCs funded by the Agency for Healthcare Research and Quality (AHRQ)[16, 17] in the United States. Rather, this growth is fueled by the heightened awareness that the value of healthcare interventions often needs to be assessed locally, and that clinical guidelines that consider local context have a greater potential to improve quality and efficiency.[9, 18, 19]

Despite the increasing number of hospital EPCs globally, their impact on administrative and clinical decision making has rarely been examined,[13, 20] especially for hospital EPCs in the United States. The few studies that have assessed the impact of hospital EPCs on institutional decision making have done so in the context of technology acquisition, neglecting the role hospital EPCs may play in the integration of evidence into clinical practice. For example, the Technology Assessment Unit at McGill University Health Center found that of the 27 reviews commissioned in their first 5 years, 25 were implemented, with 6 (24%) recommending investments in new technologies and 19 (76%) recommending rejection, for a reported net hospital savings of $10 million.[21] Understanding the activities and impact of hospital EPCs is particularly critical for hospitalist leaders, who could leverage hospital EPCs to inform efforts to support the quality, safety, and value of care provided, or who may choose to establish or lead such infrastructure. The availability of such opportunities could also support hospitalist recruitment and retention.

In 2006, the University of Pennsylvania Health System (UPHS) created the Center for Evidence‐based Practice (CEP) to support the integration of evidence into practice to strengthen quality, safety, and value.[10] Cofounded by hospitalists with formal training in clinical epidemiology, the CEP performs rapid systematic reviews of the scientific literature to inform local practice and policy. In this article, we describe the first 8 years of the CEP's evidence synthesis activities and examine its impact on decision making across the health system.

METHODS

RESULTS

Evidence Synthesis Impact

A total of 139 reports were completed between July 2010 and June 2014 for 65 individual requestors. Email invitations to participate in the survey were sent to the 64 requestors employed by the UPHS. The response rate was 72% (46/64). The proportions of physician requestors and traditional HTA topics evaluated were similar across respondents and nonrespondents (43% [20/46] vs 39% [7/18], P = 0.74; and 37% [17/46] vs 44% [8/18], P = 0.58, respectively). Aggregated survey responses are presented for items using a Likert scale in Figure 1, and for items using a yes/no or ordinal scale in Table 3.

Table 3.Responses to Yes/No and Ranking Survey Questions

Items	% of Respondents Responding Affirmatively
	Percentage of Respondents Ranking as First Choice*
NOTE: Abbreviations: CEP, Center for Evidence‐based Practice. *The sum of these percentages is greater than 100 percent because respondents could rank multiple options first.
Requestor activity
What factors prompted you to request a report from CEP? (Please select all that apply.)
My own time constraints	28% (13/46)
CEP's ability to identify and synthesize evidence	89% (41/46)
CEP's objectivity	52% (24/46)
Recommendation from colleague	30% (14/46)
Did you conduct any of your own literature searches before contacting CEP?	67% (31/46)
Did you obtain and read any of the articles cited in CEP's report?	63% (29/46)
Did you read the following sections of CEP's report?
Evidence summary (at beginning of report)	100% (45/45)
Introduction/background	93% (42/45)
Methods	84% (38/45)
Results	98% (43/43)
Conclusion	100% (43/43)
Report dissemination
Did you share CEP's report with anyone NOT involved in requesting the report or in making the final decision?	67% (30/45)
Did you share CEP's report with anyone outside of Penn?	7% (3/45)
Requestor preferences
Would it be helpful for CEP staff to call you after you receive any future CEP reports to answer any questions you might have?	55% (24/44)
Following any future reports you request from CEP, would you be willing to complete a brief questionnaire?	100% (44/44)
Please rank how you would prefer to receive reports from CEP in the future.
E‐mail containing the report as a PDF attachment	77% (34/44)
E‐mail containing a link to the report on CEP's website	16% (7/44)
In‐person presentation by the CEP analyst writing the report	18% (8/44)
In‐person presentation by the CEP director involved in the report	16% (7/44)

Figure 1

In general, respondents found reports easy to request, easy to use, timely, and relevant, resulting in high requestor satisfaction. In addition, 98% described the scope of content and level of detail as about right. Report impact was rated highly as well, with the evidence summary and conclusions rated as the most critical to decision making. A majority of respondents indicated that reports confirmed their tentative decision (77%, n = 34), whereas some changed their tentative decision (7%, n = 3), and others suggested the report had no effect on their tentative decision (16%, n = 7). Respondents indicated the amount of time that elapsed between receiving reports and making final decisions was 1 to 7 days (5%, n = 2), 8 to 30 days (40%, n = 17), 1 to 3 months (37%, n = 16), 4 to 6 months (9%, n = 4), or greater than 6 months (9%, n = 4). The most common reasons cited for requesting a report were the CEP's evidence synthesis skills and objectivity.

DISCUSSION

To our knowledge, this is the first comprehensive description and assessment of evidence synthesis activity by a hospital EPC in the United States. Our findings suggest that clinical and administrative leaders will request reports from a hospital EPC, and that hospital EPCs can promptly produce reports when requested. Moreover, these syntheses can address a wide range of clinical and policy topics, and can be disseminated through a variety of routes. Lastly, requestors are satisfied by these syntheses, and report that they inform decision making. These results suggest that EPCs may be an effective infrastructure paradigm for promoting evidence‐based decision making within healthcare provider organizations, and are consistent with previous analyses of hospital‐based EPCs.[21, 28, 29]

Over half of report requestors cited CEP's objectivity as a factor in their decision to request a report, underscoring the value of a neutral entity in an environment where clinical departments and hospital committees may have competing interests.[10] This asset was 1 of the primary drivers for establishing our hospital EPC. Concerns by clinical executives about the influence of industry and local politics on institutional decision making, and a desire to have clinical evidence more systematically and objectively integrated into decision making, fueled our center's funding.

The survey results also demonstrate that respondents were satisfied with the reports for many reasons, including readability, concision, timeliness, scope, and content, consistent with the evaluation of the French hospital‐based EPC CEDIT (French Committee for the Assessment and Dissemination of Technological Innovations).[29] Given the importance of readability, concision, and relevance that has been previously described,[16, 28, 30] nearly all CEP reports contain an evidence summary on the first page that highlights key findings in a concise, user‐friendly format.[31] The evidence summaries include bullet points that: (1) reference the most pertinent guideline recommendations along with their strength of recommendation and underlying quality of evidence; (2) organize and summarize study findings using the most critical clinical outcomes, including an assessment of the quality of the underlying evidence for each outcome; and (3) note important limitations of the findings.

Evidence syntheses must be timely to allow decision makers to act on the findings.[28, 32] The primary criticism of CEDIT was the lag between requests and report publication.[29] Rapid reviews, designed to inform urgent decisions, can overcome this challenge.[31, 33, 34] CEP reviews required approximately 2 months to complete on average, consistent with the most rapid timelines reported,[31, 33, 34] and much shorter than standard systematic review timelines, which can take up to 12 to 24 months.[33] Working with requestors to limit the scope of reviews to those issues most critical to a decision, using secondary resources when available, and hiring experienced research analysts help achieve these efficiencies.

The study by Bodeau‐Livinec also argues for the importance of report accessibility to ensure dissemination.[29] This is consistent with the CEP's approach, where all reports are posted on the UPHS internal website. Many also inform QI initiatives, as well as CDS interventions that address topics of general interest to acute care hospitals, such as venous thromboembolism (VTE) prophylaxis,[35] blood product transfusions,[36] sepsis care,[37, 38] and prevention of catheter‐associated urinary tract infections (CAUTI)[39] and hospital readmissions.[40] Most reports are also listed in an international database of rapid reviews,[23] and reports that address topics of general interest, have sufficient evidence to synthesize, and have no prior published systematic reviews are published in the peer‐reviewed literature.[41, 42]

The majority of reports completed by the CEP were evidence reviews, or systematic reviews of primary literature, suggesting that CEP reports often address questions previously unanswered by existing published systematic reviews; however, about a third of reports were evidence advisories, or summaries of evidence from preexisting secondary sources. The relative scarcity of high‐quality evidence bases in those reports where GRADE analyses were conducted might be expected, as requestors may be more likely to seek guidance when the evidence base on a topic is lacking. This was further supported by the small percentage (15%) of reports where adequate data of sufficient homogeneity existed to allow meta‐analyses. The small number of original meta‐analyses performed also reflects our reliance on secondary resources when available.

Only 7% of respondents reported that tentative decisions were changed based on their report. This is not surprising, as evidence reviews infrequently result in clear go or no go recommendations. More commonly, they address or inform complex clinical questions or pathways. In this context, the change/confirm/no effect framework may not completely reflect respondents' use of or benefit from reports. Thus, we included a diverse set of questions in our survey to best estimate the value of our reports. For example, when asked whether the report answered the question posed, informed their final decision, or was consistent with their final decision, 91%, 79%, and 71% agreed or strongly agreed, respectively. When asked whether they would request a report again if they had to do it all over, recommend CEP to their colleagues, and be likely to request reports in the future, at least 95% of survey respondents agreed or strongly agreed. In addition, no respondent indicated that their report was not timely enough to influence their decision. Moreover, only a minority of respondents expressed disappointment that the CEP's report did not provide actionable recommendations due to a lack of published evidence (9%, n = 4). Importantly, the large proportion of requestors indicating that reports confirmed their tentative decisions may be a reflection of hindsight bias.

The most apparent trend in the production of CEP reviews over time is the relative increase in requests by clinical departments, suggesting that the CEP is being increasingly consulted to help define best clinical practices. This is also supported by the relative increase in reports focused on policy or organizational/managerial systems. These findings suggest that hospital EPCs have value beyond the traditional realm of HTA.

This study has a number of limitations. First, not all of the eligible report requestors responded to our survey. Despite this, our response rate of 72% compares favorably with surveys published in medical journals.[43] In addition, nonresponse bias may be less important in physician surveys than surveys of the general population.[44] The similarity in requestor and report characteristics for respondents and nonrespondents supports this. Second, our survey of impact is self‐reported rather than an evaluation of actual decision making or patient outcomes. Thus, the survey relies on the accuracy of the responses. Third, recall bias must be considered, as some respondents were asked to evaluate reports that were greater than 1 year old. To reduce this bias, we asked respondents to consider the most recent report they requested, included that report as an attachment in the survey request, and only surveyed requestors from the most recent 4 of the CEP's 8 fiscal years. Fourth, social desirability bias could have also affected the survey responses, though it was likely minimized by the promise of confidentiality. Fifth, an examination of the impact of the CEP on costs was outside the scope of this evaluation; however, such information may be important to those assessing the sustainability or return on investment of such centers. Simple approaches we have previously used to approximate the value of our activities include: (1) estimating hospital cost savings resulting from decisions supported by our reports, such as the use of technologies like chlorhexidine for surgical site infections[45] or discontinuation of technologies like aprotinin for cardiac surgery[46]; and (2) estimating penalties avoided or rewards attained as a result of center‐led initiatives, such as those to increase VTE prophylaxis,[35] reduce CAUTI rates,[39] and reduce preventable mortality associated with sepsis.[37, 38] Similarly, given the focus of this study on the local evidence synthesis activities of our center, our examination did not include a detailed description of our CDS activities, or teaching activities, including our multidisciplinary workshops for physicians and nurses in evidence‐based QI[47] and our novel evidence‐based practice curriculum for medical students. Our study also did not include a description of our extramural activities, such as those supported by our contract with AHRQ as 1 of their 13 EPCs.[16, 17, 48, 49] A consideration of all of these activities enables a greater appreciation for the potential of such centers. Lastly, we examined a single EPC, which may not be representative of the diversity of hospitals and hospital staff across the United States. However, our EPC serves a diverse array of patient populations, clinical services, and service models throughout our multientity academic healthcare system, which may improve the generalizability of our experience to other settings.

As next steps, we recommend evaluation of other existing hospital EPCs nationally. Such studies could help hospitals and health systems ascertain which of their internal decisions might benefit from locally sourced rapid systematic reviews and determine whether an in‐house EPC could improve the value of care delivered.

In conclusion, our findings suggest that hospital EPCs within academic healthcare systems can efficiently synthesize and disseminate evidence for a variety of stakeholders. Moreover, these syntheses impact decision making in a variety of hospital contexts and clinical specialties. Hospitals and hospitalist leaders seeking to improve the implementation of evidence‐based practice at a systems level might consider establishing such infrastructure locally.

There are as many as 3 million cases of severe sepsis and 750,000 resulting deaths in the United States annually.[1] Interventions such as goal‐directed resuscitation and antibiotics can reduce sepsis mortality, but their effectiveness depends on early administration. Thus, timely recognition is critical.[2, 3, 4, 5]

Despite this, early recognition in hospitalized patients can be challenging. Using chart documentation as a surrogate for provider recognition, we recently found only 20% of patients with severe sepsis admitted to our hospital from the emergency department were recognized.[6] Given these challenges, there has been increasing interest in developing automated systems to improve the timeliness of sepsis detection.[7, 8, 9, 10] Systems described in the literature have varied considerably in triggering criteria, effector responses, and study settings. Of those examining the impact of automated surveillance and response in the nonintensive care unit (ICU) acute inpatient setting, results suggest an increase in the timeliness of diagnostic and therapeutic interventions,[10] but less impact on patient outcomes.[7] Whether these results reflect inadequacies in the criteria used to identify patients (parameters or their thresholds) or an ineffective response to the alert (magnitude or timeliness) is unclear.

Given the consequences of severe sepsis in hospitalized patients, as well as the introduction of vital sign (VS) and provider data in our electronic health record (EHR), we sought to develop and implement an electronic sepsis detection and response system to improve patient outcomes. This study describes the development, validation, and impact of that system.

METHODS

The Postimplementation (Live) Period and Impact Analysis

The EWRS went live September 12, 2012, upon which new admissions triggering the alert would result in a notification and response. Unadjusted analyses using the [2] test for dichotomous variables and the Wilcoxon rank sum test for continuous variables compared demographics and the proportion of clinical process and outcome measures for those admitted during the silent period (June 6, 2012September 4, 2012) and a similar timeframe 1 year later when the intervention was live (June 6, 2013September 4, 2013). To be included in either of the time periods, patients had to trigger the alert during the period and be discharged within 45 days of the end of the period. The pre‐ and post‐sepsis mortality index was also examined (see the Supporting Information in the online version of this article for a detailed description of study measures). Multivariable regression models estimated the impact of the EWRS on the process and outcome measures, adjusted for differences between the patients in the preimplementation and postimplementation periods with respect to age, gender, Charlson index on admission, admitting service, hospital, and admission month. Logistic regression models examined dichotomous variables. Continuous variables were log transformed and examined using linear regression models. Cox regression models explored time to ICU transfer from trigger. Among patients with sepsis, a logistic regression model was used to compare the odds of mortality between the silent and live periods, adjusted for expected mortality, both within each hospital and across all hospitals.

Because there is a risk of providers becoming overly reliant on automated systems and overlooking those not triggering the system, we also examined the discharge disposition and mortality outcomes of those in both study periods not identified by the EWRS.

The primary analysis examined the impact of the EWRS across UPHS; we also examined the EWRS impact at each of our hospitals. Last, we performed subgroup analyses examining the EWRS impact in those assigned an International Classification of Diseases, 9th Revision code for sepsis at discharge or death. All analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC).

RESULTS

In the derivation cohort, 4575 patients met the inclusion criteria. The proportion of those in each category (06) achieving our outcomes of interest are described in Supporting Table 1 in the online version of this article. We defined a positive trigger as a score 4, as this threshold identified a limited number of patients (3.9% [180/4575]) with a high proportion experiencing our composite outcome (25.6% [46/180]). The proportion of patients with an EWRS score 4 and their time to event by hospital and health system is described in Supporting Table 2 in the online version of this article. Those with a score 4 were almost 4 times as likely to be transferred to the ICU, almost 7 times as likely to experience an RRT, and almost 10 times as likely to die. The screen positive, sensitivity, specificity, and positive and negative predictive values and likelihood ratios using this threshold and our composite outcome in the derivation cohort were 6%, 16%, 97%, 26%, 94%, 5.3, and 0.9, respectively, and in our validation cohort were 6%, 17%, 97%, 28%, 95%, 5.7, and 0.9, respectively.

In the preimplementation period, 3.8% of admissions (595/15,567) triggered the alert, as compared to 3.5% (545/15,526) in the postimplementation period. Demographics were similar across periods, except that in the postimplementation period patients were slightly younger and had a lower Charlson Comorbidity Index at admission (Table 1). The distribution of alerts across medicine and surgery services were similar (Table 1).

In our postimplementation period, 99% of coordinator pages and over three‐fourths of provider notifications were sent successfully. Almost three‐fourths of nurses reviewed the initial alert notification, and over 99% completed the electronic data verification and adverse trend review, with over half documenting adverse trends. Ninety‐five percent of the time the coordinators completed the follow‐up assessment. Over 90% of the time, the entire team evaluated the patient at bedside within 30 minutes. Almost half of the time, the team thought the patient had no critical illness. Over a third of the time, they thought the patient had sepsis, but reported over 90% of the time that they were aware of the diagnosis prior to the alert. (Supporting Table 3 in the online version of this article includes more details about the responses to the electronic notifications and follow‐up assessments.)

In unadjusted and adjusted analyses, ordering of antibiotics, intravenous fluid boluses, and lactate and blood cultures within 3 hours of the trigger increased significantly, as did ordering of blood products, chest radiographs, and cardiac monitoring within 6 hours of the trigger (Tables 2 and 3).

Hospital and ICU length of stay were similar in the preimplementation and postimplementation periods. There was no difference in the proportion of patients transferred to the ICU following the alert; however, the proportion transferred within 6 hours of the alert increased, and the time to ICU transfer was halved (see Supporting Figure 4 in the online version of this article), but neither change was statistically significant in unadjusted analyses. Transfer to the ICU within 6 hours became statistically significant after adjustment. All mortality measures were lower in the postimplementation period, but none reached statistical significance. Discharge to home and sepsis documentation were both statistically higher in the postimplementation period, but discharge to home lost statistical significance after adjustment (Tables 4 and 5) (see Supporting Table 4 in the online version of this article).

In a subanalysis of EWRS impact on patients documented with sepsis at discharge, unadjusted and adjusted changes in clinical process and outcome measures across the time periods were similar to that of the total population (see Supporting Tables 5 and 6 and Supporting Figure 5 in the online version of this article). The unadjusted composite outcome of mortality or inpatient hospice was statistically lower in the postimplementation period, but lost statistical significance after adjustment.

The disposition and mortality outcomes of those not triggering the alert were unchanged across the 2 periods (see Supporting Tables 7, 8, and 9 in the online version of this article).

DISCUSSION

This study demonstrated that a predictive tool can accurately identify non‐ICU inpatients at increased risk for deterioration and death. In addition, we demonstrated the feasibility of deploying our EHR to screen patients in real time for deterioration and to trigger electronically a timely, robust, multidisciplinary bedside clinical evaluation. Compared to the control (silent) period, the EWRS resulted in a marked increase in early sepsis care, transfer to the ICU, and sepsis documentation, and an indication of a decreased sepsis mortality index and mortality, and increased discharge to home, although none of these latter 3 findings reached statistical significance.

Our study is unique in that it was implemented across a multihospital health system, which has identical EHRs, but diverse cultures, populations, staffing, and practice models. In addition, our study includes a preimplementation population similar to the postimplementation population (in terms of setting, month of admission, and adjustment for potential confounders).

Interestingly, patients identified by the EWRS who were subsequently transferred to an ICU had higher mortality rates (30% and 26% in the preimplementation and postimplementation periods, respectively, across UPHS) than those transferred to an ICU who were not identified by the EWRS (7% and 6% in the preimplementation and postimplementation periods, respectively, across UPHS) (Table 4) (see Supporting Table 7 in the online version of this article). This finding was robust to the study period, so is likely not related to the bedside evaluation prompted by the EWRS. It suggests the EWRS could help triage patients for appropriateness of ICU transfer, a particularly valuable role that should be explored further given the typical strains on ICU capacity,[13] and the mortality resulting from delays in patient transfers into ICUs.[14, 15]

Although we did not find a statistically significant mortality reduction, our study may have been underpowered to detect this outcome. Our study has other limitations. First, our preimplementation/postimplementation design may not fully account for secular changes in sepsis mortality. However, our comparison of similar time periods and our adjustment for observed demographic differences allow us to estimate with more certainty the change in sepsis care and mortality attributable to the intervention. Second, our study did not examine the effect of the EWRS on mortality after hospital discharge, where many such events occur. However, our capture of at least 45 hospital days on all study patients, as well as our inclusion of only those who died or were discharged during our study period, and our assessment of discharge disposition such as hospice, increase the chance that mortality reductions directly attributable to the EWRS were captured. Third, although the EWRS changed patient management, we did not assess the appropriateness of management changes. However, the impact of care changes was captured crudely by examining mortality rates and discharge disposition. Fourth, our study was limited to a single academic healthcare system, and our experience may not be generalizable to other healthcare systems with different EHRs and staff. However, the integration of our automated alert into a commercial EHR serving a diverse array of patient populations, clinical services, and service models throughout our healthcare system may improve the generalizability of our experience to other settings.

CONCLUSION

By leveraging readily available electronic data, an automated prediction tool identified at‐risk patients and mobilized care teams, resulting in more timely sepsis care, improved sepsis documentation, and a suggestion of reduced mortality. This alert may be scalable to other healthcare systems.

There are as many as 3 million cases of severe sepsis and 750,000 resulting deaths in the United States annually.[1] Interventions such as goal‐directed resuscitation and antibiotics can reduce sepsis mortality, but their effectiveness depends on early administration. Thus, timely recognition is critical.[2, 3, 4, 5]

Despite this, early recognition in hospitalized patients can be challenging. Using chart documentation as a surrogate for provider recognition, we recently found only 20% of patients with severe sepsis admitted to our hospital from the emergency department were recognized.[6] Given these challenges, there has been increasing interest in developing automated systems to improve the timeliness of sepsis detection.[7, 8, 9, 10] Systems described in the literature have varied considerably in triggering criteria, effector responses, and study settings. Of those examining the impact of automated surveillance and response in the nonintensive care unit (ICU) acute inpatient setting, results suggest an increase in the timeliness of diagnostic and therapeutic interventions,[10] but less impact on patient outcomes.[7] Whether these results reflect inadequacies in the criteria used to identify patients (parameters or their thresholds) or an ineffective response to the alert (magnitude or timeliness) is unclear.

Given the consequences of severe sepsis in hospitalized patients, as well as the introduction of vital sign (VS) and provider data in our electronic health record (EHR), we sought to develop and implement an electronic sepsis detection and response system to improve patient outcomes. This study describes the development, validation, and impact of that system.

METHODS

The Postimplementation (Live) Period and Impact Analysis

The EWRS went live September 12, 2012, upon which new admissions triggering the alert would result in a notification and response. Unadjusted analyses using the [2] test for dichotomous variables and the Wilcoxon rank sum test for continuous variables compared demographics and the proportion of clinical process and outcome measures for those admitted during the silent period (June 6, 2012September 4, 2012) and a similar timeframe 1 year later when the intervention was live (June 6, 2013September 4, 2013). To be included in either of the time periods, patients had to trigger the alert during the period and be discharged within 45 days of the end of the period. The pre‐ and post‐sepsis mortality index was also examined (see the Supporting Information in the online version of this article for a detailed description of study measures). Multivariable regression models estimated the impact of the EWRS on the process and outcome measures, adjusted for differences between the patients in the preimplementation and postimplementation periods with respect to age, gender, Charlson index on admission, admitting service, hospital, and admission month. Logistic regression models examined dichotomous variables. Continuous variables were log transformed and examined using linear regression models. Cox regression models explored time to ICU transfer from trigger. Among patients with sepsis, a logistic regression model was used to compare the odds of mortality between the silent and live periods, adjusted for expected mortality, both within each hospital and across all hospitals.

Because there is a risk of providers becoming overly reliant on automated systems and overlooking those not triggering the system, we also examined the discharge disposition and mortality outcomes of those in both study periods not identified by the EWRS.

The primary analysis examined the impact of the EWRS across UPHS; we also examined the EWRS impact at each of our hospitals. Last, we performed subgroup analyses examining the EWRS impact in those assigned an International Classification of Diseases, 9th Revision code for sepsis at discharge or death. All analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC).

RESULTS

In the derivation cohort, 4575 patients met the inclusion criteria. The proportion of those in each category (06) achieving our outcomes of interest are described in Supporting Table 1 in the online version of this article. We defined a positive trigger as a score 4, as this threshold identified a limited number of patients (3.9% [180/4575]) with a high proportion experiencing our composite outcome (25.6% [46/180]). The proportion of patients with an EWRS score 4 and their time to event by hospital and health system is described in Supporting Table 2 in the online version of this article. Those with a score 4 were almost 4 times as likely to be transferred to the ICU, almost 7 times as likely to experience an RRT, and almost 10 times as likely to die. The screen positive, sensitivity, specificity, and positive and negative predictive values and likelihood ratios using this threshold and our composite outcome in the derivation cohort were 6%, 16%, 97%, 26%, 94%, 5.3, and 0.9, respectively, and in our validation cohort were 6%, 17%, 97%, 28%, 95%, 5.7, and 0.9, respectively.

In the preimplementation period, 3.8% of admissions (595/15,567) triggered the alert, as compared to 3.5% (545/15,526) in the postimplementation period. Demographics were similar across periods, except that in the postimplementation period patients were slightly younger and had a lower Charlson Comorbidity Index at admission (Table 1). The distribution of alerts across medicine and surgery services were similar (Table 1).

In our postimplementation period, 99% of coordinator pages and over three‐fourths of provider notifications were sent successfully. Almost three‐fourths of nurses reviewed the initial alert notification, and over 99% completed the electronic data verification and adverse trend review, with over half documenting adverse trends. Ninety‐five percent of the time the coordinators completed the follow‐up assessment. Over 90% of the time, the entire team evaluated the patient at bedside within 30 minutes. Almost half of the time, the team thought the patient had no critical illness. Over a third of the time, they thought the patient had sepsis, but reported over 90% of the time that they were aware of the diagnosis prior to the alert. (Supporting Table 3 in the online version of this article includes more details about the responses to the electronic notifications and follow‐up assessments.)

In unadjusted and adjusted analyses, ordering of antibiotics, intravenous fluid boluses, and lactate and blood cultures within 3 hours of the trigger increased significantly, as did ordering of blood products, chest radiographs, and cardiac monitoring within 6 hours of the trigger (Tables 2 and 3).

Hospital and ICU length of stay were similar in the preimplementation and postimplementation periods. There was no difference in the proportion of patients transferred to the ICU following the alert; however, the proportion transferred within 6 hours of the alert increased, and the time to ICU transfer was halved (see Supporting Figure 4 in the online version of this article), but neither change was statistically significant in unadjusted analyses. Transfer to the ICU within 6 hours became statistically significant after adjustment. All mortality measures were lower in the postimplementation period, but none reached statistical significance. Discharge to home and sepsis documentation were both statistically higher in the postimplementation period, but discharge to home lost statistical significance after adjustment (Tables 4 and 5) (see Supporting Table 4 in the online version of this article).

In a subanalysis of EWRS impact on patients documented with sepsis at discharge, unadjusted and adjusted changes in clinical process and outcome measures across the time periods were similar to that of the total population (see Supporting Tables 5 and 6 and Supporting Figure 5 in the online version of this article). The unadjusted composite outcome of mortality or inpatient hospice was statistically lower in the postimplementation period, but lost statistical significance after adjustment.

The disposition and mortality outcomes of those not triggering the alert were unchanged across the 2 periods (see Supporting Tables 7, 8, and 9 in the online version of this article).

DISCUSSION

This study demonstrated that a predictive tool can accurately identify non‐ICU inpatients at increased risk for deterioration and death. In addition, we demonstrated the feasibility of deploying our EHR to screen patients in real time for deterioration and to trigger electronically a timely, robust, multidisciplinary bedside clinical evaluation. Compared to the control (silent) period, the EWRS resulted in a marked increase in early sepsis care, transfer to the ICU, and sepsis documentation, and an indication of a decreased sepsis mortality index and mortality, and increased discharge to home, although none of these latter 3 findings reached statistical significance.

Our study is unique in that it was implemented across a multihospital health system, which has identical EHRs, but diverse cultures, populations, staffing, and practice models. In addition, our study includes a preimplementation population similar to the postimplementation population (in terms of setting, month of admission, and adjustment for potential confounders).

Interestingly, patients identified by the EWRS who were subsequently transferred to an ICU had higher mortality rates (30% and 26% in the preimplementation and postimplementation periods, respectively, across UPHS) than those transferred to an ICU who were not identified by the EWRS (7% and 6% in the preimplementation and postimplementation periods, respectively, across UPHS) (Table 4) (see Supporting Table 7 in the online version of this article). This finding was robust to the study period, so is likely not related to the bedside evaluation prompted by the EWRS. It suggests the EWRS could help triage patients for appropriateness of ICU transfer, a particularly valuable role that should be explored further given the typical strains on ICU capacity,[13] and the mortality resulting from delays in patient transfers into ICUs.[14, 15]

Although we did not find a statistically significant mortality reduction, our study may have been underpowered to detect this outcome. Our study has other limitations. First, our preimplementation/postimplementation design may not fully account for secular changes in sepsis mortality. However, our comparison of similar time periods and our adjustment for observed demographic differences allow us to estimate with more certainty the change in sepsis care and mortality attributable to the intervention. Second, our study did not examine the effect of the EWRS on mortality after hospital discharge, where many such events occur. However, our capture of at least 45 hospital days on all study patients, as well as our inclusion of only those who died or were discharged during our study period, and our assessment of discharge disposition such as hospice, increase the chance that mortality reductions directly attributable to the EWRS were captured. Third, although the EWRS changed patient management, we did not assess the appropriateness of management changes. However, the impact of care changes was captured crudely by examining mortality rates and discharge disposition. Fourth, our study was limited to a single academic healthcare system, and our experience may not be generalizable to other healthcare systems with different EHRs and staff. However, the integration of our automated alert into a commercial EHR serving a diverse array of patient populations, clinical services, and service models throughout our healthcare system may improve the generalizability of our experience to other settings.

CONCLUSION

By leveraging readily available electronic data, an automated prediction tool identified at‐risk patients and mobilized care teams, resulting in more timely sepsis care, improved sepsis documentation, and a suggestion of reduced mortality. This alert may be scalable to other healthcare systems.

There are as many as 3 million cases of severe sepsis and 750,000 resulting deaths in the United States annually.[1] Interventions such as goal‐directed resuscitation and antibiotics can reduce sepsis mortality, but their effectiveness depends on early administration. Thus, timely recognition is critical.[2, 3, 4, 5]

Despite this, early recognition in hospitalized patients can be challenging. Using chart documentation as a surrogate for provider recognition, we recently found only 20% of patients with severe sepsis admitted to our hospital from the emergency department were recognized.[6] Given these challenges, there has been increasing interest in developing automated systems to improve the timeliness of sepsis detection.[7, 8, 9, 10] Systems described in the literature have varied considerably in triggering criteria, effector responses, and study settings. Of those examining the impact of automated surveillance and response in the nonintensive care unit (ICU) acute inpatient setting, results suggest an increase in the timeliness of diagnostic and therapeutic interventions,[10] but less impact on patient outcomes.[7] Whether these results reflect inadequacies in the criteria used to identify patients (parameters or their thresholds) or an ineffective response to the alert (magnitude or timeliness) is unclear.

Given the consequences of severe sepsis in hospitalized patients, as well as the introduction of vital sign (VS) and provider data in our electronic health record (EHR), we sought to develop and implement an electronic sepsis detection and response system to improve patient outcomes. This study describes the development, validation, and impact of that system.

METHODS

The Postimplementation (Live) Period and Impact Analysis

The EWRS went live September 12, 2012, upon which new admissions triggering the alert would result in a notification and response. Unadjusted analyses using the [2] test for dichotomous variables and the Wilcoxon rank sum test for continuous variables compared demographics and the proportion of clinical process and outcome measures for those admitted during the silent period (June 6, 2012September 4, 2012) and a similar timeframe 1 year later when the intervention was live (June 6, 2013September 4, 2013). To be included in either of the time periods, patients had to trigger the alert during the period and be discharged within 45 days of the end of the period. The pre‐ and post‐sepsis mortality index was also examined (see the Supporting Information in the online version of this article for a detailed description of study measures). Multivariable regression models estimated the impact of the EWRS on the process and outcome measures, adjusted for differences between the patients in the preimplementation and postimplementation periods with respect to age, gender, Charlson index on admission, admitting service, hospital, and admission month. Logistic regression models examined dichotomous variables. Continuous variables were log transformed and examined using linear regression models. Cox regression models explored time to ICU transfer from trigger. Among patients with sepsis, a logistic regression model was used to compare the odds of mortality between the silent and live periods, adjusted for expected mortality, both within each hospital and across all hospitals.

Because there is a risk of providers becoming overly reliant on automated systems and overlooking those not triggering the system, we also examined the discharge disposition and mortality outcomes of those in both study periods not identified by the EWRS.

The primary analysis examined the impact of the EWRS across UPHS; we also examined the EWRS impact at each of our hospitals. Last, we performed subgroup analyses examining the EWRS impact in those assigned an International Classification of Diseases, 9th Revision code for sepsis at discharge or death. All analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC).

RESULTS

In the derivation cohort, 4575 patients met the inclusion criteria. The proportion of those in each category (06) achieving our outcomes of interest are described in Supporting Table 1 in the online version of this article. We defined a positive trigger as a score 4, as this threshold identified a limited number of patients (3.9% [180/4575]) with a high proportion experiencing our composite outcome (25.6% [46/180]). The proportion of patients with an EWRS score 4 and their time to event by hospital and health system is described in Supporting Table 2 in the online version of this article. Those with a score 4 were almost 4 times as likely to be transferred to the ICU, almost 7 times as likely to experience an RRT, and almost 10 times as likely to die. The screen positive, sensitivity, specificity, and positive and negative predictive values and likelihood ratios using this threshold and our composite outcome in the derivation cohort were 6%, 16%, 97%, 26%, 94%, 5.3, and 0.9, respectively, and in our validation cohort were 6%, 17%, 97%, 28%, 95%, 5.7, and 0.9, respectively.

In the preimplementation period, 3.8% of admissions (595/15,567) triggered the alert, as compared to 3.5% (545/15,526) in the postimplementation period. Demographics were similar across periods, except that in the postimplementation period patients were slightly younger and had a lower Charlson Comorbidity Index at admission (Table 1). The distribution of alerts across medicine and surgery services were similar (Table 1).

In our postimplementation period, 99% of coordinator pages and over three‐fourths of provider notifications were sent successfully. Almost three‐fourths of nurses reviewed the initial alert notification, and over 99% completed the electronic data verification and adverse trend review, with over half documenting adverse trends. Ninety‐five percent of the time the coordinators completed the follow‐up assessment. Over 90% of the time, the entire team evaluated the patient at bedside within 30 minutes. Almost half of the time, the team thought the patient had no critical illness. Over a third of the time, they thought the patient had sepsis, but reported over 90% of the time that they were aware of the diagnosis prior to the alert. (Supporting Table 3 in the online version of this article includes more details about the responses to the electronic notifications and follow‐up assessments.)

In unadjusted and adjusted analyses, ordering of antibiotics, intravenous fluid boluses, and lactate and blood cultures within 3 hours of the trigger increased significantly, as did ordering of blood products, chest radiographs, and cardiac monitoring within 6 hours of the trigger (Tables 2 and 3).

Hospital and ICU length of stay were similar in the preimplementation and postimplementation periods. There was no difference in the proportion of patients transferred to the ICU following the alert; however, the proportion transferred within 6 hours of the alert increased, and the time to ICU transfer was halved (see Supporting Figure 4 in the online version of this article), but neither change was statistically significant in unadjusted analyses. Transfer to the ICU within 6 hours became statistically significant after adjustment. All mortality measures were lower in the postimplementation period, but none reached statistical significance. Discharge to home and sepsis documentation were both statistically higher in the postimplementation period, but discharge to home lost statistical significance after adjustment (Tables 4 and 5) (see Supporting Table 4 in the online version of this article).

In a subanalysis of EWRS impact on patients documented with sepsis at discharge, unadjusted and adjusted changes in clinical process and outcome measures across the time periods were similar to that of the total population (see Supporting Tables 5 and 6 and Supporting Figure 5 in the online version of this article). The unadjusted composite outcome of mortality or inpatient hospice was statistically lower in the postimplementation period, but lost statistical significance after adjustment.

The disposition and mortality outcomes of those not triggering the alert were unchanged across the 2 periods (see Supporting Tables 7, 8, and 9 in the online version of this article).

DISCUSSION

This study demonstrated that a predictive tool can accurately identify non‐ICU inpatients at increased risk for deterioration and death. In addition, we demonstrated the feasibility of deploying our EHR to screen patients in real time for deterioration and to trigger electronically a timely, robust, multidisciplinary bedside clinical evaluation. Compared to the control (silent) period, the EWRS resulted in a marked increase in early sepsis care, transfer to the ICU, and sepsis documentation, and an indication of a decreased sepsis mortality index and mortality, and increased discharge to home, although none of these latter 3 findings reached statistical significance.

Our study is unique in that it was implemented across a multihospital health system, which has identical EHRs, but diverse cultures, populations, staffing, and practice models. In addition, our study includes a preimplementation population similar to the postimplementation population (in terms of setting, month of admission, and adjustment for potential confounders).

Interestingly, patients identified by the EWRS who were subsequently transferred to an ICU had higher mortality rates (30% and 26% in the preimplementation and postimplementation periods, respectively, across UPHS) than those transferred to an ICU who were not identified by the EWRS (7% and 6% in the preimplementation and postimplementation periods, respectively, across UPHS) (Table 4) (see Supporting Table 7 in the online version of this article). This finding was robust to the study period, so is likely not related to the bedside evaluation prompted by the EWRS. It suggests the EWRS could help triage patients for appropriateness of ICU transfer, a particularly valuable role that should be explored further given the typical strains on ICU capacity,[13] and the mortality resulting from delays in patient transfers into ICUs.[14, 15]

Although we did not find a statistically significant mortality reduction, our study may have been underpowered to detect this outcome. Our study has other limitations. First, our preimplementation/postimplementation design may not fully account for secular changes in sepsis mortality. However, our comparison of similar time periods and our adjustment for observed demographic differences allow us to estimate with more certainty the change in sepsis care and mortality attributable to the intervention. Second, our study did not examine the effect of the EWRS on mortality after hospital discharge, where many such events occur. However, our capture of at least 45 hospital days on all study patients, as well as our inclusion of only those who died or were discharged during our study period, and our assessment of discharge disposition such as hospice, increase the chance that mortality reductions directly attributable to the EWRS were captured. Third, although the EWRS changed patient management, we did not assess the appropriateness of management changes. However, the impact of care changes was captured crudely by examining mortality rates and discharge disposition. Fourth, our study was limited to a single academic healthcare system, and our experience may not be generalizable to other healthcare systems with different EHRs and staff. However, the integration of our automated alert into a commercial EHR serving a diverse array of patient populations, clinical services, and service models throughout our healthcare system may improve the generalizability of our experience to other settings.

CONCLUSION

By leveraging readily available electronic data, an automated prediction tool identified at‐risk patients and mobilized care teams, resulting in more timely sepsis care, improved sepsis documentation, and a suggestion of reduced mortality. This alert may be scalable to other healthcare systems.

Hospital readmissions cost Medicare $15 to $17 billion per year.[1, 2] In 2010, the Hospital Readmission Reduction Program (HRRP), created by the Patient Protection and Affordable Care Act, authorized the Centers for Medicare and Medicaid Services (CMS) to penalize hospitals with higher‐than‐expected readmission rates for certain index conditions.[3] Other payers may follow suit, so hospitals and health systems nationwide are devoting significant resources to reducing readmissions.[4, 5, 6]

Implicit in these efforts are the assumptions that a significant proportion of readmissions are preventable, and that preventable readmissions can be identified. Unfortunately, estimates of preventability vary widely.[7, 8] In this article, we examine how preventable readmissions have been defined, measured, and calculated, and explore the associated implications for readmission reduction efforts.

THE MEDICARE READMISSION METRIC

The medical literature reveals substantial heterogeneity in how readmissions are assessed. Time periods range from 14 days to 4 years, and readmissions may be counted differently depending on whether they are to the same hospital or to any hospital, whether they are for the same (or a related) condition or for any condition, whether a patient is allowed to count only once during the follow‐up period, how mortality is treated, and whether observation stays are considered.[9]

Despite a lack of consensus in the literature, the approach adopted by CMS is endorsed by the National Quality Forum (NQF)[10] and has become the de facto standard for calculating readmission rates. CMS derives risk‐standardized readmission rates for acute myocardial infarction (AMI), heart failure (HF), and pneumonia (PN), using administrative claims data for each Medicare fee‐for‐service beneficiary 65 years or older.[11, 12, 13, 14] CMS counts the first readmission (but not subsequent ones) for any cause within 30 days of the index discharge, including readmissions to other facilities. Certain planned readmissions for revascularization are excluded, as are patients who left against medical advice, transferred to another acute‐care hospital, or died during the index admission. Admissions to psychiatric, rehabilitation, cancer specialty, and children's hospitals[12] are also excluded, as well as patients classified as observation status for either hospital stay.[15] Only administrative data are used in readmission calculations (ie, there are no chart reviews or interviews with healthcare personnel or patients). Details are published online and updated at least annually.[15]

EFFECTS AND LIMITATIONS OF THE HRRP AND THE CMS READMISSION METRIC

Penalizing hospitals for higher‐than‐expected readmission rates based on the CMS metric has been successful in the sense that hospitals now feel more accountable for patient outcomes after discharge; they are implementing transitional care programs, improving communication, and building relationships with community programs.[4, 5, 16] Early data suggest a small decline in readmission rates of Medicare beneficiaries nationally.[17] Previously, such readmission rates were constant.[18]

Nevertheless, significant concerns with the current approach have surfaced.[19, 20, 21] First, why choose 30 days? This time horizon was believed to be long enough to identify readmissions attributable to an index admission and short enough to reflect hospital‐delivered care and transitions to the outpatient setting, and it allows for collaboration between hospitals and their communities to reduce readmissions.[3] However, some have argued that this time horizon has little scientific basis,[22] and that hospitals are unfairly held accountable for a timeframe when outcomes may largely be influenced by the quality of outpatient care or the development of new problems.[23, 24] Approximately one‐third of 30‐day readmissions occur within the first 7 days, and more than half (55.7%) occur within the first 14 days[22, 25]; such time frames may be more appropriate for hospital accountability.[26]

Second, spurred by the focus of CMS penalties, efforts to reduce readmissions have largely concerned patients admitted for HF, AMI, or PN, although these 3 medical conditions account for only 10% of Medicare hospitalizations.[18] Programs focused on a narrow patient population may not benefit other patients with high readmission rates, such as those with gastrointestinal or psychiatric problems,[2] or lead to improvements in the underlying processes of care that could benefit patients in additional ways. Indeed, research suggests that low readmission rates may not be related to other measures of hospital quality.[27, 28]

Third, public reporting and hospital penalties are based on 3‐year historical performance, in part to accumulate a large enough sample size for each diagnosis. Hospitals that seek real‐time performance monitoring are limited to tracking surrogate outcomes, such as readmissions back to their own facility.[29, 30] Moreover, because of the long performance time frame, hospitals that achieve rapid improvement may endure penalties precisely when they are attempting to sustain their achievements.

Fourth, the CMS approach utilizes a complex risk‐standardization methodology, which has only modest ability to predict readmissions and allow hospital comparisons.[9] There is no adjustment for community characteristics, even though practice patterns are significantly associated with readmission rates,[9, 31] and more than half of the variation in readmission rates across hospitals can be explained by characteristics of the community such as access to care.[32] Moreover, patient factors, such as race and socioeconomic status, are currently not included in an attempt to hold hospitals to similar standards regardless of their patient population. This is hotly contested, however, and critics note this policy penalizes hospitals for factors outside of their control, such as patients' ability to afford medications.[33] Indeed, the June 2013 Medicare Payment Advisory Committee (MedPAC) report to Congress recommended evaluating hospital performance against facilities with a like percentage of low‐income patients as a way to take into account socioeconomic status.[34]

Fifth, observation stays are excluded, so patients who remain in observation status during their index or subsequent hospitalization cannot be counted as a readmission. Prevalence of observation care has increased, raising concerns that inpatient admissions are being shifted to observation status, producing an artificial decline in readmissions.[35] Fortunately, recent population‐level data provide some reassuring evidence to the contrary.[36]

Finally, and perhaps most significantly, the current readmission metric does not consider preventability. Recent reviews have demonstrated that estimates of preventability vary widely in individual studies, ranging from 5% to 79%, depending on study methodology and setting.[7, 8] Across these studies, on average, only 23% of 30‐day readmissions appear to be avoidable.[8] Another way to consider the preventability of hospital readmissions is by noting that the most effective multimodal care‐transition interventions reduce readmission rates by only about 30%, and most interventions are much less effective.[26] The likely fact that only 23% to 30% of readmissions are preventable has profound implications for the anticipated results of hospital readmission reduction efforts. Interventions that are 75% effective in reducing preventable readmissions should be expected to produce only an 18% to 22% reduction in overall readmission rates.[37]

FOCUSING ON PREVENTABLE READMISSIONS

A greater focus on identifying and targeting preventable readmissions would offer a number of advantages over the present approach. First, it is more meaningful to compare hospitals based on their percentage of discharges resulting in a preventable readmission, than on the basis of highly complex risk standardization procedures for selected conditions. Second, a focus on preventable readmissions more clearly identifies and permits hospitals to target opportunities for improvement. Third, if the focus were on preventable readmissions for a large number of conditions, the necessary sample size could be obtained over a shorter period of time. Overall, such a preventable readmissions metric could serve as a more agile and undiluted performance indicator, as opposed to the present 3‐year rolling average rate of all‐cause readmissions for certain conditions, the majority of which are probably not preventable.

DEFINING PREVENTABILITY

Defining a preventable readmission is critically important. However, neither a consensus definition nor a validated standard for assessing preventable hospital readmissions exists. Different conceptual frameworks and terms (eg, avoidable, potentially preventable, or urgent readmission) complicate the issue.[38, 39, 40]

Although the CMS measure does not address preventability, it is helpful to consider whether other readmission metrics incorporate this concept. The United Health Group's (UHG, formerly Pacificare) All‐Cause Readmission Index, University HealthSystem Consortium's 30‐Day Readmission Rate (all cause), and 3M Health Information Systems' (3M) Potentially Preventable Readmissions (PPR) are 3 commonly used measures.

Of these, only the 3M PPR metric includes the concept of preventability. 3M created a proprietary matrix of 98,000 readmission‐index admission All Patient Refined Diagnosis Related Group pairs based on the review of several physicians and the logical assumption that a readmission for a clinically related diagnosis is potentially preventable.[24, 41] Readmission and index admissions are considered clinically related if any of the following occur: (1) medical readmission for continuation or recurrence of an initial, or closely related, condition; (2) medical readmission for acute decompensation of a chronic condition that was not the reason for the index admission but was plausibly related to care during or immediately afterward (eg, readmission for diabetes in a patient whose index admission was AMI); (3) medical readmission for acute complication plausibly related to care during index admission; (4) readmission for surgical procedure for continuation or recurrence of initial problem (eg, readmission for appendectomy following admission for abdominal pain and fever); or (5) readmission for surgical procedure to address complication resulting from care during index admission.[24, 41] The readmission time frame is not standardized and may be set by the user. Though conceptually appealing in some ways, CMS and the NQF have expressed concern about this specific approach because of the uncertain reliability of the relatedness of the admission‐readmission diagnosis dyads.[3]

In the research literature, only a few studies have examined the 3M PPR or other preventability assessments that rely on the relatedness of diagnostic codes.[8] Using the 3M PPR, a study showed that 78% of readmissions were classified as potentially preventable,[42] which explains why the 3M PPR and all‐cause readmission metric may correlate highly.[43] Others have demonstrated that ratings of hospital performance on readmission rates vary by a moderate to large amount, depending on whether the 3M PPR, CMS, or UHG methodology is used.[43, 44] An algorithm called SQLape[45, 46] is used in Switzerland to benchmark hospitals and defines potentially avoidable readmissions as being related to index diagnoses or complications of those conditions. It has recently been tested in the United States in a single‐center study,[47] and a multihospital study is underway.

Aside from these algorithms using related diagnosis codes, most ratings of preventability have relied on subjective assessments made primarily through a review of hospital records, and approximately one‐third also included data from clinic visits or interviews with the treating medical team or patients/families.[8] Unfortunately, these reports provide insufficient detail on how to apply their preventability criteria to subsequent readmission reviews. Studies did, however, provide categories of preventability into which readmissions could be organized (see Supporting Information, Appendix Table 1, in the online version of this article for details from a subset of studies cited in van Walraven's reviews that illustrate this point).

Assessment of preventability by clinician review can be challenging. In general, such assessments have considered readmissions resulting from factors within the hospital's control to be avoidable (eg, providing appropriate discharge instructions, reconciling medications, arranging timely postdischarge follow‐up appointments), whereas readmissions resulting from factors not within the hospital's control are unavoidable (eg, patient socioeconomic status, social support, disease progression). However, readmissions resulting from patient behaviors or social reasons could potentially be classified as avoidable or unavoidable depending on the circumstances. For example, if a patient decides not to take a prescribed antibiotic and is readmitted with worsening infection, this could be classified as an unavoidable readmission from the hospital's perspective. Alternatively, if the physician prescribing the antibiotic was inattentive to the cost of the medication and the patient would have taken a less expensive medication had it been prescribed, this could be classified as an avoidable readmission. Differing interpretations of contextual factors may partially account for the variability in clinical assessments of preventability.

Indeed, despite the lack of consensus around a standard method of defining preventability, hospitals and health systems are moving forward to address the issue and reduce readmissions. A recent survey by America's Essential Hospitals (previously the National Association of Public Hospitals and Health Systems), indicated that: (1) reducing readmissions was a high priority for the majority (86%) of members, (2) most had established interdisciplinary teams to address the issue, and (3) over half had a formal process for determining which readmissions were potentially preventable. Of the survey respondents, just over one‐third rely on staff review of individual patient charts or patient and family interviews, and slightly less than one‐third rely on other mechanisms such as external consultants, criteria developed by other entities, or the Institute for Clinical Systems Improvement methodology.[48] Approximately one‐fifth make use of 3M's PPR product, and slightly fewer use the list of the Agency for Healthcare Research and Quality's ambulatory care sensitive conditions (ACSCs). These are medical conditions for which it is believed that good outpatient care could prevent the need for hospitalization (eg, asthma, congestive heart failure, diabetes) or for which early intervention minimizes complications.[49] Hospitalization rates for ACSCs may represent a good measure of excess hospitalization, with a focus on the quality of outpatient care.

RECOMMENDATIONS

We recommend that reporting of hospital readmission rates be based on preventable or potentially preventable readmissions. Although we acknowledge the challenges in doing so, the advantages are notable. At minimum, a preventable readmission rate would more accurately reflect the true gap in care and therefore hospitals' real opportunity for improvement, without being obscured by readmissions that are not preventable.

Because readmission rates are used for public reporting and financial penalties for hospitals, we favor a measure of preventability that reflects the readmissions that the hospital or hospital system has the ability to prevent. This would not penalize hospitals for factors that are under the control of others, namely patients and caregivers, community supports, or society at large. We further recommend that this measure apply to a broader composite of unplanned care, inclusive of both inpatient and observation stays, which have little distinction in patients' eyes, and both represent potentially unnecessary utilization of acute‐care resources.[50] Such a measure would require development, validation, and appropriate vetting before it is implemented.

The first step is for researchers and policy makers to agree on how a measure of preventable or potentially preventable readmissions could be defined. A common element of preventability assessment is to identify the degree to which the reasons for readmission are related to the diagnoses of the index hospitalization. To be reliable and scalable, this measure will need to be based on algorithms that relate the index and readmission diagnoses, most likely using claims data. Choosing common medical and surgical conditions and developing a consensus‐based list of related readmission diagnoses is an important first step. It would also be important to include some less common conditions, because they may reflect very different aspects of hospital care.

An approach based on a list of related diagnoses would represent potentially preventable rehospitalizations. Generally, clinical review is required to determine actual preventability, taking into account patient factors such as a high level of illness or functional impairment that leads to clinical decompensation in spite of excellent management.[51, 52] Clinical review, like a root cause analysis, also provides greater insight into hospital processes that may warrant improvement. Therefore, even if an administrative measure of potentially preventable readmissions is implemented, hospitals may wish to continue performing detailed clinical review of some readmissions for quality improvement purposes. When clinical review becomes more standardized,[53] a combined approach that uses administrative data plus clinical verification and arbitration may be feasible, as with hospital‐acquired infections.

Similar work to develop related sets of admission and readmission diagnoses has already been undertaken in development of the 3M PPR and SQLape measures.[41, 46] However, the 3M PPR is a proprietary system that has low specificity and a high false‐positive rate for identifying preventable readmissions when compared to clinical review.[42] Moreover, neither measure has yet achieved the consensus required for widespread adoption in the United States. What is needed is a nonproprietary listing of related admission and readmission diagnoses, developed with the engagement of relevant stakeholders, that goes through a period of public comment and vetting by a body such as the NQF.

Until a validated measure of potentially preventable readmission can be developed, how could the current approach evolve toward preventability? The most feasible, rapidly implementable change would be to alter the readmission time horizon from 30 days to 7 or 15 days. A 30‐day period holds hospitals accountable for complications of outpatient care or new problems that may develop weeks after discharge. Even though this may foster shared accountability and collaboration among hospitals and outpatient or community settings, research has demonstrated that early readmissions (eg, within 715 days of discharge) are more likely preventable.[54] Second, consideration of the socioeconomic status of hospital patients, as recommended by MedPAC,[34] would improve on the current model by comparing hospitals to like facilities when determining penalties for excess readmission rates. Finally, adjustment for community factors, such as practice patterns and access to care, would enable readmission metrics to better reflect factors under the hospital's control.[32]

CONCLUSION

Holding hospitals accountable for the quality of acute and transitional care is an important policy initiative that has accelerated many improvements in discharge planning and care coordination. Optimally, the policies, public reporting, and penalties should target preventable readmissions, which may represent as little as one‐quarter of all readmissions. By summarizing some of the issues in defining preventability, we hope to foster continued refinement of quality metrics used in this arena.

Hospital readmissions cost Medicare $15 to $17 billion per year.[1, 2] In 2010, the Hospital Readmission Reduction Program (HRRP), created by the Patient Protection and Affordable Care Act, authorized the Centers for Medicare and Medicaid Services (CMS) to penalize hospitals with higher‐than‐expected readmission rates for certain index conditions.[3] Other payers may follow suit, so hospitals and health systems nationwide are devoting significant resources to reducing readmissions.[4, 5, 6]

Implicit in these efforts are the assumptions that a significant proportion of readmissions are preventable, and that preventable readmissions can be identified. Unfortunately, estimates of preventability vary widely.[7, 8] In this article, we examine how preventable readmissions have been defined, measured, and calculated, and explore the associated implications for readmission reduction efforts.

THE MEDICARE READMISSION METRIC

The medical literature reveals substantial heterogeneity in how readmissions are assessed. Time periods range from 14 days to 4 years, and readmissions may be counted differently depending on whether they are to the same hospital or to any hospital, whether they are for the same (or a related) condition or for any condition, whether a patient is allowed to count only once during the follow‐up period, how mortality is treated, and whether observation stays are considered.[9]

Despite a lack of consensus in the literature, the approach adopted by CMS is endorsed by the National Quality Forum (NQF)[10] and has become the de facto standard for calculating readmission rates. CMS derives risk‐standardized readmission rates for acute myocardial infarction (AMI), heart failure (HF), and pneumonia (PN), using administrative claims data for each Medicare fee‐for‐service beneficiary 65 years or older.[11, 12, 13, 14] CMS counts the first readmission (but not subsequent ones) for any cause within 30 days of the index discharge, including readmissions to other facilities. Certain planned readmissions for revascularization are excluded, as are patients who left against medical advice, transferred to another acute‐care hospital, or died during the index admission. Admissions to psychiatric, rehabilitation, cancer specialty, and children's hospitals[12] are also excluded, as well as patients classified as observation status for either hospital stay.[15] Only administrative data are used in readmission calculations (ie, there are no chart reviews or interviews with healthcare personnel or patients). Details are published online and updated at least annually.[15]

EFFECTS AND LIMITATIONS OF THE HRRP AND THE CMS READMISSION METRIC

Penalizing hospitals for higher‐than‐expected readmission rates based on the CMS metric has been successful in the sense that hospitals now feel more accountable for patient outcomes after discharge; they are implementing transitional care programs, improving communication, and building relationships with community programs.[4, 5, 16] Early data suggest a small decline in readmission rates of Medicare beneficiaries nationally.[17] Previously, such readmission rates were constant.[18]

Nevertheless, significant concerns with the current approach have surfaced.[19, 20, 21] First, why choose 30 days? This time horizon was believed to be long enough to identify readmissions attributable to an index admission and short enough to reflect hospital‐delivered care and transitions to the outpatient setting, and it allows for collaboration between hospitals and their communities to reduce readmissions.[3] However, some have argued that this time horizon has little scientific basis,[22] and that hospitals are unfairly held accountable for a timeframe when outcomes may largely be influenced by the quality of outpatient care or the development of new problems.[23, 24] Approximately one‐third of 30‐day readmissions occur within the first 7 days, and more than half (55.7%) occur within the first 14 days[22, 25]; such time frames may be more appropriate for hospital accountability.[26]

Second, spurred by the focus of CMS penalties, efforts to reduce readmissions have largely concerned patients admitted for HF, AMI, or PN, although these 3 medical conditions account for only 10% of Medicare hospitalizations.[18] Programs focused on a narrow patient population may not benefit other patients with high readmission rates, such as those with gastrointestinal or psychiatric problems,[2] or lead to improvements in the underlying processes of care that could benefit patients in additional ways. Indeed, research suggests that low readmission rates may not be related to other measures of hospital quality.[27, 28]

Third, public reporting and hospital penalties are based on 3‐year historical performance, in part to accumulate a large enough sample size for each diagnosis. Hospitals that seek real‐time performance monitoring are limited to tracking surrogate outcomes, such as readmissions back to their own facility.[29, 30] Moreover, because of the long performance time frame, hospitals that achieve rapid improvement may endure penalties precisely when they are attempting to sustain their achievements.

Fourth, the CMS approach utilizes a complex risk‐standardization methodology, which has only modest ability to predict readmissions and allow hospital comparisons.[9] There is no adjustment for community characteristics, even though practice patterns are significantly associated with readmission rates,[9, 31] and more than half of the variation in readmission rates across hospitals can be explained by characteristics of the community such as access to care.[32] Moreover, patient factors, such as race and socioeconomic status, are currently not included in an attempt to hold hospitals to similar standards regardless of their patient population. This is hotly contested, however, and critics note this policy penalizes hospitals for factors outside of their control, such as patients' ability to afford medications.[33] Indeed, the June 2013 Medicare Payment Advisory Committee (MedPAC) report to Congress recommended evaluating hospital performance against facilities with a like percentage of low‐income patients as a way to take into account socioeconomic status.[34]

Fifth, observation stays are excluded, so patients who remain in observation status during their index or subsequent hospitalization cannot be counted as a readmission. Prevalence of observation care has increased, raising concerns that inpatient admissions are being shifted to observation status, producing an artificial decline in readmissions.[35] Fortunately, recent population‐level data provide some reassuring evidence to the contrary.[36]

Finally, and perhaps most significantly, the current readmission metric does not consider preventability. Recent reviews have demonstrated that estimates of preventability vary widely in individual studies, ranging from 5% to 79%, depending on study methodology and setting.[7, 8] Across these studies, on average, only 23% of 30‐day readmissions appear to be avoidable.[8] Another way to consider the preventability of hospital readmissions is by noting that the most effective multimodal care‐transition interventions reduce readmission rates by only about 30%, and most interventions are much less effective.[26] The likely fact that only 23% to 30% of readmissions are preventable has profound implications for the anticipated results of hospital readmission reduction efforts. Interventions that are 75% effective in reducing preventable readmissions should be expected to produce only an 18% to 22% reduction in overall readmission rates.[37]

FOCUSING ON PREVENTABLE READMISSIONS

A greater focus on identifying and targeting preventable readmissions would offer a number of advantages over the present approach. First, it is more meaningful to compare hospitals based on their percentage of discharges resulting in a preventable readmission, than on the basis of highly complex risk standardization procedures for selected conditions. Second, a focus on preventable readmissions more clearly identifies and permits hospitals to target opportunities for improvement. Third, if the focus were on preventable readmissions for a large number of conditions, the necessary sample size could be obtained over a shorter period of time. Overall, such a preventable readmissions metric could serve as a more agile and undiluted performance indicator, as opposed to the present 3‐year rolling average rate of all‐cause readmissions for certain conditions, the majority of which are probably not preventable.

DEFINING PREVENTABILITY

Defining a preventable readmission is critically important. However, neither a consensus definition nor a validated standard for assessing preventable hospital readmissions exists. Different conceptual frameworks and terms (eg, avoidable, potentially preventable, or urgent readmission) complicate the issue.[38, 39, 40]

Although the CMS measure does not address preventability, it is helpful to consider whether other readmission metrics incorporate this concept. The United Health Group's (UHG, formerly Pacificare) All‐Cause Readmission Index, University HealthSystem Consortium's 30‐Day Readmission Rate (all cause), and 3M Health Information Systems' (3M) Potentially Preventable Readmissions (PPR) are 3 commonly used measures.

Of these, only the 3M PPR metric includes the concept of preventability. 3M created a proprietary matrix of 98,000 readmission‐index admission All Patient Refined Diagnosis Related Group pairs based on the review of several physicians and the logical assumption that a readmission for a clinically related diagnosis is potentially preventable.[24, 41] Readmission and index admissions are considered clinically related if any of the following occur: (1) medical readmission for continuation or recurrence of an initial, or closely related, condition; (2) medical readmission for acute decompensation of a chronic condition that was not the reason for the index admission but was plausibly related to care during or immediately afterward (eg, readmission for diabetes in a patient whose index admission was AMI); (3) medical readmission for acute complication plausibly related to care during index admission; (4) readmission for surgical procedure for continuation or recurrence of initial problem (eg, readmission for appendectomy following admission for abdominal pain and fever); or (5) readmission for surgical procedure to address complication resulting from care during index admission.[24, 41] The readmission time frame is not standardized and may be set by the user. Though conceptually appealing in some ways, CMS and the NQF have expressed concern about this specific approach because of the uncertain reliability of the relatedness of the admission‐readmission diagnosis dyads.[3]

In the research literature, only a few studies have examined the 3M PPR or other preventability assessments that rely on the relatedness of diagnostic codes.[8] Using the 3M PPR, a study showed that 78% of readmissions were classified as potentially preventable,[42] which explains why the 3M PPR and all‐cause readmission metric may correlate highly.[43] Others have demonstrated that ratings of hospital performance on readmission rates vary by a moderate to large amount, depending on whether the 3M PPR, CMS, or UHG methodology is used.[43, 44] An algorithm called SQLape[45, 46] is used in Switzerland to benchmark hospitals and defines potentially avoidable readmissions as being related to index diagnoses or complications of those conditions. It has recently been tested in the United States in a single‐center study,[47] and a multihospital study is underway.

Aside from these algorithms using related diagnosis codes, most ratings of preventability have relied on subjective assessments made primarily through a review of hospital records, and approximately one‐third also included data from clinic visits or interviews with the treating medical team or patients/families.[8] Unfortunately, these reports provide insufficient detail on how to apply their preventability criteria to subsequent readmission reviews. Studies did, however, provide categories of preventability into which readmissions could be organized (see Supporting Information, Appendix Table 1, in the online version of this article for details from a subset of studies cited in van Walraven's reviews that illustrate this point).

Assessment of preventability by clinician review can be challenging. In general, such assessments have considered readmissions resulting from factors within the hospital's control to be avoidable (eg, providing appropriate discharge instructions, reconciling medications, arranging timely postdischarge follow‐up appointments), whereas readmissions resulting from factors not within the hospital's control are unavoidable (eg, patient socioeconomic status, social support, disease progression). However, readmissions resulting from patient behaviors or social reasons could potentially be classified as avoidable or unavoidable depending on the circumstances. For example, if a patient decides not to take a prescribed antibiotic and is readmitted with worsening infection, this could be classified as an unavoidable readmission from the hospital's perspective. Alternatively, if the physician prescribing the antibiotic was inattentive to the cost of the medication and the patient would have taken a less expensive medication had it been prescribed, this could be classified as an avoidable readmission. Differing interpretations of contextual factors may partially account for the variability in clinical assessments of preventability.

Indeed, despite the lack of consensus around a standard method of defining preventability, hospitals and health systems are moving forward to address the issue and reduce readmissions. A recent survey by America's Essential Hospitals (previously the National Association of Public Hospitals and Health Systems), indicated that: (1) reducing readmissions was a high priority for the majority (86%) of members, (2) most had established interdisciplinary teams to address the issue, and (3) over half had a formal process for determining which readmissions were potentially preventable. Of the survey respondents, just over one‐third rely on staff review of individual patient charts or patient and family interviews, and slightly less than one‐third rely on other mechanisms such as external consultants, criteria developed by other entities, or the Institute for Clinical Systems Improvement methodology.[48] Approximately one‐fifth make use of 3M's PPR product, and slightly fewer use the list of the Agency for Healthcare Research and Quality's ambulatory care sensitive conditions (ACSCs). These are medical conditions for which it is believed that good outpatient care could prevent the need for hospitalization (eg, asthma, congestive heart failure, diabetes) or for which early intervention minimizes complications.[49] Hospitalization rates for ACSCs may represent a good measure of excess hospitalization, with a focus on the quality of outpatient care.

RECOMMENDATIONS

We recommend that reporting of hospital readmission rates be based on preventable or potentially preventable readmissions. Although we acknowledge the challenges in doing so, the advantages are notable. At minimum, a preventable readmission rate would more accurately reflect the true gap in care and therefore hospitals' real opportunity for improvement, without being obscured by readmissions that are not preventable.

Because readmission rates are used for public reporting and financial penalties for hospitals, we favor a measure of preventability that reflects the readmissions that the hospital or hospital system has the ability to prevent. This would not penalize hospitals for factors that are under the control of others, namely patients and caregivers, community supports, or society at large. We further recommend that this measure apply to a broader composite of unplanned care, inclusive of both inpatient and observation stays, which have little distinction in patients' eyes, and both represent potentially unnecessary utilization of acute‐care resources.[50] Such a measure would require development, validation, and appropriate vetting before it is implemented.

The first step is for researchers and policy makers to agree on how a measure of preventable or potentially preventable readmissions could be defined. A common element of preventability assessment is to identify the degree to which the reasons for readmission are related to the diagnoses of the index hospitalization. To be reliable and scalable, this measure will need to be based on algorithms that relate the index and readmission diagnoses, most likely using claims data. Choosing common medical and surgical conditions and developing a consensus‐based list of related readmission diagnoses is an important first step. It would also be important to include some less common conditions, because they may reflect very different aspects of hospital care.

An approach based on a list of related diagnoses would represent potentially preventable rehospitalizations. Generally, clinical review is required to determine actual preventability, taking into account patient factors such as a high level of illness or functional impairment that leads to clinical decompensation in spite of excellent management.[51, 52] Clinical review, like a root cause analysis, also provides greater insight into hospital processes that may warrant improvement. Therefore, even if an administrative measure of potentially preventable readmissions is implemented, hospitals may wish to continue performing detailed clinical review of some readmissions for quality improvement purposes. When clinical review becomes more standardized,[53] a combined approach that uses administrative data plus clinical verification and arbitration may be feasible, as with hospital‐acquired infections.

Similar work to develop related sets of admission and readmission diagnoses has already been undertaken in development of the 3M PPR and SQLape measures.[41, 46] However, the 3M PPR is a proprietary system that has low specificity and a high false‐positive rate for identifying preventable readmissions when compared to clinical review.[42] Moreover, neither measure has yet achieved the consensus required for widespread adoption in the United States. What is needed is a nonproprietary listing of related admission and readmission diagnoses, developed with the engagement of relevant stakeholders, that goes through a period of public comment and vetting by a body such as the NQF.

Until a validated measure of potentially preventable readmission can be developed, how could the current approach evolve toward preventability? The most feasible, rapidly implementable change would be to alter the readmission time horizon from 30 days to 7 or 15 days. A 30‐day period holds hospitals accountable for complications of outpatient care or new problems that may develop weeks after discharge. Even though this may foster shared accountability and collaboration among hospitals and outpatient or community settings, research has demonstrated that early readmissions (eg, within 715 days of discharge) are more likely preventable.[54] Second, consideration of the socioeconomic status of hospital patients, as recommended by MedPAC,[34] would improve on the current model by comparing hospitals to like facilities when determining penalties for excess readmission rates. Finally, adjustment for community factors, such as practice patterns and access to care, would enable readmission metrics to better reflect factors under the hospital's control.[32]

CONCLUSION

Holding hospitals accountable for the quality of acute and transitional care is an important policy initiative that has accelerated many improvements in discharge planning and care coordination. Optimally, the policies, public reporting, and penalties should target preventable readmissions, which may represent as little as one‐quarter of all readmissions. By summarizing some of the issues in defining preventability, we hope to foster continued refinement of quality metrics used in this arena.

Hospital readmissions cost Medicare $15 to $17 billion per year.[1, 2] In 2010, the Hospital Readmission Reduction Program (HRRP), created by the Patient Protection and Affordable Care Act, authorized the Centers for Medicare and Medicaid Services (CMS) to penalize hospitals with higher‐than‐expected readmission rates for certain index conditions.[3] Other payers may follow suit, so hospitals and health systems nationwide are devoting significant resources to reducing readmissions.[4, 5, 6]

Implicit in these efforts are the assumptions that a significant proportion of readmissions are preventable, and that preventable readmissions can be identified. Unfortunately, estimates of preventability vary widely.[7, 8] In this article, we examine how preventable readmissions have been defined, measured, and calculated, and explore the associated implications for readmission reduction efforts.

THE MEDICARE READMISSION METRIC

The medical literature reveals substantial heterogeneity in how readmissions are assessed. Time periods range from 14 days to 4 years, and readmissions may be counted differently depending on whether they are to the same hospital or to any hospital, whether they are for the same (or a related) condition or for any condition, whether a patient is allowed to count only once during the follow‐up period, how mortality is treated, and whether observation stays are considered.[9]

Despite a lack of consensus in the literature, the approach adopted by CMS is endorsed by the National Quality Forum (NQF)[10] and has become the de facto standard for calculating readmission rates. CMS derives risk‐standardized readmission rates for acute myocardial infarction (AMI), heart failure (HF), and pneumonia (PN), using administrative claims data for each Medicare fee‐for‐service beneficiary 65 years or older.[11, 12, 13, 14] CMS counts the first readmission (but not subsequent ones) for any cause within 30 days of the index discharge, including readmissions to other facilities. Certain planned readmissions for revascularization are excluded, as are patients who left against medical advice, transferred to another acute‐care hospital, or died during the index admission. Admissions to psychiatric, rehabilitation, cancer specialty, and children's hospitals[12] are also excluded, as well as patients classified as observation status for either hospital stay.[15] Only administrative data are used in readmission calculations (ie, there are no chart reviews or interviews with healthcare personnel or patients). Details are published online and updated at least annually.[15]

EFFECTS AND LIMITATIONS OF THE HRRP AND THE CMS READMISSION METRIC

Penalizing hospitals for higher‐than‐expected readmission rates based on the CMS metric has been successful in the sense that hospitals now feel more accountable for patient outcomes after discharge; they are implementing transitional care programs, improving communication, and building relationships with community programs.[4, 5, 16] Early data suggest a small decline in readmission rates of Medicare beneficiaries nationally.[17] Previously, such readmission rates were constant.[18]

Nevertheless, significant concerns with the current approach have surfaced.[19, 20, 21] First, why choose 30 days? This time horizon was believed to be long enough to identify readmissions attributable to an index admission and short enough to reflect hospital‐delivered care and transitions to the outpatient setting, and it allows for collaboration between hospitals and their communities to reduce readmissions.[3] However, some have argued that this time horizon has little scientific basis,[22] and that hospitals are unfairly held accountable for a timeframe when outcomes may largely be influenced by the quality of outpatient care or the development of new problems.[23, 24] Approximately one‐third of 30‐day readmissions occur within the first 7 days, and more than half (55.7%) occur within the first 14 days[22, 25]; such time frames may be more appropriate for hospital accountability.[26]

Second, spurred by the focus of CMS penalties, efforts to reduce readmissions have largely concerned patients admitted for HF, AMI, or PN, although these 3 medical conditions account for only 10% of Medicare hospitalizations.[18] Programs focused on a narrow patient population may not benefit other patients with high readmission rates, such as those with gastrointestinal or psychiatric problems,[2] or lead to improvements in the underlying processes of care that could benefit patients in additional ways. Indeed, research suggests that low readmission rates may not be related to other measures of hospital quality.[27, 28]

Third, public reporting and hospital penalties are based on 3‐year historical performance, in part to accumulate a large enough sample size for each diagnosis. Hospitals that seek real‐time performance monitoring are limited to tracking surrogate outcomes, such as readmissions back to their own facility.[29, 30] Moreover, because of the long performance time frame, hospitals that achieve rapid improvement may endure penalties precisely when they are attempting to sustain their achievements.

Fourth, the CMS approach utilizes a complex risk‐standardization methodology, which has only modest ability to predict readmissions and allow hospital comparisons.[9] There is no adjustment for community characteristics, even though practice patterns are significantly associated with readmission rates,[9, 31] and more than half of the variation in readmission rates across hospitals can be explained by characteristics of the community such as access to care.[32] Moreover, patient factors, such as race and socioeconomic status, are currently not included in an attempt to hold hospitals to similar standards regardless of their patient population. This is hotly contested, however, and critics note this policy penalizes hospitals for factors outside of their control, such as patients' ability to afford medications.[33] Indeed, the June 2013 Medicare Payment Advisory Committee (MedPAC) report to Congress recommended evaluating hospital performance against facilities with a like percentage of low‐income patients as a way to take into account socioeconomic status.[34]

Fifth, observation stays are excluded, so patients who remain in observation status during their index or subsequent hospitalization cannot be counted as a readmission. Prevalence of observation care has increased, raising concerns that inpatient admissions are being shifted to observation status, producing an artificial decline in readmissions.[35] Fortunately, recent population‐level data provide some reassuring evidence to the contrary.[36]

Finally, and perhaps most significantly, the current readmission metric does not consider preventability. Recent reviews have demonstrated that estimates of preventability vary widely in individual studies, ranging from 5% to 79%, depending on study methodology and setting.[7, 8] Across these studies, on average, only 23% of 30‐day readmissions appear to be avoidable.[8] Another way to consider the preventability of hospital readmissions is by noting that the most effective multimodal care‐transition interventions reduce readmission rates by only about 30%, and most interventions are much less effective.[26] The likely fact that only 23% to 30% of readmissions are preventable has profound implications for the anticipated results of hospital readmission reduction efforts. Interventions that are 75% effective in reducing preventable readmissions should be expected to produce only an 18% to 22% reduction in overall readmission rates.[37]

FOCUSING ON PREVENTABLE READMISSIONS

A greater focus on identifying and targeting preventable readmissions would offer a number of advantages over the present approach. First, it is more meaningful to compare hospitals based on their percentage of discharges resulting in a preventable readmission, than on the basis of highly complex risk standardization procedures for selected conditions. Second, a focus on preventable readmissions more clearly identifies and permits hospitals to target opportunities for improvement. Third, if the focus were on preventable readmissions for a large number of conditions, the necessary sample size could be obtained over a shorter period of time. Overall, such a preventable readmissions metric could serve as a more agile and undiluted performance indicator, as opposed to the present 3‐year rolling average rate of all‐cause readmissions for certain conditions, the majority of which are probably not preventable.

DEFINING PREVENTABILITY

Defining a preventable readmission is critically important. However, neither a consensus definition nor a validated standard for assessing preventable hospital readmissions exists. Different conceptual frameworks and terms (eg, avoidable, potentially preventable, or urgent readmission) complicate the issue.[38, 39, 40]

Although the CMS measure does not address preventability, it is helpful to consider whether other readmission metrics incorporate this concept. The United Health Group's (UHG, formerly Pacificare) All‐Cause Readmission Index, University HealthSystem Consortium's 30‐Day Readmission Rate (all cause), and 3M Health Information Systems' (3M) Potentially Preventable Readmissions (PPR) are 3 commonly used measures.

Of these, only the 3M PPR metric includes the concept of preventability. 3M created a proprietary matrix of 98,000 readmission‐index admission All Patient Refined Diagnosis Related Group pairs based on the review of several physicians and the logical assumption that a readmission for a clinically related diagnosis is potentially preventable.[24, 41] Readmission and index admissions are considered clinically related if any of the following occur: (1) medical readmission for continuation or recurrence of an initial, or closely related, condition; (2) medical readmission for acute decompensation of a chronic condition that was not the reason for the index admission but was plausibly related to care during or immediately afterward (eg, readmission for diabetes in a patient whose index admission was AMI); (3) medical readmission for acute complication plausibly related to care during index admission; (4) readmission for surgical procedure for continuation or recurrence of initial problem (eg, readmission for appendectomy following admission for abdominal pain and fever); or (5) readmission for surgical procedure to address complication resulting from care during index admission.[24, 41] The readmission time frame is not standardized and may be set by the user. Though conceptually appealing in some ways, CMS and the NQF have expressed concern about this specific approach because of the uncertain reliability of the relatedness of the admission‐readmission diagnosis dyads.[3]

In the research literature, only a few studies have examined the 3M PPR or other preventability assessments that rely on the relatedness of diagnostic codes.[8] Using the 3M PPR, a study showed that 78% of readmissions were classified as potentially preventable,[42] which explains why the 3M PPR and all‐cause readmission metric may correlate highly.[43] Others have demonstrated that ratings of hospital performance on readmission rates vary by a moderate to large amount, depending on whether the 3M PPR, CMS, or UHG methodology is used.[43, 44] An algorithm called SQLape[45, 46] is used in Switzerland to benchmark hospitals and defines potentially avoidable readmissions as being related to index diagnoses or complications of those conditions. It has recently been tested in the United States in a single‐center study,[47] and a multihospital study is underway.