Peter Lindenauer, MD, MSc

Infective endocarditis (IE) affected an estimated 46,800 Americans in 2011, and its incidence is increasing due to greater numbers of invasive procedures and prevalence of IE risk factors.^1-3 Despite recent advances in the treatment of IE, morbidity and mortality remain high: in-hospital mortality in IE patients is 15% to 20%, and the 1-year mortality rate is approximately 40%.^2,4,5

Poor IE outcomes may be the result of difficulty in diagnosing IE and identifying its optimal treatments. The American Heart Association (AHA), the American College of Cardiology (ACC), and the European Society of Cardiology (ESC) have published guidelines to address these challenges. Recent guidelines recommend a multidisciplinary approach that includes cardiology, cardiac surgery, and infectious disease (ID) specialty involvement in decision-making.^5,6

In the absence of published quality measures for IE management, guidelines can be used to evaluate the quality of care of IE. Studies have showed poor concordance with guideline recommendations but did not examine agreement with more recently published guidelines.^7,8 Furthermore, few studies have examined the management, outcomes, and quality of care received by IE patients. Therefore, we aimed to describe a modern cohort of patients with IE admitted to a tertiary medical center over a 4-year period. In particular, we aimed to assess quality of care received by this cohort, as measured by concordance with AHA and ACC guidelines to identify gaps in care and spur quality improvement (QI) efforts.

METHODS

Design and Study Population

We conducted a retrospective cohort study of adult IE patients admitted to Baystate Medical Center (BMC), a 716-bed tertiary academic center that covers a population of 800,000 people throughout western New England. We used the International Classification of Diseases (ICD)–Ninth Revision, to identify IE patients discharged with a principal or secondary diagnosis of IE between 2007 and 2011 (codes 421.0, 421.1, 421.9, 424.9, 424.90, and 424.91). Three co-authors confirmed the diagnosis by conducting a review of the electronic health records.

We included only patients who met modified Duke criteria for definite or possible IE.⁵ Definite IE defines patients with pathological criteria (microorganisms demonstrated by culture or histologic examination or a histologic examination showing active endocarditis); or patients with 2 major criteria (positive blood culture and evidence of endocardial involvement by echocardiogram), 1 major criterion and 3 minor criteria (minor criteria: predisposing heart conditions or intravenous drug (IVD) use, fever, vascular phenomena, immunologic phenomena, and microbiologic evidence that do not meet the major criteria) or 5 minor criteria. Possible IE defines patients with 1 major and 1 minor criterion or 3 minor criteria.⁵

 

Data Collection

We used billing and clinical databases to collect demographics, comorbidities, antibiotic treatment, 6-month readmission and 1-year mortality. Comorbid conditions were classified into Elixhauser comorbidities using software provided by the Healthcare Costs and Utilization Project of the Agency for Healthcare Research and Quality.^9,10

We obtained all other data through electronic health record abstraction. These included microbiology, type of endocarditis (native valve endocarditis [NVE] or prosthetic valve endocarditis [PVE]), echocardiographic location of the vegetation, and complications involving the valve (eg, valve perforation, ruptured chorda, perivalvular abscess, or valvular insufficiency).

Using 2006 AHA/ACC guidelines,¹¹ we identified quality metrics, including the presence of at least 2 sets of blood cultures prior to start of antibiotics and use of transthoracic echocardiogram (TTE) and transesophageal echocardiogram (TEE). Guidelines recommend using TTE as first-line to detect valvular vegetations and assess IE complications. TEE is recommended if TTE is nondiagnostic and also as first-line to diagnose PVE. We assessed the rate of consultation with ID, cardiology, and cardiac surgery specialties. While these consultations were not explicitly emphasized in the 2006 AHA/ACC guidelines, there is a class I recommendation in 2014 AHA/ACC guidelines⁵ to manage IE patients with consultation of all these specialties.

We reported the number of patients with intracardiac leads (pacemaker or defibrillator) who had documentation of intracardiac lead removal. Complete removal of intracardiac leads is indicated in IE patients with infection of leads or device (class I) and suggested for IE caused by Staphylococcus aureus or fungi (even without evidence of device or lead infection), and for patients undergoing valve surgery (class IIa).⁵ We entered abstracted data elements into a RedCap database, hosted by Tufts Clinical and Translational Science Institute.¹²

Outcomes

Outcomes included embolic events, strokes, need for cardiac surgery, LOS, inhospital mortality, 6-month readmission, and 1-year mortality. We identified embolic events using documentation of clinical or imaging evidence of an embolic event to the cerebral, coronary, peripheral arterial, renal, splenic, or pulmonary vasculature. We used record extraction to identify incidence of valve surgery. Nearly all patients who require surgery at BMC have it done onsite. We compared outcomes among patients who received less than 3 vs. 3 consultations provided by ID, cardiology, and cardiac surgery specialties. We also compared outcomes among patients who received 0, 1, 2, or 3 consultations to look for a trend.

Statistical Analysis

We divided the cohort into patients with NVE and PVE because there are differences in pathophysiology, treatment, and outcomes of these groups. We calculated descriptive statistics, including means/standard deviation (SD) and n (%). We conducted univariable analyses using Fisher exact (categorical), unpaired t tests (Gaussian), or Kruskal-Wallis equality-of-populations rank test (non-Gaussian). Common language effect sizes were also calculated to quantify group differences without respect to sample size.^13,14 Analyses were performed using Stata 14.1. (StataCorp LLC, College Station, Texas). The BMC Institutional Review Board approved the protocol.

RESULTS

We identified a total of 317 hospitalizations at BMC meeting criteria for IE. Of these, 147 hospitalizations were readmissions or did not meet the clinical criteria of definite or possible IE. Thus, we included a total of 170 patients in the final analysis. Definite IE was present in 135 (79.4%) and possible IE in 35 (20.6%) patients.

Patient Characteristics

Of 170 patients, 127 (74.7%) had NVE and 43 (25.3%) had PVE. Mean ± SD age was 60.0 ± 17.9 years, 66.5% (n = 113) of patients were male, and 79.4% (n = 135) were white (Table 1). Hypertension and chronic kidney disease were the most common comorbidities. The median Gagne score¹⁵ was 4, corresponding to a 1-year mortality risk of 15%. Predisposing factors for IE included previous history of IE (n = 14 or 8.2%), IVD use (n = 23 or 13.5%), and presence of long-term venous catheters (n = 19 or 11.2%). Intracardiac leads were present in 17.1% (n = 29) of patients. Bicuspid aortic valve was reported in 6.5% (n = 11) of patients with NVE. Patients with PVE were older (+11.5 years, 95% confidence interval [CI] 5.5, 17.5) and more likely to have intracardiac leads (44.2% vs. 7.9%; P < 0.001; Table 1).

Microbiology and Antibiotics

Staphylococcus aureus was isolated in 40.0% of patients (methicillin-sensitive: 21.2%, n = 36; methicillin-resistant: 18.8%, n = 32) and vancomycin (88.2%, n = 150) was the most common initial antibiotic used. Nearly half (44.7%, n = 76) of patients received gentamicin as part of their initial antibiotic regimen. Appendix 1 provides information on final blood culture results, prosthetic versus native valve IE, and antimicrobial agents that each patient received. PVE patients were more likely to receive gentamicin as their initial antibiotic regimen than NVE (58.1% vs. 40.2%; P = 0.051; Table 1).

 

Echocardiography and Affected Valves

As per study inclusion criteria, all patients received echocardiography (either TTE, TEE, or both). Overall, the most common infected valve was mitral (41.3%), n = 59), followed by aortic valve (28.7%), n = 41). Patients in whom the location of infected valve could not be determined (15.9%, n = 27) had echocardiographic features of intracardiac device infection or intracardiac mass (Table 1).

Quality of Care

Nearly all (n = 165, 97.1%) of patients had at least 2 sets of blood cultures drawn, most on the first day of admission (71.2%). The vast majority of patients (n = 152, 89.4%) also received their first dose of antibiotics on the day of admission. Ten (5.9%) patients did not receive any consults, and 160 (94.1%) received at least 1 consultation. An ID consultation was obtained for most (147, 86.5%) patients; cardiac surgery consultation was obtained for about half of patients (92, 54.1%), and cardiology consultation was also obtained for nearly half of patients (80, 47.1%). One-third (53, 31.2%) did not receive a cardiology or cardiac surgery consult, two-thirds (117, 68.8%) received either a cardiology or a cardiac surgery consult, and one-third (55, 32.4%) received both.

Of the 29 patients who had an intracardiac lead, 6 patients had documentation of the device removal during the index hospitalization (5 or 50.0% of patients with NVE and 1 or 5.3% of patients with PVE; P = 0.02; Table 2).

Outcomes

Evidence of any embolic events was seen in 27.7% (n = 47) of patients, including stroke in 17.1% (n = 29). Median LOS for all patients was 13.5 days, and 6-month readmission among patients who survived their index admission was 51.0% (n = 74/145; 95% CI, 45.9%-62.7%). Inhospital mortality was 14.7% (n = 25; 95% CI: 10.1%-20.9%) and 12-month mortality was 22.4% (n = 38; 95% CI, 16.7%-29.3%). Inhospital mortality was more frequent among patients with PVE than NVE (20.9% vs. 12.6%; P = 0.21), although this difference was not statistically significant. Complications were more common in NVE than PVE (any embolic event: 32.3% vs. 14.0%, P = 0.03; stroke, 20.5% vs. 7.0%, P = 0.06; Table 3).

Although there was a trend toward reduction in 6-month readmission and 12-month mortality with incremental increase in the number of specialties consulted (ID, cardiology and cardiac surgery), the difference was not statistically significant (Figure). In addition, comparing outcomes of embolic events, stroke, 6-month readmission, and 12-month mortality between those who received 3 consults (28.8%, n = 49) to those with fewer than 3 (71.2%, n = 121) did not show statistically significant differences.

Of 92 patients who received a cardiac surgery consult, 73 had NVE and 19 had PVE. Of these, 47 underwent valve surgery, 39 (of 73) with NVE (53.42%) and 8 (of 19) with PVE (42.1%). Most of the NVE patients (73.2%) had more than 1 indication for surgery. The most common indications for surgery among NVE patients were significant valvular dysfunction resulting in heart failure (65.9%), followed by mobile vegetation (56.1%) and recurrent embolic events (26.8%). The most common indication for surgery in PVE was persistent bacteremia or recurrent embolic events (75.0%).

DISCUSSION

In this study, we described the management, quality of care, and outcomes of IE patients in a tertiary medical center. We found that the majority of hospitalized patients with IE were older white men with comorbidities and IE risk factors. The complication rate was high (27.7% with embolic events) and the inhospital mortality rate was in the lower range reported by prior studies [14.7% vs. 15%-20%].⁵ Nearly one-third of patients (n = 47, 27.7%) received valve surgery. Quality of care received was generally good, with most patients receiving early blood cultures, echocardiograms, early antibiotics, and timely ID consultation. We identified important gaps in care, including a failure to consult cardiac surgery in nearly half of patients and failure to consult cardiology in more than half of patients.

Our findings support work suggesting that IE is no longer primarily a chronic or subacute disease of younger patients with IVD use, positive human immunodeficiency virus status, or bicuspid aortic valves.^1,4,16,17 The International Collaboration on Endocarditis-Prospective Cohort Study,⁴ a multinational prospective cohort study (2000-2005) of 2781 adults with IE, reported a higher prevalence of patients with diabetes or on hemodialysis, IVD users, and patients with long-term venous catheter and intracardiac leads than we found. Yet both studies suggest that the demographics of IE are changing. This may partially explain why IE mortality has not improved in recent years:^2,3 patients with older age and higher comorbidity burden may not be considered good surgical candidates.

This study is among the first to contribute information on concordance with IE guidelines in a cohort of U.S. patients. Our findings suggest that most patients received timely blood culture, same-day administration of empiric antibiotics, and ID consultation, which is similar to European studies.^7,18 Guideline concordance could be improved in some areas. Overall documentation of the management plan regarding the intracardiac leads could be improved. Only 6 of 29 patients with intracardiac leads had documentation of their removal during the index hospitalization.

The 2014 AHA/ACC guidelines⁵ and the ESC guidelines⁶ emphasized the importance of multidisciplinary management of IE. As part of the Heart Valve Team at BMC, cardiologists provide expertise in diagnosis, imaging and clinical management of IE, and cardiac surgeons provide consultation on whether to pursue surgery and optimal timing of surgery. Early discussion with surgical team is considered mandatory in all complicated cases of IE.^6,18 Infectious disease consultation has been shown to improve the rate of IE diagnosis, reduce the 6-month relapse rate,¹⁹ and improve outcomes in patients with S aureus bacteremia.²⁰ In our study 86.5% of patients had documentation of an ID consultation; cardiac surgery consultation was obtained in 54.1% and cardiology consultation in 47.1% of patients.

We observed a trend towards lower rates of 6-month readmission and 12-month mortality among patients who received all 3 consults (Figure 1), despite the fact that rates of embolic events and stroke were higher in patients with 3 consults compared to those with fewer than 3. Obviously, the lack of confounder adjustment and lack of power limits our ability to make inferences about this association, but it generates hypotheses for future work. Because subjects in our study were cared for prior to 2014, multidisciplinary management of IE with involvement of cardiology, cardiac surgery, and ID physicians was observed in only one-third of patients. However, 117 (68.8%) patients received either cardiology or cardiac surgery consults. It is possible that some physicians considered involving both cardiology and cardiac surgery consultants as unnecessary and, therefore, did not consult both specialties. We will focus future QI efforts in our institution on educating physicians about the benefits of multidisciplinary care and the importance of fully implementing the 2014 AHA/ACC guidelines.

Our findings around quality of care should be placed in the context of 2 studies by González de Molina et al⁸ and Delahaye et al⁷ These studies described considerable discordance between guideline recommendations and real-world IE care. However, these studies were performed more than a decade ago and were conducted before current recommendations to consult cardiology and cardiac surgery were published.

In the 2014 AHA/ACC guidelines, surgery prior to completion of antibiotics is indicated in patients with valve dysfunction resulting in heart failure; left-sided IE caused by highly resistant organisms (including fungus or S aureus); IE complicated by heart block, aortic abscess, or penetrating lesions; and presence of persistent infection (bacteremia or fever lasting longer than 5 to 7 days) after onset of appropriate antimicrobial therapy. In addition, there is a Class IIa indication of early surgery in patients with recurrent emboli and persistent vegetation despite appropriate antibiotic therapy and a Class IIb indication of early surgery in patients with NVE with mobile vegetation greater than 10 mm in length. Surgery is recommended for patients with PVE and relapsing infection.

It is recommended that IE patients be cared for in centers with immediate access to cardiac surgery because the urgent need for surgical intervention can arise rapidly.⁵ We found that nearly one-third of included patients underwent surgery. Although we did not collect data on indications for surgery in patients who did not receive surgery, we observed that 50% had a surgery consult, suggesting the presence of 1 or more surgical indications. Of these, half underwent valve surgery. Most of the NVE patients who underwent surgery had more than 1 indication for surgery. Our surgical rate is similar to a study from Italy³ and overall in the lower range of reported surgical rate (25%-50%) from other studies.²¹ The low rate of surgery at our center may be related to the fact that the use of surgery for IE has been hotly debated in the literature,²¹ and may also be due to the low rate of cardiac surgery consultation.

Our study has several limitations. We identified eligible patients using a discharge ICD-9 coding of IE and then confirmed the presence of Duke criteria using record review. Using discharge diagnosis codes for endocarditis has been validated, and our additional manual chart review to confirm Duke criteria likely improved the specificity significantly. However, by excluding patients who did not have documented evidence of Duke criteria, we may have missed some cases, lowering sensitivity. The performance on selected quality metrics may also have been affected by our inclusion criteria. Because we included only patients who met Duke criteria, we tended to include patients who had received blood cultures and echocardiograms, which are part of the criteria. Thus, we cannot comment on use of diagnostic testing or specialty consultation in patients with suspected IE. This was a single-center study and may not represent patients or current practices seen in other institutions. We did not collect data on some of the predisposing factors to NVE (for example, baseline rheumatic heart disease or preexisting valvular heart disease) because it is estimated that less than 5% of IE in the U.S. is superimposed on rheumatic heart disease.⁴ We likely underestimated 12-month mortality rate because we did not cross-reference our findings again the National Death Index; however, this should not affect the comparison of this outcome between groups.

 

CONCLUSION

Our study confirms reports that IE epidemiology has changed significantly in recent years. It also suggests that concordance with guideline recommendations is good for some aspects of care (eg, echocardiogram, blood cultures), but can be improved in other areas, particularly in use of specialty consultation during the hospitalization. Future QI efforts should emphasize the role of the heart valve team or endocarditis team that consists of an internist, ID physician, cardiologist, cardiac surgeon, and nursing. Finally, efforts should be made to develop strategies for community hospitals that do not have access to all of these specialists (eg, early transfer, telehealth).

Disclosure

Nothing to report.

Infective endocarditis (IE) affected an estimated 46,800 Americans in 2011, and its incidence is increasing due to greater numbers of invasive procedures and prevalence of IE risk factors.^1-3 Despite recent advances in the treatment of IE, morbidity and mortality remain high: in-hospital mortality in IE patients is 15% to 20%, and the 1-year mortality rate is approximately 40%.^2,4,5

Poor IE outcomes may be the result of difficulty in diagnosing IE and identifying its optimal treatments. The American Heart Association (AHA), the American College of Cardiology (ACC), and the European Society of Cardiology (ESC) have published guidelines to address these challenges. Recent guidelines recommend a multidisciplinary approach that includes cardiology, cardiac surgery, and infectious disease (ID) specialty involvement in decision-making.^5,6

In the absence of published quality measures for IE management, guidelines can be used to evaluate the quality of care of IE. Studies have showed poor concordance with guideline recommendations but did not examine agreement with more recently published guidelines.^7,8 Furthermore, few studies have examined the management, outcomes, and quality of care received by IE patients. Therefore, we aimed to describe a modern cohort of patients with IE admitted to a tertiary medical center over a 4-year period. In particular, we aimed to assess quality of care received by this cohort, as measured by concordance with AHA and ACC guidelines to identify gaps in care and spur quality improvement (QI) efforts.

METHODS

Design and Study Population

We conducted a retrospective cohort study of adult IE patients admitted to Baystate Medical Center (BMC), a 716-bed tertiary academic center that covers a population of 800,000 people throughout western New England. We used the International Classification of Diseases (ICD)–Ninth Revision, to identify IE patients discharged with a principal or secondary diagnosis of IE between 2007 and 2011 (codes 421.0, 421.1, 421.9, 424.9, 424.90, and 424.91). Three co-authors confirmed the diagnosis by conducting a review of the electronic health records.

We included only patients who met modified Duke criteria for definite or possible IE.⁵ Definite IE defines patients with pathological criteria (microorganisms demonstrated by culture or histologic examination or a histologic examination showing active endocarditis); or patients with 2 major criteria (positive blood culture and evidence of endocardial involvement by echocardiogram), 1 major criterion and 3 minor criteria (minor criteria: predisposing heart conditions or intravenous drug (IVD) use, fever, vascular phenomena, immunologic phenomena, and microbiologic evidence that do not meet the major criteria) or 5 minor criteria. Possible IE defines patients with 1 major and 1 minor criterion or 3 minor criteria.⁵

 

Data Collection

We used billing and clinical databases to collect demographics, comorbidities, antibiotic treatment, 6-month readmission and 1-year mortality. Comorbid conditions were classified into Elixhauser comorbidities using software provided by the Healthcare Costs and Utilization Project of the Agency for Healthcare Research and Quality.^9,10

We obtained all other data through electronic health record abstraction. These included microbiology, type of endocarditis (native valve endocarditis [NVE] or prosthetic valve endocarditis [PVE]), echocardiographic location of the vegetation, and complications involving the valve (eg, valve perforation, ruptured chorda, perivalvular abscess, or valvular insufficiency).

Using 2006 AHA/ACC guidelines,¹¹ we identified quality metrics, including the presence of at least 2 sets of blood cultures prior to start of antibiotics and use of transthoracic echocardiogram (TTE) and transesophageal echocardiogram (TEE). Guidelines recommend using TTE as first-line to detect valvular vegetations and assess IE complications. TEE is recommended if TTE is nondiagnostic and also as first-line to diagnose PVE. We assessed the rate of consultation with ID, cardiology, and cardiac surgery specialties. While these consultations were not explicitly emphasized in the 2006 AHA/ACC guidelines, there is a class I recommendation in 2014 AHA/ACC guidelines⁵ to manage IE patients with consultation of all these specialties.

We reported the number of patients with intracardiac leads (pacemaker or defibrillator) who had documentation of intracardiac lead removal. Complete removal of intracardiac leads is indicated in IE patients with infection of leads or device (class I) and suggested for IE caused by Staphylococcus aureus or fungi (even without evidence of device or lead infection), and for patients undergoing valve surgery (class IIa).⁵ We entered abstracted data elements into a RedCap database, hosted by Tufts Clinical and Translational Science Institute.¹²

Outcomes

Outcomes included embolic events, strokes, need for cardiac surgery, LOS, inhospital mortality, 6-month readmission, and 1-year mortality. We identified embolic events using documentation of clinical or imaging evidence of an embolic event to the cerebral, coronary, peripheral arterial, renal, splenic, or pulmonary vasculature. We used record extraction to identify incidence of valve surgery. Nearly all patients who require surgery at BMC have it done onsite. We compared outcomes among patients who received less than 3 vs. 3 consultations provided by ID, cardiology, and cardiac surgery specialties. We also compared outcomes among patients who received 0, 1, 2, or 3 consultations to look for a trend.

Statistical Analysis

We divided the cohort into patients with NVE and PVE because there are differences in pathophysiology, treatment, and outcomes of these groups. We calculated descriptive statistics, including means/standard deviation (SD) and n (%). We conducted univariable analyses using Fisher exact (categorical), unpaired t tests (Gaussian), or Kruskal-Wallis equality-of-populations rank test (non-Gaussian). Common language effect sizes were also calculated to quantify group differences without respect to sample size.^13,14 Analyses were performed using Stata 14.1. (StataCorp LLC, College Station, Texas). The BMC Institutional Review Board approved the protocol.

RESULTS

We identified a total of 317 hospitalizations at BMC meeting criteria for IE. Of these, 147 hospitalizations were readmissions or did not meet the clinical criteria of definite or possible IE. Thus, we included a total of 170 patients in the final analysis. Definite IE was present in 135 (79.4%) and possible IE in 35 (20.6%) patients.

Patient Characteristics

Of 170 patients, 127 (74.7%) had NVE and 43 (25.3%) had PVE. Mean ± SD age was 60.0 ± 17.9 years, 66.5% (n = 113) of patients were male, and 79.4% (n = 135) were white (Table 1). Hypertension and chronic kidney disease were the most common comorbidities. The median Gagne score¹⁵ was 4, corresponding to a 1-year mortality risk of 15%. Predisposing factors for IE included previous history of IE (n = 14 or 8.2%), IVD use (n = 23 or 13.5%), and presence of long-term venous catheters (n = 19 or 11.2%). Intracardiac leads were present in 17.1% (n = 29) of patients. Bicuspid aortic valve was reported in 6.5% (n = 11) of patients with NVE. Patients with PVE were older (+11.5 years, 95% confidence interval [CI] 5.5, 17.5) and more likely to have intracardiac leads (44.2% vs. 7.9%; P < 0.001; Table 1).

Microbiology and Antibiotics

Staphylococcus aureus was isolated in 40.0% of patients (methicillin-sensitive: 21.2%, n = 36; methicillin-resistant: 18.8%, n = 32) and vancomycin (88.2%, n = 150) was the most common initial antibiotic used. Nearly half (44.7%, n = 76) of patients received gentamicin as part of their initial antibiotic regimen. Appendix 1 provides information on final blood culture results, prosthetic versus native valve IE, and antimicrobial agents that each patient received. PVE patients were more likely to receive gentamicin as their initial antibiotic regimen than NVE (58.1% vs. 40.2%; P = 0.051; Table 1).

 

Echocardiography and Affected Valves

As per study inclusion criteria, all patients received echocardiography (either TTE, TEE, or both). Overall, the most common infected valve was mitral (41.3%), n = 59), followed by aortic valve (28.7%), n = 41). Patients in whom the location of infected valve could not be determined (15.9%, n = 27) had echocardiographic features of intracardiac device infection or intracardiac mass (Table 1).

Quality of Care

Nearly all (n = 165, 97.1%) of patients had at least 2 sets of blood cultures drawn, most on the first day of admission (71.2%). The vast majority of patients (n = 152, 89.4%) also received their first dose of antibiotics on the day of admission. Ten (5.9%) patients did not receive any consults, and 160 (94.1%) received at least 1 consultation. An ID consultation was obtained for most (147, 86.5%) patients; cardiac surgery consultation was obtained for about half of patients (92, 54.1%), and cardiology consultation was also obtained for nearly half of patients (80, 47.1%). One-third (53, 31.2%) did not receive a cardiology or cardiac surgery consult, two-thirds (117, 68.8%) received either a cardiology or a cardiac surgery consult, and one-third (55, 32.4%) received both.

Of the 29 patients who had an intracardiac lead, 6 patients had documentation of the device removal during the index hospitalization (5 or 50.0% of patients with NVE and 1 or 5.3% of patients with PVE; P = 0.02; Table 2).

Outcomes

Evidence of any embolic events was seen in 27.7% (n = 47) of patients, including stroke in 17.1% (n = 29). Median LOS for all patients was 13.5 days, and 6-month readmission among patients who survived their index admission was 51.0% (n = 74/145; 95% CI, 45.9%-62.7%). Inhospital mortality was 14.7% (n = 25; 95% CI: 10.1%-20.9%) and 12-month mortality was 22.4% (n = 38; 95% CI, 16.7%-29.3%). Inhospital mortality was more frequent among patients with PVE than NVE (20.9% vs. 12.6%; P = 0.21), although this difference was not statistically significant. Complications were more common in NVE than PVE (any embolic event: 32.3% vs. 14.0%, P = 0.03; stroke, 20.5% vs. 7.0%, P = 0.06; Table 3).

Although there was a trend toward reduction in 6-month readmission and 12-month mortality with incremental increase in the number of specialties consulted (ID, cardiology and cardiac surgery), the difference was not statistically significant (Figure). In addition, comparing outcomes of embolic events, stroke, 6-month readmission, and 12-month mortality between those who received 3 consults (28.8%, n = 49) to those with fewer than 3 (71.2%, n = 121) did not show statistically significant differences.

Of 92 patients who received a cardiac surgery consult, 73 had NVE and 19 had PVE. Of these, 47 underwent valve surgery, 39 (of 73) with NVE (53.42%) and 8 (of 19) with PVE (42.1%). Most of the NVE patients (73.2%) had more than 1 indication for surgery. The most common indications for surgery among NVE patients were significant valvular dysfunction resulting in heart failure (65.9%), followed by mobile vegetation (56.1%) and recurrent embolic events (26.8%). The most common indication for surgery in PVE was persistent bacteremia or recurrent embolic events (75.0%).

DISCUSSION

In this study, we described the management, quality of care, and outcomes of IE patients in a tertiary medical center. We found that the majority of hospitalized patients with IE were older white men with comorbidities and IE risk factors. The complication rate was high (27.7% with embolic events) and the inhospital mortality rate was in the lower range reported by prior studies [14.7% vs. 15%-20%].⁵ Nearly one-third of patients (n = 47, 27.7%) received valve surgery. Quality of care received was generally good, with most patients receiving early blood cultures, echocardiograms, early antibiotics, and timely ID consultation. We identified important gaps in care, including a failure to consult cardiac surgery in nearly half of patients and failure to consult cardiology in more than half of patients.

Our findings support work suggesting that IE is no longer primarily a chronic or subacute disease of younger patients with IVD use, positive human immunodeficiency virus status, or bicuspid aortic valves.^1,4,16,17 The International Collaboration on Endocarditis-Prospective Cohort Study,⁴ a multinational prospective cohort study (2000-2005) of 2781 adults with IE, reported a higher prevalence of patients with diabetes or on hemodialysis, IVD users, and patients with long-term venous catheter and intracardiac leads than we found. Yet both studies suggest that the demographics of IE are changing. This may partially explain why IE mortality has not improved in recent years:^2,3 patients with older age and higher comorbidity burden may not be considered good surgical candidates.

This study is among the first to contribute information on concordance with IE guidelines in a cohort of U.S. patients. Our findings suggest that most patients received timely blood culture, same-day administration of empiric antibiotics, and ID consultation, which is similar to European studies.^7,18 Guideline concordance could be improved in some areas. Overall documentation of the management plan regarding the intracardiac leads could be improved. Only 6 of 29 patients with intracardiac leads had documentation of their removal during the index hospitalization.

The 2014 AHA/ACC guidelines⁵ and the ESC guidelines⁶ emphasized the importance of multidisciplinary management of IE. As part of the Heart Valve Team at BMC, cardiologists provide expertise in diagnosis, imaging and clinical management of IE, and cardiac surgeons provide consultation on whether to pursue surgery and optimal timing of surgery. Early discussion with surgical team is considered mandatory in all complicated cases of IE.^6,18 Infectious disease consultation has been shown to improve the rate of IE diagnosis, reduce the 6-month relapse rate,¹⁹ and improve outcomes in patients with S aureus bacteremia.²⁰ In our study 86.5% of patients had documentation of an ID consultation; cardiac surgery consultation was obtained in 54.1% and cardiology consultation in 47.1% of patients.

We observed a trend towards lower rates of 6-month readmission and 12-month mortality among patients who received all 3 consults (Figure 1), despite the fact that rates of embolic events and stroke were higher in patients with 3 consults compared to those with fewer than 3. Obviously, the lack of confounder adjustment and lack of power limits our ability to make inferences about this association, but it generates hypotheses for future work. Because subjects in our study were cared for prior to 2014, multidisciplinary management of IE with involvement of cardiology, cardiac surgery, and ID physicians was observed in only one-third of patients. However, 117 (68.8%) patients received either cardiology or cardiac surgery consults. It is possible that some physicians considered involving both cardiology and cardiac surgery consultants as unnecessary and, therefore, did not consult both specialties. We will focus future QI efforts in our institution on educating physicians about the benefits of multidisciplinary care and the importance of fully implementing the 2014 AHA/ACC guidelines.

Our findings around quality of care should be placed in the context of 2 studies by González de Molina et al⁸ and Delahaye et al⁷ These studies described considerable discordance between guideline recommendations and real-world IE care. However, these studies were performed more than a decade ago and were conducted before current recommendations to consult cardiology and cardiac surgery were published.

In the 2014 AHA/ACC guidelines, surgery prior to completion of antibiotics is indicated in patients with valve dysfunction resulting in heart failure; left-sided IE caused by highly resistant organisms (including fungus or S aureus); IE complicated by heart block, aortic abscess, or penetrating lesions; and presence of persistent infection (bacteremia or fever lasting longer than 5 to 7 days) after onset of appropriate antimicrobial therapy. In addition, there is a Class IIa indication of early surgery in patients with recurrent emboli and persistent vegetation despite appropriate antibiotic therapy and a Class IIb indication of early surgery in patients with NVE with mobile vegetation greater than 10 mm in length. Surgery is recommended for patients with PVE and relapsing infection.

It is recommended that IE patients be cared for in centers with immediate access to cardiac surgery because the urgent need for surgical intervention can arise rapidly.⁵ We found that nearly one-third of included patients underwent surgery. Although we did not collect data on indications for surgery in patients who did not receive surgery, we observed that 50% had a surgery consult, suggesting the presence of 1 or more surgical indications. Of these, half underwent valve surgery. Most of the NVE patients who underwent surgery had more than 1 indication for surgery. Our surgical rate is similar to a study from Italy³ and overall in the lower range of reported surgical rate (25%-50%) from other studies.²¹ The low rate of surgery at our center may be related to the fact that the use of surgery for IE has been hotly debated in the literature,²¹ and may also be due to the low rate of cardiac surgery consultation.

Our study has several limitations. We identified eligible patients using a discharge ICD-9 coding of IE and then confirmed the presence of Duke criteria using record review. Using discharge diagnosis codes for endocarditis has been validated, and our additional manual chart review to confirm Duke criteria likely improved the specificity significantly. However, by excluding patients who did not have documented evidence of Duke criteria, we may have missed some cases, lowering sensitivity. The performance on selected quality metrics may also have been affected by our inclusion criteria. Because we included only patients who met Duke criteria, we tended to include patients who had received blood cultures and echocardiograms, which are part of the criteria. Thus, we cannot comment on use of diagnostic testing or specialty consultation in patients with suspected IE. This was a single-center study and may not represent patients or current practices seen in other institutions. We did not collect data on some of the predisposing factors to NVE (for example, baseline rheumatic heart disease or preexisting valvular heart disease) because it is estimated that less than 5% of IE in the U.S. is superimposed on rheumatic heart disease.⁴ We likely underestimated 12-month mortality rate because we did not cross-reference our findings again the National Death Index; however, this should not affect the comparison of this outcome between groups.

 

CONCLUSION

Our study confirms reports that IE epidemiology has changed significantly in recent years. It also suggests that concordance with guideline recommendations is good for some aspects of care (eg, echocardiogram, blood cultures), but can be improved in other areas, particularly in use of specialty consultation during the hospitalization. Future QI efforts should emphasize the role of the heart valve team or endocarditis team that consists of an internist, ID physician, cardiologist, cardiac surgeon, and nursing. Finally, efforts should be made to develop strategies for community hospitals that do not have access to all of these specialists (eg, early transfer, telehealth).

Disclosure

Nothing to report.

Infective endocarditis (IE) affected an estimated 46,800 Americans in 2011, and its incidence is increasing due to greater numbers of invasive procedures and prevalence of IE risk factors.^1-3 Despite recent advances in the treatment of IE, morbidity and mortality remain high: in-hospital mortality in IE patients is 15% to 20%, and the 1-year mortality rate is approximately 40%.^2,4,5

Poor IE outcomes may be the result of difficulty in diagnosing IE and identifying its optimal treatments. The American Heart Association (AHA), the American College of Cardiology (ACC), and the European Society of Cardiology (ESC) have published guidelines to address these challenges. Recent guidelines recommend a multidisciplinary approach that includes cardiology, cardiac surgery, and infectious disease (ID) specialty involvement in decision-making.^5,6

In the absence of published quality measures for IE management, guidelines can be used to evaluate the quality of care of IE. Studies have showed poor concordance with guideline recommendations but did not examine agreement with more recently published guidelines.^7,8 Furthermore, few studies have examined the management, outcomes, and quality of care received by IE patients. Therefore, we aimed to describe a modern cohort of patients with IE admitted to a tertiary medical center over a 4-year period. In particular, we aimed to assess quality of care received by this cohort, as measured by concordance with AHA and ACC guidelines to identify gaps in care and spur quality improvement (QI) efforts.

METHODS

Design and Study Population

We conducted a retrospective cohort study of adult IE patients admitted to Baystate Medical Center (BMC), a 716-bed tertiary academic center that covers a population of 800,000 people throughout western New England. We used the International Classification of Diseases (ICD)–Ninth Revision, to identify IE patients discharged with a principal or secondary diagnosis of IE between 2007 and 2011 (codes 421.0, 421.1, 421.9, 424.9, 424.90, and 424.91). Three co-authors confirmed the diagnosis by conducting a review of the electronic health records.

We included only patients who met modified Duke criteria for definite or possible IE.⁵ Definite IE defines patients with pathological criteria (microorganisms demonstrated by culture or histologic examination or a histologic examination showing active endocarditis); or patients with 2 major criteria (positive blood culture and evidence of endocardial involvement by echocardiogram), 1 major criterion and 3 minor criteria (minor criteria: predisposing heart conditions or intravenous drug (IVD) use, fever, vascular phenomena, immunologic phenomena, and microbiologic evidence that do not meet the major criteria) or 5 minor criteria. Possible IE defines patients with 1 major and 1 minor criterion or 3 minor criteria.⁵

 

Data Collection

We used billing and clinical databases to collect demographics, comorbidities, antibiotic treatment, 6-month readmission and 1-year mortality. Comorbid conditions were classified into Elixhauser comorbidities using software provided by the Healthcare Costs and Utilization Project of the Agency for Healthcare Research and Quality.^9,10

We obtained all other data through electronic health record abstraction. These included microbiology, type of endocarditis (native valve endocarditis [NVE] or prosthetic valve endocarditis [PVE]), echocardiographic location of the vegetation, and complications involving the valve (eg, valve perforation, ruptured chorda, perivalvular abscess, or valvular insufficiency).

Using 2006 AHA/ACC guidelines,¹¹ we identified quality metrics, including the presence of at least 2 sets of blood cultures prior to start of antibiotics and use of transthoracic echocardiogram (TTE) and transesophageal echocardiogram (TEE). Guidelines recommend using TTE as first-line to detect valvular vegetations and assess IE complications. TEE is recommended if TTE is nondiagnostic and also as first-line to diagnose PVE. We assessed the rate of consultation with ID, cardiology, and cardiac surgery specialties. While these consultations were not explicitly emphasized in the 2006 AHA/ACC guidelines, there is a class I recommendation in 2014 AHA/ACC guidelines⁵ to manage IE patients with consultation of all these specialties.

We reported the number of patients with intracardiac leads (pacemaker or defibrillator) who had documentation of intracardiac lead removal. Complete removal of intracardiac leads is indicated in IE patients with infection of leads or device (class I) and suggested for IE caused by Staphylococcus aureus or fungi (even without evidence of device or lead infection), and for patients undergoing valve surgery (class IIa).⁵ We entered abstracted data elements into a RedCap database, hosted by Tufts Clinical and Translational Science Institute.¹²

Outcomes

Outcomes included embolic events, strokes, need for cardiac surgery, LOS, inhospital mortality, 6-month readmission, and 1-year mortality. We identified embolic events using documentation of clinical or imaging evidence of an embolic event to the cerebral, coronary, peripheral arterial, renal, splenic, or pulmonary vasculature. We used record extraction to identify incidence of valve surgery. Nearly all patients who require surgery at BMC have it done onsite. We compared outcomes among patients who received less than 3 vs. 3 consultations provided by ID, cardiology, and cardiac surgery specialties. We also compared outcomes among patients who received 0, 1, 2, or 3 consultations to look for a trend.

Statistical Analysis

We divided the cohort into patients with NVE and PVE because there are differences in pathophysiology, treatment, and outcomes of these groups. We calculated descriptive statistics, including means/standard deviation (SD) and n (%). We conducted univariable analyses using Fisher exact (categorical), unpaired t tests (Gaussian), or Kruskal-Wallis equality-of-populations rank test (non-Gaussian). Common language effect sizes were also calculated to quantify group differences without respect to sample size.^13,14 Analyses were performed using Stata 14.1. (StataCorp LLC, College Station, Texas). The BMC Institutional Review Board approved the protocol.

RESULTS

We identified a total of 317 hospitalizations at BMC meeting criteria for IE. Of these, 147 hospitalizations were readmissions or did not meet the clinical criteria of definite or possible IE. Thus, we included a total of 170 patients in the final analysis. Definite IE was present in 135 (79.4%) and possible IE in 35 (20.6%) patients.

Patient Characteristics

Of 170 patients, 127 (74.7%) had NVE and 43 (25.3%) had PVE. Mean ± SD age was 60.0 ± 17.9 years, 66.5% (n = 113) of patients were male, and 79.4% (n = 135) were white (Table 1). Hypertension and chronic kidney disease were the most common comorbidities. The median Gagne score¹⁵ was 4, corresponding to a 1-year mortality risk of 15%. Predisposing factors for IE included previous history of IE (n = 14 or 8.2%), IVD use (n = 23 or 13.5%), and presence of long-term venous catheters (n = 19 or 11.2%). Intracardiac leads were present in 17.1% (n = 29) of patients. Bicuspid aortic valve was reported in 6.5% (n = 11) of patients with NVE. Patients with PVE were older (+11.5 years, 95% confidence interval [CI] 5.5, 17.5) and more likely to have intracardiac leads (44.2% vs. 7.9%; P < 0.001; Table 1).

Microbiology and Antibiotics

Staphylococcus aureus was isolated in 40.0% of patients (methicillin-sensitive: 21.2%, n = 36; methicillin-resistant: 18.8%, n = 32) and vancomycin (88.2%, n = 150) was the most common initial antibiotic used. Nearly half (44.7%, n = 76) of patients received gentamicin as part of their initial antibiotic regimen. Appendix 1 provides information on final blood culture results, prosthetic versus native valve IE, and antimicrobial agents that each patient received. PVE patients were more likely to receive gentamicin as their initial antibiotic regimen than NVE (58.1% vs. 40.2%; P = 0.051; Table 1).

 

Echocardiography and Affected Valves

As per study inclusion criteria, all patients received echocardiography (either TTE, TEE, or both). Overall, the most common infected valve was mitral (41.3%), n = 59), followed by aortic valve (28.7%), n = 41). Patients in whom the location of infected valve could not be determined (15.9%, n = 27) had echocardiographic features of intracardiac device infection or intracardiac mass (Table 1).

Quality of Care

Nearly all (n = 165, 97.1%) of patients had at least 2 sets of blood cultures drawn, most on the first day of admission (71.2%). The vast majority of patients (n = 152, 89.4%) also received their first dose of antibiotics on the day of admission. Ten (5.9%) patients did not receive any consults, and 160 (94.1%) received at least 1 consultation. An ID consultation was obtained for most (147, 86.5%) patients; cardiac surgery consultation was obtained for about half of patients (92, 54.1%), and cardiology consultation was also obtained for nearly half of patients (80, 47.1%). One-third (53, 31.2%) did not receive a cardiology or cardiac surgery consult, two-thirds (117, 68.8%) received either a cardiology or a cardiac surgery consult, and one-third (55, 32.4%) received both.

Of the 29 patients who had an intracardiac lead, 6 patients had documentation of the device removal during the index hospitalization (5 or 50.0% of patients with NVE and 1 or 5.3% of patients with PVE; P = 0.02; Table 2).

Outcomes

Evidence of any embolic events was seen in 27.7% (n = 47) of patients, including stroke in 17.1% (n = 29). Median LOS for all patients was 13.5 days, and 6-month readmission among patients who survived their index admission was 51.0% (n = 74/145; 95% CI, 45.9%-62.7%). Inhospital mortality was 14.7% (n = 25; 95% CI: 10.1%-20.9%) and 12-month mortality was 22.4% (n = 38; 95% CI, 16.7%-29.3%). Inhospital mortality was more frequent among patients with PVE than NVE (20.9% vs. 12.6%; P = 0.21), although this difference was not statistically significant. Complications were more common in NVE than PVE (any embolic event: 32.3% vs. 14.0%, P = 0.03; stroke, 20.5% vs. 7.0%, P = 0.06; Table 3).

Although there was a trend toward reduction in 6-month readmission and 12-month mortality with incremental increase in the number of specialties consulted (ID, cardiology and cardiac surgery), the difference was not statistically significant (Figure). In addition, comparing outcomes of embolic events, stroke, 6-month readmission, and 12-month mortality between those who received 3 consults (28.8%, n = 49) to those with fewer than 3 (71.2%, n = 121) did not show statistically significant differences.

Of 92 patients who received a cardiac surgery consult, 73 had NVE and 19 had PVE. Of these, 47 underwent valve surgery, 39 (of 73) with NVE (53.42%) and 8 (of 19) with PVE (42.1%). Most of the NVE patients (73.2%) had more than 1 indication for surgery. The most common indications for surgery among NVE patients were significant valvular dysfunction resulting in heart failure (65.9%), followed by mobile vegetation (56.1%) and recurrent embolic events (26.8%). The most common indication for surgery in PVE was persistent bacteremia or recurrent embolic events (75.0%).

DISCUSSION

In this study, we described the management, quality of care, and outcomes of IE patients in a tertiary medical center. We found that the majority of hospitalized patients with IE were older white men with comorbidities and IE risk factors. The complication rate was high (27.7% with embolic events) and the inhospital mortality rate was in the lower range reported by prior studies [14.7% vs. 15%-20%].⁵ Nearly one-third of patients (n = 47, 27.7%) received valve surgery. Quality of care received was generally good, with most patients receiving early blood cultures, echocardiograms, early antibiotics, and timely ID consultation. We identified important gaps in care, including a failure to consult cardiac surgery in nearly half of patients and failure to consult cardiology in more than half of patients.

Our findings support work suggesting that IE is no longer primarily a chronic or subacute disease of younger patients with IVD use, positive human immunodeficiency virus status, or bicuspid aortic valves.^1,4,16,17 The International Collaboration on Endocarditis-Prospective Cohort Study,⁴ a multinational prospective cohort study (2000-2005) of 2781 adults with IE, reported a higher prevalence of patients with diabetes or on hemodialysis, IVD users, and patients with long-term venous catheter and intracardiac leads than we found. Yet both studies suggest that the demographics of IE are changing. This may partially explain why IE mortality has not improved in recent years:^2,3 patients with older age and higher comorbidity burden may not be considered good surgical candidates.

This study is among the first to contribute information on concordance with IE guidelines in a cohort of U.S. patients. Our findings suggest that most patients received timely blood culture, same-day administration of empiric antibiotics, and ID consultation, which is similar to European studies.^7,18 Guideline concordance could be improved in some areas. Overall documentation of the management plan regarding the intracardiac leads could be improved. Only 6 of 29 patients with intracardiac leads had documentation of their removal during the index hospitalization.

The 2014 AHA/ACC guidelines⁵ and the ESC guidelines⁶ emphasized the importance of multidisciplinary management of IE. As part of the Heart Valve Team at BMC, cardiologists provide expertise in diagnosis, imaging and clinical management of IE, and cardiac surgeons provide consultation on whether to pursue surgery and optimal timing of surgery. Early discussion with surgical team is considered mandatory in all complicated cases of IE.^6,18 Infectious disease consultation has been shown to improve the rate of IE diagnosis, reduce the 6-month relapse rate,¹⁹ and improve outcomes in patients with S aureus bacteremia.²⁰ In our study 86.5% of patients had documentation of an ID consultation; cardiac surgery consultation was obtained in 54.1% and cardiology consultation in 47.1% of patients.

We observed a trend towards lower rates of 6-month readmission and 12-month mortality among patients who received all 3 consults (Figure 1), despite the fact that rates of embolic events and stroke were higher in patients with 3 consults compared to those with fewer than 3. Obviously, the lack of confounder adjustment and lack of power limits our ability to make inferences about this association, but it generates hypotheses for future work. Because subjects in our study were cared for prior to 2014, multidisciplinary management of IE with involvement of cardiology, cardiac surgery, and ID physicians was observed in only one-third of patients. However, 117 (68.8%) patients received either cardiology or cardiac surgery consults. It is possible that some physicians considered involving both cardiology and cardiac surgery consultants as unnecessary and, therefore, did not consult both specialties. We will focus future QI efforts in our institution on educating physicians about the benefits of multidisciplinary care and the importance of fully implementing the 2014 AHA/ACC guidelines.

Our findings around quality of care should be placed in the context of 2 studies by González de Molina et al⁸ and Delahaye et al⁷ These studies described considerable discordance between guideline recommendations and real-world IE care. However, these studies were performed more than a decade ago and were conducted before current recommendations to consult cardiology and cardiac surgery were published.

In the 2014 AHA/ACC guidelines, surgery prior to completion of antibiotics is indicated in patients with valve dysfunction resulting in heart failure; left-sided IE caused by highly resistant organisms (including fungus or S aureus); IE complicated by heart block, aortic abscess, or penetrating lesions; and presence of persistent infection (bacteremia or fever lasting longer than 5 to 7 days) after onset of appropriate antimicrobial therapy. In addition, there is a Class IIa indication of early surgery in patients with recurrent emboli and persistent vegetation despite appropriate antibiotic therapy and a Class IIb indication of early surgery in patients with NVE with mobile vegetation greater than 10 mm in length. Surgery is recommended for patients with PVE and relapsing infection.

It is recommended that IE patients be cared for in centers with immediate access to cardiac surgery because the urgent need for surgical intervention can arise rapidly.⁵ We found that nearly one-third of included patients underwent surgery. Although we did not collect data on indications for surgery in patients who did not receive surgery, we observed that 50% had a surgery consult, suggesting the presence of 1 or more surgical indications. Of these, half underwent valve surgery. Most of the NVE patients who underwent surgery had more than 1 indication for surgery. Our surgical rate is similar to a study from Italy³ and overall in the lower range of reported surgical rate (25%-50%) from other studies.²¹ The low rate of surgery at our center may be related to the fact that the use of surgery for IE has been hotly debated in the literature,²¹ and may also be due to the low rate of cardiac surgery consultation.

Our study has several limitations. We identified eligible patients using a discharge ICD-9 coding of IE and then confirmed the presence of Duke criteria using record review. Using discharge diagnosis codes for endocarditis has been validated, and our additional manual chart review to confirm Duke criteria likely improved the specificity significantly. However, by excluding patients who did not have documented evidence of Duke criteria, we may have missed some cases, lowering sensitivity. The performance on selected quality metrics may also have been affected by our inclusion criteria. Because we included only patients who met Duke criteria, we tended to include patients who had received blood cultures and echocardiograms, which are part of the criteria. Thus, we cannot comment on use of diagnostic testing or specialty consultation in patients with suspected IE. This was a single-center study and may not represent patients or current practices seen in other institutions. We did not collect data on some of the predisposing factors to NVE (for example, baseline rheumatic heart disease or preexisting valvular heart disease) because it is estimated that less than 5% of IE in the U.S. is superimposed on rheumatic heart disease.⁴ We likely underestimated 12-month mortality rate because we did not cross-reference our findings again the National Death Index; however, this should not affect the comparison of this outcome between groups.

 

CONCLUSION

Our study confirms reports that IE epidemiology has changed significantly in recent years. It also suggests that concordance with guideline recommendations is good for some aspects of care (eg, echocardiogram, blood cultures), but can be improved in other areas, particularly in use of specialty consultation during the hospitalization. Future QI efforts should emphasize the role of the heart valve team or endocarditis team that consists of an internist, ID physician, cardiologist, cardiac surgeon, and nursing. Finally, efforts should be made to develop strategies for community hospitals that do not have access to all of these specialists (eg, early transfer, telehealth).

Disclosure

Nothing to report.

Hospital quality measures and rankings are now widely available to the public online, but is public reporting of this information an effective strategy for improving health care? Using a case study of a hospital that suffered negative publicity as a result of a quality report, this article explores the use of public reporting of performance data and pay-for-performance reimbursement strategies to foster quality improvement in the US health care system.

CASE STUDY: A SURGICAL PROGRAM GETS A BAD REPORT―IN THE HEADLINES

In September 2005, The Boston Globe ran a prominent story reporting that the UMass Memorial Medical Center in Worcester, Mass., was abruptly suspending its elective cardiac surgery program.¹ The program’s suspension came after state public health officials presented UMass Memorial with a detailed analysis showing that the hospital’s mortality rate for coronary artery bypass graft surgery (CABG) patients was the highest in the state and almost double the average for Massachusetts hospitals.¹

Key personnel from UMass Memorial described the events preceding and following the program’s suspension in a journal article published in 2008.² In 2002, UMass Memorial hired a new chief of cardiothoracic surgery, who resigned in early 2005. A few months after that resignation, state public health officials alerted the hospital to the abovementioned CABG mortality data (from 2002 and 2003), which they said would soon be reported publicly. UMass Memorial then conducted an internal review of its data from the most recent years (2004 and 2005) and found that its risk-adjusted CABG mortality had actually worsened, at which point the hospital voluntarily suspended its cardiac surgery program.²

More news stories arose about UMass Memorial’s program and its problems. The hospital hired consultants and senior surgeons from around the state and New England to completely review its cardiac surgery program. They concluded that “many essential systems were not in place” and made 68 key recommendations, including a complete overhaul of the hospital’s quality-improvement structure. The prior cardiac surgeons departed.²

The cardiac surgery program resumed after a 6-week hiatus, with day-to-day supervision by two senior cardiac surgeons from a Boston teaching hospital. A nationally recognized cardiac surgeon was brought on as chief of cardiac surgery in January 2006. In the 18 months after the program resumed, risk-adjusted CABG mortality rates declined substantially, but patient volume failed to return to presuspension levels and the hospital reported $22 million in lost revenue in fiscal year 2006 as a result of the suspension.²

This case raises a number of questions that help to frame discussion of the benefits and risks of public reporting of hospital quality measures:

• To what extent does public reporting accelerate quality improvement?
• How typical was the subsequent mortality reduction reported by UMass Memorial—ie, can public reporting be expected to improve outcomes?
• Was the effect on patient volume expected—ie, how much does public reporting affect market share?
• Would a pay-for-performance reimbursement model have accelerated improvement?
• Why do public reporting and pay-for-performance programs remain controversial?
• Do patients have a right to know?

WHAT HAS FUELED THE MOVE TOWARD PUBLIC REPORTING?

Drivers of public reporting

Massachusetts is one of a number of states that publicly report outcomes from cardiac surgery and other procedures and processes of care. Three basic factors have helped drive the development of public reporting (and, in some cases, pay-for-performance) programs:

• National policy imperatives designed to improve quality and safety and to reduce costs
• Cultural factors in society, which include consumerism in health care and the desire for transparency
• The growth of information technology and use of the World Wide Web, which has been a huge enabler of public reporting. Public reporting could be done prior to the Web era but would not have reached such a wide audience had the results been released in a book that had to be ordered from a government printing office.

The rationale for public reporting

In theory, how might public reporting and pay-for-performance programs improve quality? Several different mechanisms or factors are likely to be involved:

• Feedback. The basic premise of the National Surgical Quality Improvement Program, to cite one example, is that peer comparison and performance feedback will stimulate quality improvement.
• Reputation. Hospital personnel fear being embarrassed if data show that they are performing poorly compared with other hospitals. Likewise, in recent years we have seen hospitals with the best quality rankings publicly advertise their performance.
• Market share. Here the premise is that patients will tend to select providers with higher quality rankings and shun those with lower rankings.
• Financial incentives. Pay-for-performance programs link payment or reimbursement directly to the desired outcomes and thereby stimulate quality improvement without working through the abovementioned mechanisms.

Approaches to quality measurement

Public reporting of hospital performance requires selection of an approach to measuring quality of care. Generally speaking, measures of health care quality reflect one of three domains of care:

Structural (or environmental) aspects, such as staffing in the intensive care unit (ICU), surgical volume, or availablity of emergency medical responders. An example of a structure-oriented reporting system is the Leapfrog Group’s online posting of hospital ratings based on surgical volumes for high-risk procedures, the degree of computerized order entry implementation, and the presence or absence of various patient safety practices.³

Processes of care, such as whether beta-blockers are prescribed for all patients after a myocardial infarction (MI), or whether thromboprophylaxis measures are ordered for surgical patients in keeping with guideline recommendations. Examples of process-oriented reporting systems include the US Department of Health and Human Services’ Hospital Compare Web site⁴ and the Commonwealth Fund’s WhyNotTheBest.org site.⁵

Outcomes of care, such as rates of mortality or complications, or patient satisfaction rates. An example of an outcomes-oriented reporting system is the annual report of institution-specific hospital-acquired infection rates put out by Pennsylvania⁶ and most other states.

 

 

IS THERE EVIDENCE OF BENEFIT?

A consistent effect in spurring quality-improvement efforts

Nearly a dozen published studies have evaluated whether public reporting stimulates quality-improvement activities, and the results have shown fairly consistently that it does. A 2003 study by Hibbard et al is representative of the results.⁷ This survey-based investigation measured the number of quality-improvement activities in cardiac and obstetric care undertaken by 24 Wisconsin hospitals that were included in an existing public reporting system compared with the number undertaken by 98 other Wisconsin hospitals that received either a private report on their own quality performance (without the information being made public) or no quality report at all. The study found that the hospitals that participated in public reporting were engaged in significantly more quality-improvement activities in both of the clinical areas assessed than were the hospitals receiving private reporting or no reporting.

A mixed effect on patient outcomes

In contrast, the data on whether public reporting improves patient outcomes have so far been mixed. A 2008 systematic review of the literature identified 11 studies that addressed this issue: five studies found that public reporting had a positive effect on patient outcomes, while six studies demonstrated a negative effect or no effect.⁸ Unfortunately, the methodological quality of most studies was poor: most were before-and-after comparisons without controls.

One of the positive studies in this review examined the effects of New York State’s pioneering institution of provider-specific CABG mortality reports (provider profiling) in 1989.⁹ The analysis found that between 1987 and 1992 (during which time provider profiling was instituted), unadjusted 30-day mortality rates following bypass surgery declined to a significantly larger degree among New York Medicare patients (33% reduction) than among Medicare patients nationwide (19% reduction) (P < .001).

In contrast, a time-series study from Cleveland Health Quality Choice (CHQC)—an early and innovative public reporting program—exemplifies a case in which public reporting of hospital performance had no discernible effect.¹⁰ The study examined trends in 30-day mortality across a range of conditions over a 6-year period for 30 hospitals in the Cleveland area participating in a public reporting system. It found that the hospitals that started out in the worst-performing groups (based on baseline mortality rates) showed no significant change in mortality over time.

DOES PUBLIC REPORTING AFFECT PATIENT CHOICES?

How a high-profile bypass patient chooses a hospital

When former President Bill Clinton developed chest pain and shortness of breath in 2004, he was seen at a small community hospital in Westchester County, N.Y., and then transferred to New York-Presbyterian Hospital/Columbia University Medical Center for bypass surgery.¹¹ Although one would think President Clinton would have chosen the best hospital for CABG in New York, Presbyterian/Columbia’s risk-adjusted mortality rate for CABG was actually about twice the average for New York hospitals and one of the worst in the state, according to the most recent “report card” for New York hospitals available at the time.¹²

Why did President Clinton choose the hospital he did? Chances are that he, like most other patients, did not base his decision on publicly reported data. His choice probably was heavily influenced by the normal referral patterns of the community hospital where he was first seen.

Surveys show low patient use of data on quality...

The question raised by President Clinton’s case has been formally studied. In 1996, Schneider and Epstein surveyed patients who had recently undergone CABG in Pennsylvania (where surgeon- and hospital-specific mortality rates for cardiac surgery are publicly available) and found that fewer than 1% of patients said that provider ratings had a moderate or major impact on their choice of provider.¹³ 

The Kaiser Family Foundation regularly surveys the public about its knowledge and use of publicly available hospital comparison data. In the latest Kaiser survey, conducted in 2008,¹⁴ 41% of respondents said they believe there are “big differences” in quality among their local hospitals, yet 59% said they would choose a hospital that is familiar to them rather than a higher-rated facility. These findings may be explained, in part, by a lack of awareness that data on hospital quality are available: only 7% of survey participants said they had seen and used information comparing the quality of hospitals to make health care decisions in the prior year, and only 6% said they had seen and used information comparing physicians.

...But a trend toward greater acceptance

Although consumers’ use of publicly reported quality data remains low, their recognition of the value of such data has grown over time. Kaiser has conducted similar public surveys dating back to 1996, and the period from 1996 to 2008 saw a substantial decrease (from 72% to 59%) in the percentage of Americans who would choose a hospital based on familiarity more than on quality ratings. Similarly, the percentage of Americans who would prefer a surgeon with high quality ratings over a surgeon who has treated friends or family more than doubled from 1996 (20%) to 2008 (47%).¹⁴

What effect on market share?

Studies on the effects that public reporting has on hospital market share have been limited.

Schneider and Epstein surveyed cardiologists in Pennsylvania in 1995 and found that 87% of them said the state’s public reporting of surgeon- and hospital-specific mortality rates for CABG had no influence or minimal influence on their referral recommendations.¹⁵

Similarly, a review of New York State’s public reporting system for CABG 15 years after its launch found that hospital performance was not associated with a subsequent change in market share, not even among those hospitals with the highest mortality rate in a given year.¹⁶ Interestingly, however, this review also showed that surgeons in the bottom performance quartile were four times as likely as other surgeons to leave practice in the year following their poor report, which is one of the most prominent outcomes associated with provider profiling reported to date.

PAY-FOR-PERFORMANCE PROGRAMS

Evidence on the impact of pay-for-performance programs in the hospital setting is even more limited than that for public reporting.

Some evidence has come from the CMS/Premier Hospital Quality Incentive Demonstration, a pay-for-performance collaboration between the Centers for Medicare and Medicaid Services (CMS) and Premier, Inc., a nationwide alliance of hospitals that promotes best practices.¹⁷ The demonstration calls for hospitals that rank in the top quintile or decile for performance to receive a 1% or 2% Medicare payment bonus for five clinical focus areas: cardiac surgery, hip and knee surgery, pneumonia, heart failure, and acute MI. Performance ratings are based primarily on process measures as well as a few clinical outcome measures. Results from the first 21 months of the demonstration showed a consistent improvement in the hospitals’ composite quality scores in each of the five clinical areas.¹⁷

It is important to recognize, however, that this improvement occurred against the backdrop of broad national adoption of public reporting of hospital quality data, which makes it difficult to tease out how much of the improvement was truly attributable to pay-for-performance, especially in the absence of a control group.

To address this question, my colleagues and I evaluated adherence to quality measures over a 2-year period at 613 hospitals participating in a national public reporting initiative,¹⁸ including 207 hospitals that simultaneously took part in the CMS/Premier Hospital Quality Incentive Demonstration’s pay-for-performance program described above. We found that the hospitals participating in both public reporting and the pay-for-performance initiative achieved only modestly greater improvements in quality than did the hospitals engaged solely in public reporting; the difference amounted to only about a 1% improvement in process measures per year.

In another controlled study, Glickman et al compared quality improvement in the management of acute MI between 54 hospitals in a CMS pay-for-performance pilot project and 446 control hospitals without pay-for-performance incentives.¹⁹ They found that the pay-for-performance hospitals achieved a statistically significantly greater degree of improvement compared with control hospitals on two of six process-of-care measures (use of aspirin at discharge and smoking-cessation counseling) but not on the composite process-of-care measure. There was no significant difference between the groups in improvements in in-hospital mortality.

Why have the effects of pay-for-performance initiatives so far been so limited? It may be that the bonuses are too small and that public reporting is already effective at stimulating quality improvement, so that the incremental benefit of adding financial incentives is small. In the case of my group’s study,¹⁸ another possible factor was that the hospitals’ baseline performance on the quality measures assessed was already high—approaching or exceeding 90% on 5 of the 10 measures—thereby limiting our power to detect differences between the groups.

 

 

CONTROVERSIES AND CHALLENGES

Many issues continue to surround public reporting and pay-for-performance programs:

• Are the measures used to evaluate health care systems suitable and evidence-based? Do they truly reflect the quality of care that providers are giving?
• Do the programs encourage “teaching to the test” rather than stimulating real and comprehensive improvement? Do they make the system prone to misuse or overuse of measured services?
• How much of the variation in hospital outcomes can be explained by the current process-of-care measures?
• Should quality be measured by outcomes or processes? Outcomes matter more to patients, but they require risk adjustment to ensure valid comparisons, and risk adjustment can be difficult and expensive to conduct.
• How much is chance a factor in apparent performance differences between hospitals?
• How much is patient selection a factor? Might public reporting lead to “cherry-picking” of low-risk patients and thereby reduce access to care for other patients?

Unidirectional measures can lead to misuse, overuse

In 2003, the Infectious Diseases Society of America updated its guidelines on community-acquired pneumonia to recommend that patients receive antibiotics within 4 hours of hospital admission. This recommendation was widely adopted as an incentive-linked performance measure by CMS and other third-party payers. Kanwar et al studied the impact of this guidelines-based incentive in a pre/post study at one large teaching hospital.²⁰ They found that while significantly more patients received antibiotics in a timely fashion after publication of the guidelines (2005) versus before the guidelines (2003), almost one-third of patients receiving antibiotics in 2005 had normal chest radiographs and thus were not appropriate candidates for therapy. Moreover, significantly fewer patients in 2005 had a final diagnosis of pneumonia at discharge, and there was no difference between the two periods in rates of mortality or ICU transfer. The researchers concluded that linking the quality indicator of early antibiotic use to financial incentives may lead to misdiagnosis of pneumonia and inappropriate antibiotic use.

Of course, antibiotic timing is not the only quality measure subject to overuse or misuse; other measures pose similar risks, including prophylaxis for deep vein thrombosis, glycemic control measures, and target immunization rates.

More-nuanced measures needed

We must also consider how well reported quality measures actually reflect our objectives. For example, an evaluation of 962 hospitals’ performance in managing acute MI found that the publicly reported core process measures for acute MI (beta-blocker and aspirin at admission and discharge, ACE inhibitor at discharge, smoking-cessation counseling, timely reperfusion) together explained only 6% of the variance among the hospitals in risk-adjusted 30-day mortality.²¹ This underscores how complicated the factors affecting mortality are, and how existing process measures have only begun to scratch the surface.

How much of a role does chance play?

Another issue is the role of chance and our limited power to detect real differences in outcomes, as illustrated by an analysis by Dimick et al of all discharges from a nationally representative sample of nearly 1,000 hospitals.²² The objective was to determine whether the seven operations for which mortality is advocated as a quality indicator by the Agency for Healthcare Research and Quality are performed often enough to reliably identify hospitals with increased mortality rates. The researchers found that only for one of the seven procedures—CABG—is there sufficient caseload over a 3-year period at the majority of US hospitals to accurately detect a mortality rate twice the national average.

Although CMS is highly committed to public reporting, the comparative mortality data available on its Hospital Compare Web site are not very useful for driving consumer choice or motivating hospitals to improve. For example, of the nearly 4,500 US hospitals that reported data on 30-day mortality from MI, only 17 hospitals were considered to be better than the national average and only 7 were considered worse than the national average.⁴

CASE REVISITED: LESSONS FROM THE UMASS MEMORIAL EXPERIENCE

Returning to our case study, what can the UMass Memorial experience teach us, and how well does it reflect the literature about the usefulness of public reporting?

Did public reporting accelerate quality improvement efforts? Yes. Reporting led to the suspension of cardiac surgery and substantive reorganization, which is consistent with the literature.

Was the mortality reduction typical? No. An optimist’s view would be that the drastic actions spurred by the media coverage had strong effects. A skeptic might say that perhaps UMass Memorial did some “cherry-picking” of patients, or that they got better at coding procedures in a way that reflected more favorably on the hospital.

Were the declines in patient volumes predictable? No. So far, the data suggest that public reporting has its greatest effects on providers rather than on institutions. This may change, however, with the introduction of tiered copayments, whereby patients are asked to pay more if they get their care from lower rated institutions.

Would financial incentives have accelerated improvement? It is too early to tell. The evidence for pay-for-performance programs is limited, and the benefits demonstrated so far have been modest. But in many ways the alternative is worse: our current system of financing and paying for hospital care offers no financial incentives to hospitals for investing in the personnel or systems required to achieve better outcomes—and instead rewards (through supplemental payments) adverse outcomes.

Did prospective patients have a right to know? Despite the limitations of public reporting, one of the most compelling arguments in its favor is that patients at UMass Memorial had the right to know about the program’s outcomes. This alone may ultimately justify the expense and efforts involved. Transparency and accountability are core values of open democratic societies, and US society relies on public reporting in many other realms: the National Highway Traffic Safety Administration publicizes crash test ratings, the Securities and Exchange Commission enforces public reporting by financial institutions, and the Federal Aviation Administration reports on airline safety, timeliness of flights, and lost baggage rates.

FUTURE DIRECTIONS

In the future, we can expect more measurement and reporting of health care factors that patients care most about, such as clinical outcomes and the patient experience. It is likely that public reporting and pay-for-performance programs will address a broader range of conditions and comprise a larger number of measures. CMS has outlined plans to increase the number of publicly reported measures to more than 70 by 2010 and more than 100 by 2011. My hope is that this expansion of data, along with improved data synthesis and presentation, will foster greater use of publicly reported data. Further, the continued evolution of the Web and social networking sites is very likely to enhance public awareness of hospital performance and change the ways in which patients use these data.

 

 

DISCUSSION

Question from the audience: I’m concerned about what seems to be a unilateral effort to improve quality. There are many components of health care delivery beyond those you’ve described, including the efforts of patients, insurers, employers, and the government. The reality is that patients don’t plan for illness, insurance companies often deny care, more and more employers are providing less coverage or no coverage, and Medicare is on the road to insolvency. Is the battle for quality winnable when all these other components of delivery are failing?

Dr. Lindenauer: You make good points. But from the standpoint of professionalism, I think we have a compelling duty to constantly strive to improve the quality of care in our hospitals and practices. I have presented strategies for potentially accelerating improvements that providers are trying to make anyway. Public reporting and financial incentives are likely to be with us for a while, and their use is likely to grow. But as you said, they address only part of the problem confronting American health care.

Question from the audience: For the savvy health care consumer, is there one particular Web site for hospital or provider comparisons that you would especially recommend? Do you actually recommend using such Web sites to patients before they undergo certain procedures?

Dr. Lindenauer: I think the Hospital Compare site from the Department of Health and Human Services is the key Web site. The California Hospital Assessment and Reporting Taskforce (CHART) has a good site, and the Commonwealth Fund’s WhyNotTheBest.org is an interesting newcomer. 

However, even the most ardent advocates for public reporting wouldn’t say the information available today is sufficient for making decisions. There’s still an important role for getting recommendations from other doctors who are familiar with local hospitals and providers.

I’m optimistic that the changes that are coming to these Web sites will provide a better user experience and make it harder to ignore the results of public reporting. Today we can say, “Hospital A is better at discharge instructions or smoking cessation counseling.” But we all can appreciate how weak those kinds of measures are because their implementation is subject to local interpretations. Once risk-adjusted outcomes and more-meaningful process measures are available, I’d be surprised if more patients weren’t willing to base their decisions on published comparisons.

Hospital quality measures and rankings are now widely available to the public online, but is public reporting of this information an effective strategy for improving health care? Using a case study of a hospital that suffered negative publicity as a result of a quality report, this article explores the use of public reporting of performance data and pay-for-performance reimbursement strategies to foster quality improvement in the US health care system.

CASE STUDY: A SURGICAL PROGRAM GETS A BAD REPORT―IN THE HEADLINES

In September 2005, The Boston Globe ran a prominent story reporting that the UMass Memorial Medical Center in Worcester, Mass., was abruptly suspending its elective cardiac surgery program.¹ The program’s suspension came after state public health officials presented UMass Memorial with a detailed analysis showing that the hospital’s mortality rate for coronary artery bypass graft surgery (CABG) patients was the highest in the state and almost double the average for Massachusetts hospitals.¹

Key personnel from UMass Memorial described the events preceding and following the program’s suspension in a journal article published in 2008.² In 2002, UMass Memorial hired a new chief of cardiothoracic surgery, who resigned in early 2005. A few months after that resignation, state public health officials alerted the hospital to the abovementioned CABG mortality data (from 2002 and 2003), which they said would soon be reported publicly. UMass Memorial then conducted an internal review of its data from the most recent years (2004 and 2005) and found that its risk-adjusted CABG mortality had actually worsened, at which point the hospital voluntarily suspended its cardiac surgery program.²

More news stories arose about UMass Memorial’s program and its problems. The hospital hired consultants and senior surgeons from around the state and New England to completely review its cardiac surgery program. They concluded that “many essential systems were not in place” and made 68 key recommendations, including a complete overhaul of the hospital’s quality-improvement structure. The prior cardiac surgeons departed.²

The cardiac surgery program resumed after a 6-week hiatus, with day-to-day supervision by two senior cardiac surgeons from a Boston teaching hospital. A nationally recognized cardiac surgeon was brought on as chief of cardiac surgery in January 2006. In the 18 months after the program resumed, risk-adjusted CABG mortality rates declined substantially, but patient volume failed to return to presuspension levels and the hospital reported $22 million in lost revenue in fiscal year 2006 as a result of the suspension.²

This case raises a number of questions that help to frame discussion of the benefits and risks of public reporting of hospital quality measures:

• To what extent does public reporting accelerate quality improvement?
• How typical was the subsequent mortality reduction reported by UMass Memorial—ie, can public reporting be expected to improve outcomes?
• Was the effect on patient volume expected—ie, how much does public reporting affect market share?
• Would a pay-for-performance reimbursement model have accelerated improvement?
• Why do public reporting and pay-for-performance programs remain controversial?
• Do patients have a right to know?

WHAT HAS FUELED THE MOVE TOWARD PUBLIC REPORTING?

Drivers of public reporting

Massachusetts is one of a number of states that publicly report outcomes from cardiac surgery and other procedures and processes of care. Three basic factors have helped drive the development of public reporting (and, in some cases, pay-for-performance) programs:

• National policy imperatives designed to improve quality and safety and to reduce costs
• Cultural factors in society, which include consumerism in health care and the desire for transparency
• The growth of information technology and use of the World Wide Web, which has been a huge enabler of public reporting. Public reporting could be done prior to the Web era but would not have reached such a wide audience had the results been released in a book that had to be ordered from a government printing office.

The rationale for public reporting

In theory, how might public reporting and pay-for-performance programs improve quality? Several different mechanisms or factors are likely to be involved:

• Feedback. The basic premise of the National Surgical Quality Improvement Program, to cite one example, is that peer comparison and performance feedback will stimulate quality improvement.
• Reputation. Hospital personnel fear being embarrassed if data show that they are performing poorly compared with other hospitals. Likewise, in recent years we have seen hospitals with the best quality rankings publicly advertise their performance.
• Market share. Here the premise is that patients will tend to select providers with higher quality rankings and shun those with lower rankings.
• Financial incentives. Pay-for-performance programs link payment or reimbursement directly to the desired outcomes and thereby stimulate quality improvement without working through the abovementioned mechanisms.

Approaches to quality measurement

Public reporting of hospital performance requires selection of an approach to measuring quality of care. Generally speaking, measures of health care quality reflect one of three domains of care:

Structural (or environmental) aspects, such as staffing in the intensive care unit (ICU), surgical volume, or availablity of emergency medical responders. An example of a structure-oriented reporting system is the Leapfrog Group’s online posting of hospital ratings based on surgical volumes for high-risk procedures, the degree of computerized order entry implementation, and the presence or absence of various patient safety practices.³

Processes of care, such as whether beta-blockers are prescribed for all patients after a myocardial infarction (MI), or whether thromboprophylaxis measures are ordered for surgical patients in keeping with guideline recommendations. Examples of process-oriented reporting systems include the US Department of Health and Human Services’ Hospital Compare Web site⁴ and the Commonwealth Fund’s WhyNotTheBest.org site.⁵

Outcomes of care, such as rates of mortality or complications, or patient satisfaction rates. An example of an outcomes-oriented reporting system is the annual report of institution-specific hospital-acquired infection rates put out by Pennsylvania⁶ and most other states.

 

 

IS THERE EVIDENCE OF BENEFIT?

A consistent effect in spurring quality-improvement efforts

Nearly a dozen published studies have evaluated whether public reporting stimulates quality-improvement activities, and the results have shown fairly consistently that it does. A 2003 study by Hibbard et al is representative of the results.⁷ This survey-based investigation measured the number of quality-improvement activities in cardiac and obstetric care undertaken by 24 Wisconsin hospitals that were included in an existing public reporting system compared with the number undertaken by 98 other Wisconsin hospitals that received either a private report on their own quality performance (without the information being made public) or no quality report at all. The study found that the hospitals that participated in public reporting were engaged in significantly more quality-improvement activities in both of the clinical areas assessed than were the hospitals receiving private reporting or no reporting.

A mixed effect on patient outcomes

In contrast, the data on whether public reporting improves patient outcomes have so far been mixed. A 2008 systematic review of the literature identified 11 studies that addressed this issue: five studies found that public reporting had a positive effect on patient outcomes, while six studies demonstrated a negative effect or no effect.⁸ Unfortunately, the methodological quality of most studies was poor: most were before-and-after comparisons without controls.

One of the positive studies in this review examined the effects of New York State’s pioneering institution of provider-specific CABG mortality reports (provider profiling) in 1989.⁹ The analysis found that between 1987 and 1992 (during which time provider profiling was instituted), unadjusted 30-day mortality rates following bypass surgery declined to a significantly larger degree among New York Medicare patients (33% reduction) than among Medicare patients nationwide (19% reduction) (P < .001).

In contrast, a time-series study from Cleveland Health Quality Choice (CHQC)—an early and innovative public reporting program—exemplifies a case in which public reporting of hospital performance had no discernible effect.¹⁰ The study examined trends in 30-day mortality across a range of conditions over a 6-year period for 30 hospitals in the Cleveland area participating in a public reporting system. It found that the hospitals that started out in the worst-performing groups (based on baseline mortality rates) showed no significant change in mortality over time.

DOES PUBLIC REPORTING AFFECT PATIENT CHOICES?

How a high-profile bypass patient chooses a hospital

When former President Bill Clinton developed chest pain and shortness of breath in 2004, he was seen at a small community hospital in Westchester County, N.Y., and then transferred to New York-Presbyterian Hospital/Columbia University Medical Center for bypass surgery.¹¹ Although one would think President Clinton would have chosen the best hospital for CABG in New York, Presbyterian/Columbia’s risk-adjusted mortality rate for CABG was actually about twice the average for New York hospitals and one of the worst in the state, according to the most recent “report card” for New York hospitals available at the time.¹²

Why did President Clinton choose the hospital he did? Chances are that he, like most other patients, did not base his decision on publicly reported data. His choice probably was heavily influenced by the normal referral patterns of the community hospital where he was first seen.

Surveys show low patient use of data on quality...

The question raised by President Clinton’s case has been formally studied. In 1996, Schneider and Epstein surveyed patients who had recently undergone CABG in Pennsylvania (where surgeon- and hospital-specific mortality rates for cardiac surgery are publicly available) and found that fewer than 1% of patients said that provider ratings had a moderate or major impact on their choice of provider.¹³ 

The Kaiser Family Foundation regularly surveys the public about its knowledge and use of publicly available hospital comparison data. In the latest Kaiser survey, conducted in 2008,¹⁴ 41% of respondents said they believe there are “big differences” in quality among their local hospitals, yet 59% said they would choose a hospital that is familiar to them rather than a higher-rated facility. These findings may be explained, in part, by a lack of awareness that data on hospital quality are available: only 7% of survey participants said they had seen and used information comparing the quality of hospitals to make health care decisions in the prior year, and only 6% said they had seen and used information comparing physicians.

...But a trend toward greater acceptance

Although consumers’ use of publicly reported quality data remains low, their recognition of the value of such data has grown over time. Kaiser has conducted similar public surveys dating back to 1996, and the period from 1996 to 2008 saw a substantial decrease (from 72% to 59%) in the percentage of Americans who would choose a hospital based on familiarity more than on quality ratings. Similarly, the percentage of Americans who would prefer a surgeon with high quality ratings over a surgeon who has treated friends or family more than doubled from 1996 (20%) to 2008 (47%).¹⁴

What effect on market share?

Studies on the effects that public reporting has on hospital market share have been limited.

Schneider and Epstein surveyed cardiologists in Pennsylvania in 1995 and found that 87% of them said the state’s public reporting of surgeon- and hospital-specific mortality rates for CABG had no influence or minimal influence on their referral recommendations.¹⁵

Similarly, a review of New York State’s public reporting system for CABG 15 years after its launch found that hospital performance was not associated with a subsequent change in market share, not even among those hospitals with the highest mortality rate in a given year.¹⁶ Interestingly, however, this review also showed that surgeons in the bottom performance quartile were four times as likely as other surgeons to leave practice in the year following their poor report, which is one of the most prominent outcomes associated with provider profiling reported to date.

PAY-FOR-PERFORMANCE PROGRAMS

Evidence on the impact of pay-for-performance programs in the hospital setting is even more limited than that for public reporting.

Some evidence has come from the CMS/Premier Hospital Quality Incentive Demonstration, a pay-for-performance collaboration between the Centers for Medicare and Medicaid Services (CMS) and Premier, Inc., a nationwide alliance of hospitals that promotes best practices.¹⁷ The demonstration calls for hospitals that rank in the top quintile or decile for performance to receive a 1% or 2% Medicare payment bonus for five clinical focus areas: cardiac surgery, hip and knee surgery, pneumonia, heart failure, and acute MI. Performance ratings are based primarily on process measures as well as a few clinical outcome measures. Results from the first 21 months of the demonstration showed a consistent improvement in the hospitals’ composite quality scores in each of the five clinical areas.¹⁷

It is important to recognize, however, that this improvement occurred against the backdrop of broad national adoption of public reporting of hospital quality data, which makes it difficult to tease out how much of the improvement was truly attributable to pay-for-performance, especially in the absence of a control group.

To address this question, my colleagues and I evaluated adherence to quality measures over a 2-year period at 613 hospitals participating in a national public reporting initiative,¹⁸ including 207 hospitals that simultaneously took part in the CMS/Premier Hospital Quality Incentive Demonstration’s pay-for-performance program described above. We found that the hospitals participating in both public reporting and the pay-for-performance initiative achieved only modestly greater improvements in quality than did the hospitals engaged solely in public reporting; the difference amounted to only about a 1% improvement in process measures per year.

In another controlled study, Glickman et al compared quality improvement in the management of acute MI between 54 hospitals in a CMS pay-for-performance pilot project and 446 control hospitals without pay-for-performance incentives.¹⁹ They found that the pay-for-performance hospitals achieved a statistically significantly greater degree of improvement compared with control hospitals on two of six process-of-care measures (use of aspirin at discharge and smoking-cessation counseling) but not on the composite process-of-care measure. There was no significant difference between the groups in improvements in in-hospital mortality.

Why have the effects of pay-for-performance initiatives so far been so limited? It may be that the bonuses are too small and that public reporting is already effective at stimulating quality improvement, so that the incremental benefit of adding financial incentives is small. In the case of my group’s study,¹⁸ another possible factor was that the hospitals’ baseline performance on the quality measures assessed was already high—approaching or exceeding 90% on 5 of the 10 measures—thereby limiting our power to detect differences between the groups.

 

 

CONTROVERSIES AND CHALLENGES

Many issues continue to surround public reporting and pay-for-performance programs:

• Are the measures used to evaluate health care systems suitable and evidence-based? Do they truly reflect the quality of care that providers are giving?
• Do the programs encourage “teaching to the test” rather than stimulating real and comprehensive improvement? Do they make the system prone to misuse or overuse of measured services?
• How much of the variation in hospital outcomes can be explained by the current process-of-care measures?
• Should quality be measured by outcomes or processes? Outcomes matter more to patients, but they require risk adjustment to ensure valid comparisons, and risk adjustment can be difficult and expensive to conduct.
• How much is chance a factor in apparent performance differences between hospitals?
• How much is patient selection a factor? Might public reporting lead to “cherry-picking” of low-risk patients and thereby reduce access to care for other patients?

Unidirectional measures can lead to misuse, overuse

In 2003, the Infectious Diseases Society of America updated its guidelines on community-acquired pneumonia to recommend that patients receive antibiotics within 4 hours of hospital admission. This recommendation was widely adopted as an incentive-linked performance measure by CMS and other third-party payers. Kanwar et al studied the impact of this guidelines-based incentive in a pre/post study at one large teaching hospital.²⁰ They found that while significantly more patients received antibiotics in a timely fashion after publication of the guidelines (2005) versus before the guidelines (2003), almost one-third of patients receiving antibiotics in 2005 had normal chest radiographs and thus were not appropriate candidates for therapy. Moreover, significantly fewer patients in 2005 had a final diagnosis of pneumonia at discharge, and there was no difference between the two periods in rates of mortality or ICU transfer. The researchers concluded that linking the quality indicator of early antibiotic use to financial incentives may lead to misdiagnosis of pneumonia and inappropriate antibiotic use.

Of course, antibiotic timing is not the only quality measure subject to overuse or misuse; other measures pose similar risks, including prophylaxis for deep vein thrombosis, glycemic control measures, and target immunization rates.

More-nuanced measures needed

We must also consider how well reported quality measures actually reflect our objectives. For example, an evaluation of 962 hospitals’ performance in managing acute MI found that the publicly reported core process measures for acute MI (beta-blocker and aspirin at admission and discharge, ACE inhibitor at discharge, smoking-cessation counseling, timely reperfusion) together explained only 6% of the variance among the hospitals in risk-adjusted 30-day mortality.²¹ This underscores how complicated the factors affecting mortality are, and how existing process measures have only begun to scratch the surface.

How much of a role does chance play?

Another issue is the role of chance and our limited power to detect real differences in outcomes, as illustrated by an analysis by Dimick et al of all discharges from a nationally representative sample of nearly 1,000 hospitals.²² The objective was to determine whether the seven operations for which mortality is advocated as a quality indicator by the Agency for Healthcare Research and Quality are performed often enough to reliably identify hospitals with increased mortality rates. The researchers found that only for one of the seven procedures—CABG—is there sufficient caseload over a 3-year period at the majority of US hospitals to accurately detect a mortality rate twice the national average.

Although CMS is highly committed to public reporting, the comparative mortality data available on its Hospital Compare Web site are not very useful for driving consumer choice or motivating hospitals to improve. For example, of the nearly 4,500 US hospitals that reported data on 30-day mortality from MI, only 17 hospitals were considered to be better than the national average and only 7 were considered worse than the national average.⁴

CASE REVISITED: LESSONS FROM THE UMASS MEMORIAL EXPERIENCE

Returning to our case study, what can the UMass Memorial experience teach us, and how well does it reflect the literature about the usefulness of public reporting?

Did public reporting accelerate quality improvement efforts? Yes. Reporting led to the suspension of cardiac surgery and substantive reorganization, which is consistent with the literature.

Was the mortality reduction typical? No. An optimist’s view would be that the drastic actions spurred by the media coverage had strong effects. A skeptic might say that perhaps UMass Memorial did some “cherry-picking” of patients, or that they got better at coding procedures in a way that reflected more favorably on the hospital.

Were the declines in patient volumes predictable? No. So far, the data suggest that public reporting has its greatest effects on providers rather than on institutions. This may change, however, with the introduction of tiered copayments, whereby patients are asked to pay more if they get their care from lower rated institutions.

Would financial incentives have accelerated improvement? It is too early to tell. The evidence for pay-for-performance programs is limited, and the benefits demonstrated so far have been modest. But in many ways the alternative is worse: our current system of financing and paying for hospital care offers no financial incentives to hospitals for investing in the personnel or systems required to achieve better outcomes—and instead rewards (through supplemental payments) adverse outcomes.

Did prospective patients have a right to know? Despite the limitations of public reporting, one of the most compelling arguments in its favor is that patients at UMass Memorial had the right to know about the program’s outcomes. This alone may ultimately justify the expense and efforts involved. Transparency and accountability are core values of open democratic societies, and US society relies on public reporting in many other realms: the National Highway Traffic Safety Administration publicizes crash test ratings, the Securities and Exchange Commission enforces public reporting by financial institutions, and the Federal Aviation Administration reports on airline safety, timeliness of flights, and lost baggage rates.

FUTURE DIRECTIONS

In the future, we can expect more measurement and reporting of health care factors that patients care most about, such as clinical outcomes and the patient experience. It is likely that public reporting and pay-for-performance programs will address a broader range of conditions and comprise a larger number of measures. CMS has outlined plans to increase the number of publicly reported measures to more than 70 by 2010 and more than 100 by 2011. My hope is that this expansion of data, along with improved data synthesis and presentation, will foster greater use of publicly reported data. Further, the continued evolution of the Web and social networking sites is very likely to enhance public awareness of hospital performance and change the ways in which patients use these data.

 

 

DISCUSSION

Question from the audience: I’m concerned about what seems to be a unilateral effort to improve quality. There are many components of health care delivery beyond those you’ve described, including the efforts of patients, insurers, employers, and the government. The reality is that patients don’t plan for illness, insurance companies often deny care, more and more employers are providing less coverage or no coverage, and Medicare is on the road to insolvency. Is the battle for quality winnable when all these other components of delivery are failing?

Dr. Lindenauer: You make good points. But from the standpoint of professionalism, I think we have a compelling duty to constantly strive to improve the quality of care in our hospitals and practices. I have presented strategies for potentially accelerating improvements that providers are trying to make anyway. Public reporting and financial incentives are likely to be with us for a while, and their use is likely to grow. But as you said, they address only part of the problem confronting American health care.

Question from the audience: For the savvy health care consumer, is there one particular Web site for hospital or provider comparisons that you would especially recommend? Do you actually recommend using such Web sites to patients before they undergo certain procedures?

Dr. Lindenauer: I think the Hospital Compare site from the Department of Health and Human Services is the key Web site. The California Hospital Assessment and Reporting Taskforce (CHART) has a good site, and the Commonwealth Fund’s WhyNotTheBest.org is an interesting newcomer. 

However, even the most ardent advocates for public reporting wouldn’t say the information available today is sufficient for making decisions. There’s still an important role for getting recommendations from other doctors who are familiar with local hospitals and providers.

I’m optimistic that the changes that are coming to these Web sites will provide a better user experience and make it harder to ignore the results of public reporting. Today we can say, “Hospital A is better at discharge instructions or smoking cessation counseling.” But we all can appreciate how weak those kinds of measures are because their implementation is subject to local interpretations. Once risk-adjusted outcomes and more-meaningful process measures are available, I’d be surprised if more patients weren’t willing to base their decisions on published comparisons.

Hospital quality measures and rankings are now widely available to the public online, but is public reporting of this information an effective strategy for improving health care? Using a case study of a hospital that suffered negative publicity as a result of a quality report, this article explores the use of public reporting of performance data and pay-for-performance reimbursement strategies to foster quality improvement in the US health care system.

CASE STUDY: A SURGICAL PROGRAM GETS A BAD REPORT―IN THE HEADLINES

In September 2005, The Boston Globe ran a prominent story reporting that the UMass Memorial Medical Center in Worcester, Mass., was abruptly suspending its elective cardiac surgery program.¹ The program’s suspension came after state public health officials presented UMass Memorial with a detailed analysis showing that the hospital’s mortality rate for coronary artery bypass graft surgery (CABG) patients was the highest in the state and almost double the average for Massachusetts hospitals.¹

Key personnel from UMass Memorial described the events preceding and following the program’s suspension in a journal article published in 2008.² In 2002, UMass Memorial hired a new chief of cardiothoracic surgery, who resigned in early 2005. A few months after that resignation, state public health officials alerted the hospital to the abovementioned CABG mortality data (from 2002 and 2003), which they said would soon be reported publicly. UMass Memorial then conducted an internal review of its data from the most recent years (2004 and 2005) and found that its risk-adjusted CABG mortality had actually worsened, at which point the hospital voluntarily suspended its cardiac surgery program.²

More news stories arose about UMass Memorial’s program and its problems. The hospital hired consultants and senior surgeons from around the state and New England to completely review its cardiac surgery program. They concluded that “many essential systems were not in place” and made 68 key recommendations, including a complete overhaul of the hospital’s quality-improvement structure. The prior cardiac surgeons departed.²

The cardiac surgery program resumed after a 6-week hiatus, with day-to-day supervision by two senior cardiac surgeons from a Boston teaching hospital. A nationally recognized cardiac surgeon was brought on as chief of cardiac surgery in January 2006. In the 18 months after the program resumed, risk-adjusted CABG mortality rates declined substantially, but patient volume failed to return to presuspension levels and the hospital reported $22 million in lost revenue in fiscal year 2006 as a result of the suspension.²

This case raises a number of questions that help to frame discussion of the benefits and risks of public reporting of hospital quality measures:

• To what extent does public reporting accelerate quality improvement?
• How typical was the subsequent mortality reduction reported by UMass Memorial—ie, can public reporting be expected to improve outcomes?
• Was the effect on patient volume expected—ie, how much does public reporting affect market share?
• Would a pay-for-performance reimbursement model have accelerated improvement?
• Why do public reporting and pay-for-performance programs remain controversial?
• Do patients have a right to know?

WHAT HAS FUELED THE MOVE TOWARD PUBLIC REPORTING?

Drivers of public reporting

Massachusetts is one of a number of states that publicly report outcomes from cardiac surgery and other procedures and processes of care. Three basic factors have helped drive the development of public reporting (and, in some cases, pay-for-performance) programs:

• National policy imperatives designed to improve quality and safety and to reduce costs
• Cultural factors in society, which include consumerism in health care and the desire for transparency
• The growth of information technology and use of the World Wide Web, which has been a huge enabler of public reporting. Public reporting could be done prior to the Web era but would not have reached such a wide audience had the results been released in a book that had to be ordered from a government printing office.

The rationale for public reporting

In theory, how might public reporting and pay-for-performance programs improve quality? Several different mechanisms or factors are likely to be involved:

• Feedback. The basic premise of the National Surgical Quality Improvement Program, to cite one example, is that peer comparison and performance feedback will stimulate quality improvement.
• Reputation. Hospital personnel fear being embarrassed if data show that they are performing poorly compared with other hospitals. Likewise, in recent years we have seen hospitals with the best quality rankings publicly advertise their performance.
• Market share. Here the premise is that patients will tend to select providers with higher quality rankings and shun those with lower rankings.
• Financial incentives. Pay-for-performance programs link payment or reimbursement directly to the desired outcomes and thereby stimulate quality improvement without working through the abovementioned mechanisms.

Approaches to quality measurement

Public reporting of hospital performance requires selection of an approach to measuring quality of care. Generally speaking, measures of health care quality reflect one of three domains of care:

Structural (or environmental) aspects, such as staffing in the intensive care unit (ICU), surgical volume, or availablity of emergency medical responders. An example of a structure-oriented reporting system is the Leapfrog Group’s online posting of hospital ratings based on surgical volumes for high-risk procedures, the degree of computerized order entry implementation, and the presence or absence of various patient safety practices.³

Processes of care, such as whether beta-blockers are prescribed for all patients after a myocardial infarction (MI), or whether thromboprophylaxis measures are ordered for surgical patients in keeping with guideline recommendations. Examples of process-oriented reporting systems include the US Department of Health and Human Services’ Hospital Compare Web site⁴ and the Commonwealth Fund’s WhyNotTheBest.org site.⁵

Outcomes of care, such as rates of mortality or complications, or patient satisfaction rates. An example of an outcomes-oriented reporting system is the annual report of institution-specific hospital-acquired infection rates put out by Pennsylvania⁶ and most other states.

 

 

IS THERE EVIDENCE OF BENEFIT?

A consistent effect in spurring quality-improvement efforts

Nearly a dozen published studies have evaluated whether public reporting stimulates quality-improvement activities, and the results have shown fairly consistently that it does. A 2003 study by Hibbard et al is representative of the results.⁷ This survey-based investigation measured the number of quality-improvement activities in cardiac and obstetric care undertaken by 24 Wisconsin hospitals that were included in an existing public reporting system compared with the number undertaken by 98 other Wisconsin hospitals that received either a private report on their own quality performance (without the information being made public) or no quality report at all. The study found that the hospitals that participated in public reporting were engaged in significantly more quality-improvement activities in both of the clinical areas assessed than were the hospitals receiving private reporting or no reporting.

A mixed effect on patient outcomes

In contrast, the data on whether public reporting improves patient outcomes have so far been mixed. A 2008 systematic review of the literature identified 11 studies that addressed this issue: five studies found that public reporting had a positive effect on patient outcomes, while six studies demonstrated a negative effect or no effect.⁸ Unfortunately, the methodological quality of most studies was poor: most were before-and-after comparisons without controls.

One of the positive studies in this review examined the effects of New York State’s pioneering institution of provider-specific CABG mortality reports (provider profiling) in 1989.⁹ The analysis found that between 1987 and 1992 (during which time provider profiling was instituted), unadjusted 30-day mortality rates following bypass surgery declined to a significantly larger degree among New York Medicare patients (33% reduction) than among Medicare patients nationwide (19% reduction) (P < .001).

In contrast, a time-series study from Cleveland Health Quality Choice (CHQC)—an early and innovative public reporting program—exemplifies a case in which public reporting of hospital performance had no discernible effect.¹⁰ The study examined trends in 30-day mortality across a range of conditions over a 6-year period for 30 hospitals in the Cleveland area participating in a public reporting system. It found that the hospitals that started out in the worst-performing groups (based on baseline mortality rates) showed no significant change in mortality over time.

DOES PUBLIC REPORTING AFFECT PATIENT CHOICES?

How a high-profile bypass patient chooses a hospital

When former President Bill Clinton developed chest pain and shortness of breath in 2004, he was seen at a small community hospital in Westchester County, N.Y., and then transferred to New York-Presbyterian Hospital/Columbia University Medical Center for bypass surgery.¹¹ Although one would think President Clinton would have chosen the best hospital for CABG in New York, Presbyterian/Columbia’s risk-adjusted mortality rate for CABG was actually about twice the average for New York hospitals and one of the worst in the state, according to the most recent “report card” for New York hospitals available at the time.¹²

Why did President Clinton choose the hospital he did? Chances are that he, like most other patients, did not base his decision on publicly reported data. His choice probably was heavily influenced by the normal referral patterns of the community hospital where he was first seen.

Surveys show low patient use of data on quality...

The question raised by President Clinton’s case has been formally studied. In 1996, Schneider and Epstein surveyed patients who had recently undergone CABG in Pennsylvania (where surgeon- and hospital-specific mortality rates for cardiac surgery are publicly available) and found that fewer than 1% of patients said that provider ratings had a moderate or major impact on their choice of provider.¹³ 

The Kaiser Family Foundation regularly surveys the public about its knowledge and use of publicly available hospital comparison data. In the latest Kaiser survey, conducted in 2008,¹⁴ 41% of respondents said they believe there are “big differences” in quality among their local hospitals, yet 59% said they would choose a hospital that is familiar to them rather than a higher-rated facility. These findings may be explained, in part, by a lack of awareness that data on hospital quality are available: only 7% of survey participants said they had seen and used information comparing the quality of hospitals to make health care decisions in the prior year, and only 6% said they had seen and used information comparing physicians.

...But a trend toward greater acceptance

Although consumers’ use of publicly reported quality data remains low, their recognition of the value of such data has grown over time. Kaiser has conducted similar public surveys dating back to 1996, and the period from 1996 to 2008 saw a substantial decrease (from 72% to 59%) in the percentage of Americans who would choose a hospital based on familiarity more than on quality ratings. Similarly, the percentage of Americans who would prefer a surgeon with high quality ratings over a surgeon who has treated friends or family more than doubled from 1996 (20%) to 2008 (47%).¹⁴

What effect on market share?

Studies on the effects that public reporting has on hospital market share have been limited.

Schneider and Epstein surveyed cardiologists in Pennsylvania in 1995 and found that 87% of them said the state’s public reporting of surgeon- and hospital-specific mortality rates for CABG had no influence or minimal influence on their referral recommendations.¹⁵

Similarly, a review of New York State’s public reporting system for CABG 15 years after its launch found that hospital performance was not associated with a subsequent change in market share, not even among those hospitals with the highest mortality rate in a given year.¹⁶ Interestingly, however, this review also showed that surgeons in the bottom performance quartile were four times as likely as other surgeons to leave practice in the year following their poor report, which is one of the most prominent outcomes associated with provider profiling reported to date.

PAY-FOR-PERFORMANCE PROGRAMS

Evidence on the impact of pay-for-performance programs in the hospital setting is even more limited than that for public reporting.

Some evidence has come from the CMS/Premier Hospital Quality Incentive Demonstration, a pay-for-performance collaboration between the Centers for Medicare and Medicaid Services (CMS) and Premier, Inc., a nationwide alliance of hospitals that promotes best practices.¹⁷ The demonstration calls for hospitals that rank in the top quintile or decile for performance to receive a 1% or 2% Medicare payment bonus for five clinical focus areas: cardiac surgery, hip and knee surgery, pneumonia, heart failure, and acute MI. Performance ratings are based primarily on process measures as well as a few clinical outcome measures. Results from the first 21 months of the demonstration showed a consistent improvement in the hospitals’ composite quality scores in each of the five clinical areas.¹⁷

It is important to recognize, however, that this improvement occurred against the backdrop of broad national adoption of public reporting of hospital quality data, which makes it difficult to tease out how much of the improvement was truly attributable to pay-for-performance, especially in the absence of a control group.

To address this question, my colleagues and I evaluated adherence to quality measures over a 2-year period at 613 hospitals participating in a national public reporting initiative,¹⁸ including 207 hospitals that simultaneously took part in the CMS/Premier Hospital Quality Incentive Demonstration’s pay-for-performance program described above. We found that the hospitals participating in both public reporting and the pay-for-performance initiative achieved only modestly greater improvements in quality than did the hospitals engaged solely in public reporting; the difference amounted to only about a 1% improvement in process measures per year.

In another controlled study, Glickman et al compared quality improvement in the management of acute MI between 54 hospitals in a CMS pay-for-performance pilot project and 446 control hospitals without pay-for-performance incentives.¹⁹ They found that the pay-for-performance hospitals achieved a statistically significantly greater degree of improvement compared with control hospitals on two of six process-of-care measures (use of aspirin at discharge and smoking-cessation counseling) but not on the composite process-of-care measure. There was no significant difference between the groups in improvements in in-hospital mortality.

Why have the effects of pay-for-performance initiatives so far been so limited? It may be that the bonuses are too small and that public reporting is already effective at stimulating quality improvement, so that the incremental benefit of adding financial incentives is small. In the case of my group’s study,¹⁸ another possible factor was that the hospitals’ baseline performance on the quality measures assessed was already high—approaching or exceeding 90% on 5 of the 10 measures—thereby limiting our power to detect differences between the groups.

 

 

CONTROVERSIES AND CHALLENGES

Many issues continue to surround public reporting and pay-for-performance programs:

• Are the measures used to evaluate health care systems suitable and evidence-based? Do they truly reflect the quality of care that providers are giving?
• Do the programs encourage “teaching to the test” rather than stimulating real and comprehensive improvement? Do they make the system prone to misuse or overuse of measured services?
• How much of the variation in hospital outcomes can be explained by the current process-of-care measures?
• Should quality be measured by outcomes or processes? Outcomes matter more to patients, but they require risk adjustment to ensure valid comparisons, and risk adjustment can be difficult and expensive to conduct.
• How much is chance a factor in apparent performance differences between hospitals?
• How much is patient selection a factor? Might public reporting lead to “cherry-picking” of low-risk patients and thereby reduce access to care for other patients?

Unidirectional measures can lead to misuse, overuse

In 2003, the Infectious Diseases Society of America updated its guidelines on community-acquired pneumonia to recommend that patients receive antibiotics within 4 hours of hospital admission. This recommendation was widely adopted as an incentive-linked performance measure by CMS and other third-party payers. Kanwar et al studied the impact of this guidelines-based incentive in a pre/post study at one large teaching hospital.²⁰ They found that while significantly more patients received antibiotics in a timely fashion after publication of the guidelines (2005) versus before the guidelines (2003), almost one-third of patients receiving antibiotics in 2005 had normal chest radiographs and thus were not appropriate candidates for therapy. Moreover, significantly fewer patients in 2005 had a final diagnosis of pneumonia at discharge, and there was no difference between the two periods in rates of mortality or ICU transfer. The researchers concluded that linking the quality indicator of early antibiotic use to financial incentives may lead to misdiagnosis of pneumonia and inappropriate antibiotic use.

Of course, antibiotic timing is not the only quality measure subject to overuse or misuse; other measures pose similar risks, including prophylaxis for deep vein thrombosis, glycemic control measures, and target immunization rates.

More-nuanced measures needed

We must also consider how well reported quality measures actually reflect our objectives. For example, an evaluation of 962 hospitals’ performance in managing acute MI found that the publicly reported core process measures for acute MI (beta-blocker and aspirin at admission and discharge, ACE inhibitor at discharge, smoking-cessation counseling, timely reperfusion) together explained only 6% of the variance among the hospitals in risk-adjusted 30-day mortality.²¹ This underscores how complicated the factors affecting mortality are, and how existing process measures have only begun to scratch the surface.

How much of a role does chance play?

Another issue is the role of chance and our limited power to detect real differences in outcomes, as illustrated by an analysis by Dimick et al of all discharges from a nationally representative sample of nearly 1,000 hospitals.²² The objective was to determine whether the seven operations for which mortality is advocated as a quality indicator by the Agency for Healthcare Research and Quality are performed often enough to reliably identify hospitals with increased mortality rates. The researchers found that only for one of the seven procedures—CABG—is there sufficient caseload over a 3-year period at the majority of US hospitals to accurately detect a mortality rate twice the national average.

Although CMS is highly committed to public reporting, the comparative mortality data available on its Hospital Compare Web site are not very useful for driving consumer choice or motivating hospitals to improve. For example, of the nearly 4,500 US hospitals that reported data on 30-day mortality from MI, only 17 hospitals were considered to be better than the national average and only 7 were considered worse than the national average.⁴

CASE REVISITED: LESSONS FROM THE UMASS MEMORIAL EXPERIENCE

Returning to our case study, what can the UMass Memorial experience teach us, and how well does it reflect the literature about the usefulness of public reporting?

Did public reporting accelerate quality improvement efforts? Yes. Reporting led to the suspension of cardiac surgery and substantive reorganization, which is consistent with the literature.

Was the mortality reduction typical? No. An optimist’s view would be that the drastic actions spurred by the media coverage had strong effects. A skeptic might say that perhaps UMass Memorial did some “cherry-picking” of patients, or that they got better at coding procedures in a way that reflected more favorably on the hospital.

Were the declines in patient volumes predictable? No. So far, the data suggest that public reporting has its greatest effects on providers rather than on institutions. This may change, however, with the introduction of tiered copayments, whereby patients are asked to pay more if they get their care from lower rated institutions.

Would financial incentives have accelerated improvement? It is too early to tell. The evidence for pay-for-performance programs is limited, and the benefits demonstrated so far have been modest. But in many ways the alternative is worse: our current system of financing and paying for hospital care offers no financial incentives to hospitals for investing in the personnel or systems required to achieve better outcomes—and instead rewards (through supplemental payments) adverse outcomes.

Did prospective patients have a right to know? Despite the limitations of public reporting, one of the most compelling arguments in its favor is that patients at UMass Memorial had the right to know about the program’s outcomes. This alone may ultimately justify the expense and efforts involved. Transparency and accountability are core values of open democratic societies, and US society relies on public reporting in many other realms: the National Highway Traffic Safety Administration publicizes crash test ratings, the Securities and Exchange Commission enforces public reporting by financial institutions, and the Federal Aviation Administration reports on airline safety, timeliness of flights, and lost baggage rates.

FUTURE DIRECTIONS

In the future, we can expect more measurement and reporting of health care factors that patients care most about, such as clinical outcomes and the patient experience. It is likely that public reporting and pay-for-performance programs will address a broader range of conditions and comprise a larger number of measures. CMS has outlined plans to increase the number of publicly reported measures to more than 70 by 2010 and more than 100 by 2011. My hope is that this expansion of data, along with improved data synthesis and presentation, will foster greater use of publicly reported data. Further, the continued evolution of the Web and social networking sites is very likely to enhance public awareness of hospital performance and change the ways in which patients use these data.

 

 

DISCUSSION

Question from the audience: I’m concerned about what seems to be a unilateral effort to improve quality. There are many components of health care delivery beyond those you’ve described, including the efforts of patients, insurers, employers, and the government. The reality is that patients don’t plan for illness, insurance companies often deny care, more and more employers are providing less coverage or no coverage, and Medicare is on the road to insolvency. Is the battle for quality winnable when all these other components of delivery are failing?

Dr. Lindenauer: You make good points. But from the standpoint of professionalism, I think we have a compelling duty to constantly strive to improve the quality of care in our hospitals and practices. I have presented strategies for potentially accelerating improvements that providers are trying to make anyway. Public reporting and financial incentives are likely to be with us for a while, and their use is likely to grow. But as you said, they address only part of the problem confronting American health care.

Question from the audience: For the savvy health care consumer, is there one particular Web site for hospital or provider comparisons that you would especially recommend? Do you actually recommend using such Web sites to patients before they undergo certain procedures?

Dr. Lindenauer: I think the Hospital Compare site from the Department of Health and Human Services is the key Web site. The California Hospital Assessment and Reporting Taskforce (CHART) has a good site, and the Commonwealth Fund’s WhyNotTheBest.org is an interesting newcomer. 

However, even the most ardent advocates for public reporting wouldn’t say the information available today is sufficient for making decisions. There’s still an important role for getting recommendations from other doctors who are familiar with local hospitals and providers.

I’m optimistic that the changes that are coming to these Web sites will provide a better user experience and make it harder to ignore the results of public reporting. Today we can say, “Hospital A is better at discharge instructions or smoking cessation counseling.” But we all can appreciate how weak those kinds of measures are because their implementation is subject to local interpretations. Once risk-adjusted outcomes and more-meaningful process measures are available, I’d be surprised if more patients weren’t willing to base their decisions on published comparisons.