Disparities in National Hospital Ratings Systems Produce No Clear Winners, Losers

A recent study found different approaches used by four popular hospital ratings systems resulted in disagreement about the ranking of many U.S. hospitals.

"Only 10% of hospitals that were rated as a high performer on one of the systems were rated as a high performer on another rating system," says lead author John Matthew Austin, MS, PhD, assistant professor of anesthesiology and critical care medicine at Johns Hopkins University School of Medicine in Baltimore. "There was no one hospital that was rated as a high performer on all four."

The study, which appeared in Health Affairs, looked at the hospital ratings systems of U.S. News, Healthgrades, The Leapfrog Group, and Consumer Reports, and found none took the same approach to assessing hospital quality. Of the 83 hospitals rated by all four systems, none were universally recognized as either a high performer or a low performer.

"I think the impact, or the influence, on consumers is that these conflicting ratings could generate confusion," Dr. Austin says. "Depending on which rating system you look at, it may give you a different answer on which hospital or where you should seek care. If you look at these four rating systems in a community, you may actually wind up being directed to four different hospitals."

David Pressel, MD, PhD, medical director of inpatient care at Alfred I. duPont Hospital for Children in Wilmington, Del., emphasized the difficulty of defining quality.

"Measuring quality is really difficult and hard, and people, physicians, and hospitals struggle with this," he explains. "I think it's very important that people recognize in rating systems there’s always going to be a top 10% or top 50% and a bottom 10% or 50%, and what's really important is not if you're in the top or bottom but what the scatter of the data is. If the scatter of the data is very narrow, they may not be providing much worse care than the top hospitals. I think that is lost on the public."

Dr. Austin's study identified varying missions and methodologies for each ratings system.

"U.S. News' Best Hospitals is actually intending to identify the best medical centers for the most complicated cases," Dr. Austin says. "The Leapfrog Hospital Safety Score has a very laser focus on patient safety, so freedom from harm, things like errors, infections."

Dr. Austin also noticed that some rating systems were more descriptive about their methods than others.

"Some of the ratings are much more transparent in what they share, in terms of how hospitals are rated, and others are less clear," he notes. "Healthgrades lists the top 100 hospitals. They're supposedly looking at hospital outcomes, but they don't publicly make their methodology available in terms of their risk-adjustment levels."

The rating systems also communicated their ratings differently. Leapfrog issues letter grades A–F; Consumer Reports and U.S. News issue scores from 0–100; and Healthgrades identifies the top 50 and top 100 hospitals but doesn’t rank hospitals, and hospitals that are not in the top 100 are not rated at all.

The study authors outlined possible improvements to eliminate some of these disparities, including reaching out to the sponsoring organizations and encouraging them to be more transparent about their ratings to allow for easier patient interpretation.

"Patients should understand what's being measured," Dr. Austin says. "Hospitals should be able to duplicate their ratings, so full transparency of the measures themselves, of the methodologies, is really important. We feel like these are a great start, but we definitely have some issues that still need to be resolved around measurements. We need better standardized measures."

Visit our website for more information on hospital ratings.

A recent study found different approaches used by four popular hospital ratings systems resulted in disagreement about the ranking of many U.S. hospitals.

"Only 10% of hospitals that were rated as a high performer on one of the systems were rated as a high performer on another rating system," says lead author John Matthew Austin, MS, PhD, assistant professor of anesthesiology and critical care medicine at Johns Hopkins University School of Medicine in Baltimore. "There was no one hospital that was rated as a high performer on all four."

The study, which appeared in Health Affairs, looked at the hospital ratings systems of U.S. News, Healthgrades, The Leapfrog Group, and Consumer Reports, and found none took the same approach to assessing hospital quality. Of the 83 hospitals rated by all four systems, none were universally recognized as either a high performer or a low performer.

"I think the impact, or the influence, on consumers is that these conflicting ratings could generate confusion," Dr. Austin says. "Depending on which rating system you look at, it may give you a different answer on which hospital or where you should seek care. If you look at these four rating systems in a community, you may actually wind up being directed to four different hospitals."

David Pressel, MD, PhD, medical director of inpatient care at Alfred I. duPont Hospital for Children in Wilmington, Del., emphasized the difficulty of defining quality.

"Measuring quality is really difficult and hard, and people, physicians, and hospitals struggle with this," he explains. "I think it's very important that people recognize in rating systems there’s always going to be a top 10% or top 50% and a bottom 10% or 50%, and what's really important is not if you're in the top or bottom but what the scatter of the data is. If the scatter of the data is very narrow, they may not be providing much worse care than the top hospitals. I think that is lost on the public."

Dr. Austin's study identified varying missions and methodologies for each ratings system.

"U.S. News' Best Hospitals is actually intending to identify the best medical centers for the most complicated cases," Dr. Austin says. "The Leapfrog Hospital Safety Score has a very laser focus on patient safety, so freedom from harm, things like errors, infections."

Dr. Austin also noticed that some rating systems were more descriptive about their methods than others.

"Some of the ratings are much more transparent in what they share, in terms of how hospitals are rated, and others are less clear," he notes. "Healthgrades lists the top 100 hospitals. They're supposedly looking at hospital outcomes, but they don't publicly make their methodology available in terms of their risk-adjustment levels."

The rating systems also communicated their ratings differently. Leapfrog issues letter grades A–F; Consumer Reports and U.S. News issue scores from 0–100; and Healthgrades identifies the top 50 and top 100 hospitals but doesn’t rank hospitals, and hospitals that are not in the top 100 are not rated at all.

The study authors outlined possible improvements to eliminate some of these disparities, including reaching out to the sponsoring organizations and encouraging them to be more transparent about their ratings to allow for easier patient interpretation.

"Patients should understand what's being measured," Dr. Austin says. "Hospitals should be able to duplicate their ratings, so full transparency of the measures themselves, of the methodologies, is really important. We feel like these are a great start, but we definitely have some issues that still need to be resolved around measurements. We need better standardized measures."

Visit our website for more information on hospital ratings.

A recent study found different approaches used by four popular hospital ratings systems resulted in disagreement about the ranking of many U.S. hospitals.

"Only 10% of hospitals that were rated as a high performer on one of the systems were rated as a high performer on another rating system," says lead author John Matthew Austin, MS, PhD, assistant professor of anesthesiology and critical care medicine at Johns Hopkins University School of Medicine in Baltimore. "There was no one hospital that was rated as a high performer on all four."

The study, which appeared in Health Affairs, looked at the hospital ratings systems of U.S. News, Healthgrades, The Leapfrog Group, and Consumer Reports, and found none took the same approach to assessing hospital quality. Of the 83 hospitals rated by all four systems, none were universally recognized as either a high performer or a low performer.

"I think the impact, or the influence, on consumers is that these conflicting ratings could generate confusion," Dr. Austin says. "Depending on which rating system you look at, it may give you a different answer on which hospital or where you should seek care. If you look at these four rating systems in a community, you may actually wind up being directed to four different hospitals."

David Pressel, MD, PhD, medical director of inpatient care at Alfred I. duPont Hospital for Children in Wilmington, Del., emphasized the difficulty of defining quality.

"Measuring quality is really difficult and hard, and people, physicians, and hospitals struggle with this," he explains. "I think it's very important that people recognize in rating systems there’s always going to be a top 10% or top 50% and a bottom 10% or 50%, and what's really important is not if you're in the top or bottom but what the scatter of the data is. If the scatter of the data is very narrow, they may not be providing much worse care than the top hospitals. I think that is lost on the public."

Dr. Austin's study identified varying missions and methodologies for each ratings system.

"U.S. News' Best Hospitals is actually intending to identify the best medical centers for the most complicated cases," Dr. Austin says. "The Leapfrog Hospital Safety Score has a very laser focus on patient safety, so freedom from harm, things like errors, infections."

Dr. Austin also noticed that some rating systems were more descriptive about their methods than others.

"Some of the ratings are much more transparent in what they share, in terms of how hospitals are rated, and others are less clear," he notes. "Healthgrades lists the top 100 hospitals. They're supposedly looking at hospital outcomes, but they don't publicly make their methodology available in terms of their risk-adjustment levels."

The rating systems also communicated their ratings differently. Leapfrog issues letter grades A–F; Consumer Reports and U.S. News issue scores from 0–100; and Healthgrades identifies the top 50 and top 100 hospitals but doesn’t rank hospitals, and hospitals that are not in the top 100 are not rated at all.

The study authors outlined possible improvements to eliminate some of these disparities, including reaching out to the sponsoring organizations and encouraging them to be more transparent about their ratings to allow for easier patient interpretation.

"Patients should understand what's being measured," Dr. Austin says. "Hospitals should be able to duplicate their ratings, so full transparency of the measures themselves, of the methodologies, is really important. We feel like these are a great start, but we definitely have some issues that still need to be resolved around measurements. We need better standardized measures."

Visit our website for more information on hospital ratings.