Individual SCIP Measures Don't Cut Infection

Patients who received two out of six infection prevention measures recommended by the Surgical Care Improvement Project had a lower risk of postoperative infection, compared with patients who hadn't received at least two of the measures, according to a large retrospective study.

However, adherence on individual SCIP measures did not significantly lower the probability of postoperative infection, the study found. Currently, those individual measures are the only ones reported publicly by the Centers for Medicare and Medicaid Services (CMS).

“Based on our findings, the individual item performance rates reported publicly do not fulfill their stated purpose of pointing consumers toward high-quality hospitals,” the authors wrote. “However, when taken in aggregate, improved performance on our global all-or-none composite measure is associated with improved outcomes at the discharge level. Therefore, while the individual items may not imply quality differences, the overall ability to demonstrate adherence to multiple SCIP processes of care may.”

The study, led by a research team from Case Western Reserve University, Cleveland, used data on 405,720 discharges from 398 hospitals in which SCIP performance was recorded (JAMA 2010;303:2479-85).

The researchers analyzed individual SCIP postoperative infection prevention measures. They also combined these six measures into two global measures of discharge-level adherence.

In the first global measure, patients needed to receive prophylactic antibiotics within 1 hour prior to surgical incision (2 hours if they received vancomycin). They also needed to have their prophylactic antibiotics discontinued within 24 hours after surgery (48 hours for cardiac surgery).

In the second measure, the researchers recorded whether patients had received any two of the six measures recommended by SCIP to reduce infection.

Demonstrated adherence to SCIP on the first, “all-or-none” composite measure decreased the likelihood of developing a postoperative infection from 11.5 infections per 1,000 discharges to 5.3 infections per 1,000 discharges, which was not statistically significant.

The second measure—receipt of two of six SCIP-recommended infection prevention procedures—did show a statistically significant association: Risk of postoperative infection fell from 14.2 to 6.8 postoperative infections per 1,000.

The CMS uses SCIP results as a component of its Medicare value-based purchasing initiatives, which are intended to improve quality of care through incentive-based reimbursement, and also publishes individual SCIP measures to help patients select surgical “centers of excellence.”

“This publicized use implies that reported adherence on these measures is directly related to improved outcomes. Our findings are unable to support this assertion,” the study's authors concluded.

Disclosures: No financial disclosures were reported. The study was supported by grants from the Agency for Healthcare Research and Quality and the National Cancer Institute.

My Take

Narrow Measures Are Not Effective

The Surgical Care Improvement Project (SCIP) was established in 2006 with the goal of reducing surgical complications by 25% in 2010. Of the nine performance measures, six are related to surgical site infection prevention. Efforts to reduce surgical site infection are important because this complication results in significant morbidity and additional resource use.

The SCIP measures were added to the Hospital Compare Web site to help patients determine where to receive surgical care. The underlying assumption was that higher performance on these measures equals better surgical outcomes. Furthermore, the Centers for Medicare and Medicaid Services has proposed use of the SCIP measures for value-based purchasing and payment to hospitals.

The report by Stulberg et al. is the largest study to date that fails to demonstrate an association between adherence to SCIP measures and the occurrence of postoperative infections.

Ideal surgical quality-improvement efforts would measure whether the right patient receives the right operation at the right time and whether the operation is effective. This is more challenging to measure given the complex nature of the health care system, including barriers to access, financial incentives, and the uniqueness of the therapeutic intent of individual surgical procedures.

Current mandated surgical quality–improvement processes such as SCIP focus on incremental and narrow process measures that are purported to measure the overall quality of an episode of surgical care. Despite enormous resources committed to these measures and marked improvement in adherence, the evidence to date suggests that SCIP has not improved surgical outcomes. Future quality improvement endeavors should have linkage between discrete performance and outcome measures so the effectiveness of combined efforts can be unequivocally measured and clearly evaluated.

MARY T. HAWN, M.D., is from the Center for Surgical and Medical Acute Care Research and Transitions at the Veterans Affairs Medical Center in Birmingham, and the department of surgery, University of Alabama at Birmingham. Her comments were made in a published editorial accompanying the study (JAMA 2010;303:2527-8).

Patients who received two out of six infection prevention measures recommended by the Surgical Care Improvement Project had a lower risk of postoperative infection, compared with patients who hadn't received at least two of the measures, according to a large retrospective study.

However, adherence on individual SCIP measures did not significantly lower the probability of postoperative infection, the study found. Currently, those individual measures are the only ones reported publicly by the Centers for Medicare and Medicaid Services (CMS).

“Based on our findings, the individual item performance rates reported publicly do not fulfill their stated purpose of pointing consumers toward high-quality hospitals,” the authors wrote. “However, when taken in aggregate, improved performance on our global all-or-none composite measure is associated with improved outcomes at the discharge level. Therefore, while the individual items may not imply quality differences, the overall ability to demonstrate adherence to multiple SCIP processes of care may.”

The study, led by a research team from Case Western Reserve University, Cleveland, used data on 405,720 discharges from 398 hospitals in which SCIP performance was recorded (JAMA 2010;303:2479-85).

The researchers analyzed individual SCIP postoperative infection prevention measures. They also combined these six measures into two global measures of discharge-level adherence.

In the first global measure, patients needed to receive prophylactic antibiotics within 1 hour prior to surgical incision (2 hours if they received vancomycin). They also needed to have their prophylactic antibiotics discontinued within 24 hours after surgery (48 hours for cardiac surgery).

In the second measure, the researchers recorded whether patients had received any two of the six measures recommended by SCIP to reduce infection.

Demonstrated adherence to SCIP on the first, “all-or-none” composite measure decreased the likelihood of developing a postoperative infection from 11.5 infections per 1,000 discharges to 5.3 infections per 1,000 discharges, which was not statistically significant.

The second measure—receipt of two of six SCIP-recommended infection prevention procedures—did show a statistically significant association: Risk of postoperative infection fell from 14.2 to 6.8 postoperative infections per 1,000.

The CMS uses SCIP results as a component of its Medicare value-based purchasing initiatives, which are intended to improve quality of care through incentive-based reimbursement, and also publishes individual SCIP measures to help patients select surgical “centers of excellence.”

“This publicized use implies that reported adherence on these measures is directly related to improved outcomes. Our findings are unable to support this assertion,” the study's authors concluded.

Disclosures: No financial disclosures were reported. The study was supported by grants from the Agency for Healthcare Research and Quality and the National Cancer Institute.

My Take

Narrow Measures Are Not Effective

The Surgical Care Improvement Project (SCIP) was established in 2006 with the goal of reducing surgical complications by 25% in 2010. Of the nine performance measures, six are related to surgical site infection prevention. Efforts to reduce surgical site infection are important because this complication results in significant morbidity and additional resource use.

The SCIP measures were added to the Hospital Compare Web site to help patients determine where to receive surgical care. The underlying assumption was that higher performance on these measures equals better surgical outcomes. Furthermore, the Centers for Medicare and Medicaid Services has proposed use of the SCIP measures for value-based purchasing and payment to hospitals.

The report by Stulberg et al. is the largest study to date that fails to demonstrate an association between adherence to SCIP measures and the occurrence of postoperative infections.

Ideal surgical quality-improvement efforts would measure whether the right patient receives the right operation at the right time and whether the operation is effective. This is more challenging to measure given the complex nature of the health care system, including barriers to access, financial incentives, and the uniqueness of the therapeutic intent of individual surgical procedures.

Current mandated surgical quality–improvement processes such as SCIP focus on incremental and narrow process measures that are purported to measure the overall quality of an episode of surgical care. Despite enormous resources committed to these measures and marked improvement in adherence, the evidence to date suggests that SCIP has not improved surgical outcomes. Future quality improvement endeavors should have linkage between discrete performance and outcome measures so the effectiveness of combined efforts can be unequivocally measured and clearly evaluated.

MARY T. HAWN, M.D., is from the Center for Surgical and Medical Acute Care Research and Transitions at the Veterans Affairs Medical Center in Birmingham, and the department of surgery, University of Alabama at Birmingham. Her comments were made in a published editorial accompanying the study (JAMA 2010;303:2527-8).

Patients who received two out of six infection prevention measures recommended by the Surgical Care Improvement Project had a lower risk of postoperative infection, compared with patients who hadn't received at least two of the measures, according to a large retrospective study.

However, adherence on individual SCIP measures did not significantly lower the probability of postoperative infection, the study found. Currently, those individual measures are the only ones reported publicly by the Centers for Medicare and Medicaid Services (CMS).

“Based on our findings, the individual item performance rates reported publicly do not fulfill their stated purpose of pointing consumers toward high-quality hospitals,” the authors wrote. “However, when taken in aggregate, improved performance on our global all-or-none composite measure is associated with improved outcomes at the discharge level. Therefore, while the individual items may not imply quality differences, the overall ability to demonstrate adherence to multiple SCIP processes of care may.”

The study, led by a research team from Case Western Reserve University, Cleveland, used data on 405,720 discharges from 398 hospitals in which SCIP performance was recorded (JAMA 2010;303:2479-85).

The researchers analyzed individual SCIP postoperative infection prevention measures. They also combined these six measures into two global measures of discharge-level adherence.

In the first global measure, patients needed to receive prophylactic antibiotics within 1 hour prior to surgical incision (2 hours if they received vancomycin). They also needed to have their prophylactic antibiotics discontinued within 24 hours after surgery (48 hours for cardiac surgery).

In the second measure, the researchers recorded whether patients had received any two of the six measures recommended by SCIP to reduce infection.

Demonstrated adherence to SCIP on the first, “all-or-none” composite measure decreased the likelihood of developing a postoperative infection from 11.5 infections per 1,000 discharges to 5.3 infections per 1,000 discharges, which was not statistically significant.

The second measure—receipt of two of six SCIP-recommended infection prevention procedures—did show a statistically significant association: Risk of postoperative infection fell from 14.2 to 6.8 postoperative infections per 1,000.

The CMS uses SCIP results as a component of its Medicare value-based purchasing initiatives, which are intended to improve quality of care through incentive-based reimbursement, and also publishes individual SCIP measures to help patients select surgical “centers of excellence.”

“This publicized use implies that reported adherence on these measures is directly related to improved outcomes. Our findings are unable to support this assertion,” the study's authors concluded.

Disclosures: No financial disclosures were reported. The study was supported by grants from the Agency for Healthcare Research and Quality and the National Cancer Institute.

My Take

Narrow Measures Are Not Effective

The Surgical Care Improvement Project (SCIP) was established in 2006 with the goal of reducing surgical complications by 25% in 2010. Of the nine performance measures, six are related to surgical site infection prevention. Efforts to reduce surgical site infection are important because this complication results in significant morbidity and additional resource use.

The SCIP measures were added to the Hospital Compare Web site to help patients determine where to receive surgical care. The underlying assumption was that higher performance on these measures equals better surgical outcomes. Furthermore, the Centers for Medicare and Medicaid Services has proposed use of the SCIP measures for value-based purchasing and payment to hospitals.

The report by Stulberg et al. is the largest study to date that fails to demonstrate an association between adherence to SCIP measures and the occurrence of postoperative infections.

Ideal surgical quality-improvement efforts would measure whether the right patient receives the right operation at the right time and whether the operation is effective. This is more challenging to measure given the complex nature of the health care system, including barriers to access, financial incentives, and the uniqueness of the therapeutic intent of individual surgical procedures.

Current mandated surgical quality–improvement processes such as SCIP focus on incremental and narrow process measures that are purported to measure the overall quality of an episode of surgical care. Despite enormous resources committed to these measures and marked improvement in adherence, the evidence to date suggests that SCIP has not improved surgical outcomes. Future quality improvement endeavors should have linkage between discrete performance and outcome measures so the effectiveness of combined efforts can be unequivocally measured and clearly evaluated.

MARY T. HAWN, M.D., is from the Center for Surgical and Medical Acute Care Research and Transitions at the Veterans Affairs Medical Center in Birmingham, and the department of surgery, University of Alabama at Birmingham. Her comments were made in a published editorial accompanying the study (JAMA 2010;303:2527-8).