The Problem with Ps

Most studies include a measure of the significance of treatment effects such as a P value or confidence interval (CI). CIs (Journal of family practice, December 2003, 53:970) are usually preferred to P values, which have notable limitations.

1. P values are easily misinterpreted

A P value is the probability of obtaining a result (usually a difference between treatments) as large or larger than that observed in a study if the null hypothesis (ie, no difference exists between treatments) is true. Differences in treatment effects can be expressed as absolute differences or as odds ratios. No difference, for example, corresponds to an absolute difference of zero or an odds ratio of 1.0.

Consider a recent primary care study from the UK comparing the effectiveness of different lipid-lowering drugs to simvastatin.¹ The odds ratio for achieving a cholesterol level ≤5 mmol/L with pravastatin compared with simvastatin was 0.58, with a P value of .003 (ie, simvastatin superior to pravastatin). This means that if there is no difference between pravastatin and simvastatin (ie, null hypothesis is true), the probability of getting an odds ratio of 0.58 or less is just .003 (0.3%).

A P<.05 (sometimes <.01) is usually considered to be sufficient evidence to reject the null hypothesis. This is not intuitively obvious and does not appear to provide useful information.

Many clinicians misinterpret the P value “backwards” as the probability of the null hypothesis assuming the results. In the example above, the misinterpretation would be that there is a 0.3% probability of there being no difference between simvastatin and pravastatin based on the results. Misinterpreting the P value in this way is serious, since the true probability of the null hypothesis based on the results is often much greater than the P value.

2. P values tell us nothing about the magnitude of a significant difference

In the example above, the odds ratio of 0.58, P=.003 has a 95% CI of 0.40​0.83. The confidence interval, unlike the P value, provides a measure of the precision of the odds ratio.

3. P values are very sensitive to sample size

A small difference between 2 treatments that is clinically insignificant (eg, 1-week difference in mean life expectancy between 2 lipid-lowering treatments) may have a statistically significant P value (ie, <.05) if the sample size is large enough. P values, therefore, can exaggerate the significance of results.

Correspondence
Goutham Rao, MD, 3518 Fifth Avenue, Pittsburgh, PA 15361. E-mail: grao@upmc.edu.

Most studies include a measure of the significance of treatment effects such as a P value or confidence interval (CI). CIs (Journal of family practice, December 2003, 53:970) are usually preferred to P values, which have notable limitations.

1. P values are easily misinterpreted

A P value is the probability of obtaining a result (usually a difference between treatments) as large or larger than that observed in a study if the null hypothesis (ie, no difference exists between treatments) is true. Differences in treatment effects can be expressed as absolute differences or as odds ratios. No difference, for example, corresponds to an absolute difference of zero or an odds ratio of 1.0.

Consider a recent primary care study from the UK comparing the effectiveness of different lipid-lowering drugs to simvastatin.¹ The odds ratio for achieving a cholesterol level ≤5 mmol/L with pravastatin compared with simvastatin was 0.58, with a P value of .003 (ie, simvastatin superior to pravastatin). This means that if there is no difference between pravastatin and simvastatin (ie, null hypothesis is true), the probability of getting an odds ratio of 0.58 or less is just .003 (0.3%).

A P<.05 (sometimes <.01) is usually considered to be sufficient evidence to reject the null hypothesis. This is not intuitively obvious and does not appear to provide useful information.

Many clinicians misinterpret the P value “backwards” as the probability of the null hypothesis assuming the results. In the example above, the misinterpretation would be that there is a 0.3% probability of there being no difference between simvastatin and pravastatin based on the results. Misinterpreting the P value in this way is serious, since the true probability of the null hypothesis based on the results is often much greater than the P value.

2. P values tell us nothing about the magnitude of a significant difference

In the example above, the odds ratio of 0.58, P=.003 has a 95% CI of 0.40​0.83. The confidence interval, unlike the P value, provides a measure of the precision of the odds ratio.

3. P values are very sensitive to sample size

A small difference between 2 treatments that is clinically insignificant (eg, 1-week difference in mean life expectancy between 2 lipid-lowering treatments) may have a statistically significant P value (ie, <.05) if the sample size is large enough. P values, therefore, can exaggerate the significance of results.

Correspondence
Goutham Rao, MD, 3518 Fifth Avenue, Pittsburgh, PA 15361. E-mail: grao@upmc.edu.

Most studies include a measure of the significance of treatment effects such as a P value or confidence interval (CI). CIs (Journal of family practice, December 2003, 53:970) are usually preferred to P values, which have notable limitations.

1. P values are easily misinterpreted

A P value is the probability of obtaining a result (usually a difference between treatments) as large or larger than that observed in a study if the null hypothesis (ie, no difference exists between treatments) is true. Differences in treatment effects can be expressed as absolute differences or as odds ratios. No difference, for example, corresponds to an absolute difference of zero or an odds ratio of 1.0.

Consider a recent primary care study from the UK comparing the effectiveness of different lipid-lowering drugs to simvastatin.¹ The odds ratio for achieving a cholesterol level ≤5 mmol/L with pravastatin compared with simvastatin was 0.58, with a P value of .003 (ie, simvastatin superior to pravastatin). This means that if there is no difference between pravastatin and simvastatin (ie, null hypothesis is true), the probability of getting an odds ratio of 0.58 or less is just .003 (0.3%).

A P<.05 (sometimes <.01) is usually considered to be sufficient evidence to reject the null hypothesis. This is not intuitively obvious and does not appear to provide useful information.

Many clinicians misinterpret the P value “backwards” as the probability of the null hypothesis assuming the results. In the example above, the misinterpretation would be that there is a 0.3% probability of there being no difference between simvastatin and pravastatin based on the results. Misinterpreting the P value in this way is serious, since the true probability of the null hypothesis based on the results is often much greater than the P value.

2. P values tell us nothing about the magnitude of a significant difference

In the example above, the odds ratio of 0.58, P=.003 has a 95% CI of 0.40​0.83. The confidence interval, unlike the P value, provides a measure of the precision of the odds ratio.

3. P values are very sensitive to sample size

A small difference between 2 treatments that is clinically insignificant (eg, 1-week difference in mean life expectancy between 2 lipid-lowering treatments) may have a statistically significant P value (ie, <.05) if the sample size is large enough. P values, therefore, can exaggerate the significance of results.

Correspondence
Goutham Rao, MD, 3518 Fifth Avenue, Pittsburgh, PA 15361. E-mail: grao@upmc.edu.