Meta-analyses, metaphysics, and reality

About 15 years ago I attended a National Institutes of Health consensus conference on the treatment of hypercholesterolemia. The data on lowering LDL-C and plaque regression were strikingly encouraging. However, preliminary pooled data were presented that suggested that lowering cholesterol increased the risk of death due to accidents and suicides. At face value, this seemed to lack scientific validity, and a member of the panel, disparaging the concept of meta-analysis in general, joked, “Meta-analysis is to analysis as metaphysics is to physics.” I am not sure if he was the first to say this, and I do not think it is entirely accurate, but most of us chuckled.

Dr. Jeffrey Aronson, in an editorial in the British Journal of Clinical Pharmacology,¹ discussed the term meta-analysis and its link to Aristotle and metaphysics. It seems that Andronicus, an editor of Aristotle’s work, titled the first set of Aristotle’s papers on natural sciences The Physics, and a second set of papers The Metaphysics because they were written after The Physics. The Metaphysics, however, dealt more with philosophy, and thus the term metaphysics acquired over time the connotation of “not real” physics.

A meta-analysis is, in fact, a structured analysis of prior analyses (often randomized trials). In this issue of the Journal, Dr. Esteban Walker and colleagues² clarify the specific rules that must be followed when doing a meta-analysis.

While I believe that meta-analyses often have significant issues that must be resolved before they can be translated into a change in clinical practice, a generic condemnation (or acceptance) of the tool is not appropriate. One study compared the results of meta-analyses with subsequently performed randomized clinical trials, and in only 12% were the conclusions significantly different.³

A goal of meta-analysis is to overcome limitations of small sample sizes by pooling results in an appropriate and orderly way. One problem has been the inability to access unpublished (usually “negative”) trials. With the Food and Drug Administration Amendments Act of 2007, all clinical trials beyond phase I studies will be posted on the Web, and because of this, much of the concern about publication bias may be overcome.

But the issue remains of how best to interpret the clinical significance of small effects or rare but serious events. As Walker et al note, clinicians need to be very careful about directly translating conclusions from meta-analyses into clinical practice. Patients (and the news media) would be wise to exercise the same caution.

About 15 years ago I attended a National Institutes of Health consensus conference on the treatment of hypercholesterolemia. The data on lowering LDL-C and plaque regression were strikingly encouraging. However, preliminary pooled data were presented that suggested that lowering cholesterol increased the risk of death due to accidents and suicides. At face value, this seemed to lack scientific validity, and a member of the panel, disparaging the concept of meta-analysis in general, joked, “Meta-analysis is to analysis as metaphysics is to physics.” I am not sure if he was the first to say this, and I do not think it is entirely accurate, but most of us chuckled.

Dr. Jeffrey Aronson, in an editorial in the British Journal of Clinical Pharmacology,¹ discussed the term meta-analysis and its link to Aristotle and metaphysics. It seems that Andronicus, an editor of Aristotle’s work, titled the first set of Aristotle’s papers on natural sciences The Physics, and a second set of papers The Metaphysics because they were written after The Physics. The Metaphysics, however, dealt more with philosophy, and thus the term metaphysics acquired over time the connotation of “not real” physics.

A meta-analysis is, in fact, a structured analysis of prior analyses (often randomized trials). In this issue of the Journal, Dr. Esteban Walker and colleagues² clarify the specific rules that must be followed when doing a meta-analysis.

While I believe that meta-analyses often have significant issues that must be resolved before they can be translated into a change in clinical practice, a generic condemnation (or acceptance) of the tool is not appropriate. One study compared the results of meta-analyses with subsequently performed randomized clinical trials, and in only 12% were the conclusions significantly different.³

A goal of meta-analysis is to overcome limitations of small sample sizes by pooling results in an appropriate and orderly way. One problem has been the inability to access unpublished (usually “negative”) trials. With the Food and Drug Administration Amendments Act of 2007, all clinical trials beyond phase I studies will be posted on the Web, and because of this, much of the concern about publication bias may be overcome.

But the issue remains of how best to interpret the clinical significance of small effects or rare but serious events. As Walker et al note, clinicians need to be very careful about directly translating conclusions from meta-analyses into clinical practice. Patients (and the news media) would be wise to exercise the same caution.

About 15 years ago I attended a National Institutes of Health consensus conference on the treatment of hypercholesterolemia. The data on lowering LDL-C and plaque regression were strikingly encouraging. However, preliminary pooled data were presented that suggested that lowering cholesterol increased the risk of death due to accidents and suicides. At face value, this seemed to lack scientific validity, and a member of the panel, disparaging the concept of meta-analysis in general, joked, “Meta-analysis is to analysis as metaphysics is to physics.” I am not sure if he was the first to say this, and I do not think it is entirely accurate, but most of us chuckled.

Dr. Jeffrey Aronson, in an editorial in the British Journal of Clinical Pharmacology,¹ discussed the term meta-analysis and its link to Aristotle and metaphysics. It seems that Andronicus, an editor of Aristotle’s work, titled the first set of Aristotle’s papers on natural sciences The Physics, and a second set of papers The Metaphysics because they were written after The Physics. The Metaphysics, however, dealt more with philosophy, and thus the term metaphysics acquired over time the connotation of “not real” physics.

A meta-analysis is, in fact, a structured analysis of prior analyses (often randomized trials). In this issue of the Journal, Dr. Esteban Walker and colleagues² clarify the specific rules that must be followed when doing a meta-analysis.

While I believe that meta-analyses often have significant issues that must be resolved before they can be translated into a change in clinical practice, a generic condemnation (or acceptance) of the tool is not appropriate. One study compared the results of meta-analyses with subsequently performed randomized clinical trials, and in only 12% were the conclusions significantly different.³

A goal of meta-analysis is to overcome limitations of small sample sizes by pooling results in an appropriate and orderly way. One problem has been the inability to access unpublished (usually “negative”) trials. With the Food and Drug Administration Amendments Act of 2007, all clinical trials beyond phase I studies will be posted on the Web, and because of this, much of the concern about publication bias may be overcome.

But the issue remains of how best to interpret the clinical significance of small effects or rare but serious events. As Walker et al note, clinicians need to be very careful about directly translating conclusions from meta-analyses into clinical practice. Patients (and the news media) would be wise to exercise the same caution.