User login
The vitamin D story exudes teaching points: It offers a master class in critical appraisal, connecting the concepts of biologic plausibility, flawed surrogate markers, confounded observational studies, and slews of randomized controlled trials (RCTs) showing no benefits on health outcomes.
Yet despite the utter lack of benefit seen in trials, the hype continues. And the pandemic has only enhanced this hype as an onslaught of papers have reported the association of low vitamin D levels and COVID-19 disease.
My questions are simple: Why doesn’t the evidence persuade people? How many nonsignificant trials do we need before researchers stop studying vitamin D, doctors stop (routinely) measuring levels, and patients stop wasting money on the unhelpful supplement? What are the implications for this lack of persuasion?
Before exploring these questions, I want to set out that symptomatic vitamin deficiencies of any sort ought to be corrected.
Biologic plausibility and the pull of observational studies
It has long been known that vitamin D is crucial for bone health and that it can be produced in the skin with sun exposure. In the last decade, however, experts note that nearly every tissue and cell in our body has a vitamin D receptor. It then follows that if this many cells in the body can activate vitamin D, it must be vital for cardiovascular health, immune function, cancer prevention: basically, everything health related.
Oodles of observational studies have found that low serum levels of vitamin D correlate with higher mortality from all causes, cancer, cardiovascular disease, and now even COVID-19. Yet no matter the amount of statistical adjustment in these studies, we cannot know whether these associations are due to true causality.
The major issue is confounding: That is, people with low vitamin D levels have other conditions or diseases that lead to higher rates of ill health. Consider a patient with obesity, arthritis, and cognitive decline; this person is unlikely to do much exercise in the sun and may have low vitamin D levels. The low vitamin D level is simply a marker of overall poor health.
The randomized controlled trials tell a clear story
There are hundreds of vitamin D RCTs. The results simplify into one sentence: Vitamin D supplements do not improve health outcomes.
Here is a short summary of some recent studies.
VITAL, a massive (N > 25,000) RCT with 5 years of follow-up, compared vitamin D supplements to placebo and found no differences in the primary endpoints of cancer or cardiac events. Rates of death from any cause were nearly identical. Crucially, in subgroup analyses, the effects did not vary according to vitamin D levels at baseline.
The D-Health investigators randomly assigned more than 21,000 adults to vitamin D or placebo and after 5.7 years of follow-up reported no differences in the primary endpoint of overall mortality. There also were no differences in cardiovascular disease mortality.
Then you have the Mendelian randomized studies, which some have called nature’s RCT. These studies take advantage of the fact that some people are born with gene variations that predispose to low vitamin D levels. More than 60 Mendelian randomization studies have evaluated the consequences of lifelong genetically lowered vitamin D levels on various outcomes; most of these have found null effects.
Then there are the meta-analyses and systematic reviews. I loved the conclusion of this review of systematic reviews from the BMJ (emphasis mine):
“Despite a few hundred systematic reviews and meta-analyses, highly convincing evidence of a clear role of vitamin D does not exist for any outcome, but associations with a selection of outcomes are probable.”
The failure to persuade
My original plan was to emphasize the power of the RCT. Despite strong associations of low vitamin D levels with poor outcomes, the trials show no benefit to treatment. This strongly suggests (or nearly proves) that low vitamin D levels are akin to premature ventricular complexes after myocardial infarction: a marker for risk but not a target for therapy.
But I now see the more important issue as why scientists, funders, clinicians, and patients are not persuaded by clear evidence. Every day in clinic I see patients on vitamin D supplements; the journals keep publishing vitamin D studies. The proponents of vitamin D remain positive. And lately there is outsized attention and hope that vitamin D will mitigate SARS-CoV2 infection – based only on observational data.
You might argue against this point by saying vitamin D is natural and relatively innocuous, so who cares?
I offer three rebuttals to that point: Opportunity costs, distraction, and the insidious danger of poor critical appraisal skills. If you are burning money on vitamin D research, there is less available to study other important issues. If a patient is distracted by low vitamin D levels, she may pay less attention to her high body mass index or hypertension. And on the matter of critical appraisal, trust in medicine requires clinicians to be competent in critical appraisal. And these days, what could be more important than trust in medical professionals?
One major reason for the failure of persuasion of evidence is spin – or language that distracts from the primary endpoint. Here are two (of many) examples:
A meta-analysis of 50 vitamin D trials set out to study mortality. The authors found no significant difference in that primary endpoint. But the second sentence in their conclusion was that vitamin D supplements reduced the risk for cancer deaths by 15%. That’s a secondary endpoint in a study with nonsignificance in the primary endpoint. That is spin. This meta-analysis was completed before the Australian D-Health trial found that cancer deaths were 15% higher in the vitamin D arm, a difference that did not reach statistical significance.
The following example is worse: The authors of the VITAL trial, which found that vitamin D supplements had no effect on the primary endpoint of invasive cancer or cardiovascular disease, published a secondary analysis of the trial looking at a different endpoint: A composite incidence of metastatic and fatal invasive total cancer. They reported a 0.4% lower rate for the vitamin D group, a difference that barely made statistical significance at a P value of .04.
But everyone knows the dangers of reanalyzing data with a new endpoint after you have seen the data. What’s more, even if this were a reasonable post hoc analysis, the results are neither clinically meaningful nor statistically robust. Yet the fatally flawed paper has been viewed 60,000 times and picked up by 48 news outlets.
Another way to distract from nonsignificant primary outcomes is to nitpick the trials. The vitamin D dose wasn’t high enough, for instance. This might persuade me if there were one or two vitamin D trials, but there are hundreds of trials and meta-analyses, and their results are consistently null.
Conclusion: No, it is not hopeless
A nihilist would argue that fighting spin is futile. They would say you can’t fight incentives and business models. The incentive structure to publish is strong, and the journals and media know vitamin D studies garner attention – which is their currency.
I am not a nihilist and believe strongly that we must continue to teach critical appraisal and numerical literacy.
In fact, I would speculate that decades of poor critical appraisal by the medical profession have fostered outsized hope and created erroneous norms.
Imagine a counter-factual world in which clinicians have taught society that the human body is unlike an engine that can be repaired by fixing one part (i.e., the vitamin D level), that magic bullets (insulin) are rare, that most treatments fail, or that you can’t rely on association studies to prove efficacy.
In this world, people would be immune from spin and hype.
The norm would be that pills, supplements, and procedures are not what delivers good health. What delivers health is an amalgam of good luck, healthy habits, and lots of time spent outside playing in the sun.
Dr. Mandrola practices cardiac electrophysiology in Louisville, Ky., and is a writer and podcaster for Medscape. He espouses a conservative approach to medical practice. He participates in clinical research and writes often about the state of medical evidence. He has disclosed no relevant financial relationships. A version of this article first appeared on Medscape.com.
The vitamin D story exudes teaching points: It offers a master class in critical appraisal, connecting the concepts of biologic plausibility, flawed surrogate markers, confounded observational studies, and slews of randomized controlled trials (RCTs) showing no benefits on health outcomes.
Yet despite the utter lack of benefit seen in trials, the hype continues. And the pandemic has only enhanced this hype as an onslaught of papers have reported the association of low vitamin D levels and COVID-19 disease.
My questions are simple: Why doesn’t the evidence persuade people? How many nonsignificant trials do we need before researchers stop studying vitamin D, doctors stop (routinely) measuring levels, and patients stop wasting money on the unhelpful supplement? What are the implications for this lack of persuasion?
Before exploring these questions, I want to set out that symptomatic vitamin deficiencies of any sort ought to be corrected.
Biologic plausibility and the pull of observational studies
It has long been known that vitamin D is crucial for bone health and that it can be produced in the skin with sun exposure. In the last decade, however, experts note that nearly every tissue and cell in our body has a vitamin D receptor. It then follows that if this many cells in the body can activate vitamin D, it must be vital for cardiovascular health, immune function, cancer prevention: basically, everything health related.
Oodles of observational studies have found that low serum levels of vitamin D correlate with higher mortality from all causes, cancer, cardiovascular disease, and now even COVID-19. Yet no matter the amount of statistical adjustment in these studies, we cannot know whether these associations are due to true causality.
The major issue is confounding: That is, people with low vitamin D levels have other conditions or diseases that lead to higher rates of ill health. Consider a patient with obesity, arthritis, and cognitive decline; this person is unlikely to do much exercise in the sun and may have low vitamin D levels. The low vitamin D level is simply a marker of overall poor health.
The randomized controlled trials tell a clear story
There are hundreds of vitamin D RCTs. The results simplify into one sentence: Vitamin D supplements do not improve health outcomes.
Here is a short summary of some recent studies.
VITAL, a massive (N > 25,000) RCT with 5 years of follow-up, compared vitamin D supplements to placebo and found no differences in the primary endpoints of cancer or cardiac events. Rates of death from any cause were nearly identical. Crucially, in subgroup analyses, the effects did not vary according to vitamin D levels at baseline.
The D-Health investigators randomly assigned more than 21,000 adults to vitamin D or placebo and after 5.7 years of follow-up reported no differences in the primary endpoint of overall mortality. There also were no differences in cardiovascular disease mortality.
Then you have the Mendelian randomized studies, which some have called nature’s RCT. These studies take advantage of the fact that some people are born with gene variations that predispose to low vitamin D levels. More than 60 Mendelian randomization studies have evaluated the consequences of lifelong genetically lowered vitamin D levels on various outcomes; most of these have found null effects.
Then there are the meta-analyses and systematic reviews. I loved the conclusion of this review of systematic reviews from the BMJ (emphasis mine):
“Despite a few hundred systematic reviews and meta-analyses, highly convincing evidence of a clear role of vitamin D does not exist for any outcome, but associations with a selection of outcomes are probable.”
The failure to persuade
My original plan was to emphasize the power of the RCT. Despite strong associations of low vitamin D levels with poor outcomes, the trials show no benefit to treatment. This strongly suggests (or nearly proves) that low vitamin D levels are akin to premature ventricular complexes after myocardial infarction: a marker for risk but not a target for therapy.
But I now see the more important issue as why scientists, funders, clinicians, and patients are not persuaded by clear evidence. Every day in clinic I see patients on vitamin D supplements; the journals keep publishing vitamin D studies. The proponents of vitamin D remain positive. And lately there is outsized attention and hope that vitamin D will mitigate SARS-CoV2 infection – based only on observational data.
You might argue against this point by saying vitamin D is natural and relatively innocuous, so who cares?
I offer three rebuttals to that point: Opportunity costs, distraction, and the insidious danger of poor critical appraisal skills. If you are burning money on vitamin D research, there is less available to study other important issues. If a patient is distracted by low vitamin D levels, she may pay less attention to her high body mass index or hypertension. And on the matter of critical appraisal, trust in medicine requires clinicians to be competent in critical appraisal. And these days, what could be more important than trust in medical professionals?
One major reason for the failure of persuasion of evidence is spin – or language that distracts from the primary endpoint. Here are two (of many) examples:
A meta-analysis of 50 vitamin D trials set out to study mortality. The authors found no significant difference in that primary endpoint. But the second sentence in their conclusion was that vitamin D supplements reduced the risk for cancer deaths by 15%. That’s a secondary endpoint in a study with nonsignificance in the primary endpoint. That is spin. This meta-analysis was completed before the Australian D-Health trial found that cancer deaths were 15% higher in the vitamin D arm, a difference that did not reach statistical significance.
The following example is worse: The authors of the VITAL trial, which found that vitamin D supplements had no effect on the primary endpoint of invasive cancer or cardiovascular disease, published a secondary analysis of the trial looking at a different endpoint: A composite incidence of metastatic and fatal invasive total cancer. They reported a 0.4% lower rate for the vitamin D group, a difference that barely made statistical significance at a P value of .04.
But everyone knows the dangers of reanalyzing data with a new endpoint after you have seen the data. What’s more, even if this were a reasonable post hoc analysis, the results are neither clinically meaningful nor statistically robust. Yet the fatally flawed paper has been viewed 60,000 times and picked up by 48 news outlets.
Another way to distract from nonsignificant primary outcomes is to nitpick the trials. The vitamin D dose wasn’t high enough, for instance. This might persuade me if there were one or two vitamin D trials, but there are hundreds of trials and meta-analyses, and their results are consistently null.
Conclusion: No, it is not hopeless
A nihilist would argue that fighting spin is futile. They would say you can’t fight incentives and business models. The incentive structure to publish is strong, and the journals and media know vitamin D studies garner attention – which is their currency.
I am not a nihilist and believe strongly that we must continue to teach critical appraisal and numerical literacy.
In fact, I would speculate that decades of poor critical appraisal by the medical profession have fostered outsized hope and created erroneous norms.
Imagine a counter-factual world in which clinicians have taught society that the human body is unlike an engine that can be repaired by fixing one part (i.e., the vitamin D level), that magic bullets (insulin) are rare, that most treatments fail, or that you can’t rely on association studies to prove efficacy.
In this world, people would be immune from spin and hype.
The norm would be that pills, supplements, and procedures are not what delivers good health. What delivers health is an amalgam of good luck, healthy habits, and lots of time spent outside playing in the sun.
Dr. Mandrola practices cardiac electrophysiology in Louisville, Ky., and is a writer and podcaster for Medscape. He espouses a conservative approach to medical practice. He participates in clinical research and writes often about the state of medical evidence. He has disclosed no relevant financial relationships. A version of this article first appeared on Medscape.com.
The vitamin D story exudes teaching points: It offers a master class in critical appraisal, connecting the concepts of biologic plausibility, flawed surrogate markers, confounded observational studies, and slews of randomized controlled trials (RCTs) showing no benefits on health outcomes.
Yet despite the utter lack of benefit seen in trials, the hype continues. And the pandemic has only enhanced this hype as an onslaught of papers have reported the association of low vitamin D levels and COVID-19 disease.
My questions are simple: Why doesn’t the evidence persuade people? How many nonsignificant trials do we need before researchers stop studying vitamin D, doctors stop (routinely) measuring levels, and patients stop wasting money on the unhelpful supplement? What are the implications for this lack of persuasion?
Before exploring these questions, I want to set out that symptomatic vitamin deficiencies of any sort ought to be corrected.
Biologic plausibility and the pull of observational studies
It has long been known that vitamin D is crucial for bone health and that it can be produced in the skin with sun exposure. In the last decade, however, experts note that nearly every tissue and cell in our body has a vitamin D receptor. It then follows that if this many cells in the body can activate vitamin D, it must be vital for cardiovascular health, immune function, cancer prevention: basically, everything health related.
Oodles of observational studies have found that low serum levels of vitamin D correlate with higher mortality from all causes, cancer, cardiovascular disease, and now even COVID-19. Yet no matter the amount of statistical adjustment in these studies, we cannot know whether these associations are due to true causality.
The major issue is confounding: That is, people with low vitamin D levels have other conditions or diseases that lead to higher rates of ill health. Consider a patient with obesity, arthritis, and cognitive decline; this person is unlikely to do much exercise in the sun and may have low vitamin D levels. The low vitamin D level is simply a marker of overall poor health.
The randomized controlled trials tell a clear story
There are hundreds of vitamin D RCTs. The results simplify into one sentence: Vitamin D supplements do not improve health outcomes.
Here is a short summary of some recent studies.
VITAL, a massive (N > 25,000) RCT with 5 years of follow-up, compared vitamin D supplements to placebo and found no differences in the primary endpoints of cancer or cardiac events. Rates of death from any cause were nearly identical. Crucially, in subgroup analyses, the effects did not vary according to vitamin D levels at baseline.
The D-Health investigators randomly assigned more than 21,000 adults to vitamin D or placebo and after 5.7 years of follow-up reported no differences in the primary endpoint of overall mortality. There also were no differences in cardiovascular disease mortality.
Then you have the Mendelian randomized studies, which some have called nature’s RCT. These studies take advantage of the fact that some people are born with gene variations that predispose to low vitamin D levels. More than 60 Mendelian randomization studies have evaluated the consequences of lifelong genetically lowered vitamin D levels on various outcomes; most of these have found null effects.
Then there are the meta-analyses and systematic reviews. I loved the conclusion of this review of systematic reviews from the BMJ (emphasis mine):
“Despite a few hundred systematic reviews and meta-analyses, highly convincing evidence of a clear role of vitamin D does not exist for any outcome, but associations with a selection of outcomes are probable.”
The failure to persuade
My original plan was to emphasize the power of the RCT. Despite strong associations of low vitamin D levels with poor outcomes, the trials show no benefit to treatment. This strongly suggests (or nearly proves) that low vitamin D levels are akin to premature ventricular complexes after myocardial infarction: a marker for risk but not a target for therapy.
But I now see the more important issue as why scientists, funders, clinicians, and patients are not persuaded by clear evidence. Every day in clinic I see patients on vitamin D supplements; the journals keep publishing vitamin D studies. The proponents of vitamin D remain positive. And lately there is outsized attention and hope that vitamin D will mitigate SARS-CoV2 infection – based only on observational data.
You might argue against this point by saying vitamin D is natural and relatively innocuous, so who cares?
I offer three rebuttals to that point: Opportunity costs, distraction, and the insidious danger of poor critical appraisal skills. If you are burning money on vitamin D research, there is less available to study other important issues. If a patient is distracted by low vitamin D levels, she may pay less attention to her high body mass index or hypertension. And on the matter of critical appraisal, trust in medicine requires clinicians to be competent in critical appraisal. And these days, what could be more important than trust in medical professionals?
One major reason for the failure of persuasion of evidence is spin – or language that distracts from the primary endpoint. Here are two (of many) examples:
A meta-analysis of 50 vitamin D trials set out to study mortality. The authors found no significant difference in that primary endpoint. But the second sentence in their conclusion was that vitamin D supplements reduced the risk for cancer deaths by 15%. That’s a secondary endpoint in a study with nonsignificance in the primary endpoint. That is spin. This meta-analysis was completed before the Australian D-Health trial found that cancer deaths were 15% higher in the vitamin D arm, a difference that did not reach statistical significance.
The following example is worse: The authors of the VITAL trial, which found that vitamin D supplements had no effect on the primary endpoint of invasive cancer or cardiovascular disease, published a secondary analysis of the trial looking at a different endpoint: A composite incidence of metastatic and fatal invasive total cancer. They reported a 0.4% lower rate for the vitamin D group, a difference that barely made statistical significance at a P value of .04.
But everyone knows the dangers of reanalyzing data with a new endpoint after you have seen the data. What’s more, even if this were a reasonable post hoc analysis, the results are neither clinically meaningful nor statistically robust. Yet the fatally flawed paper has been viewed 60,000 times and picked up by 48 news outlets.
Another way to distract from nonsignificant primary outcomes is to nitpick the trials. The vitamin D dose wasn’t high enough, for instance. This might persuade me if there were one or two vitamin D trials, but there are hundreds of trials and meta-analyses, and their results are consistently null.
Conclusion: No, it is not hopeless
A nihilist would argue that fighting spin is futile. They would say you can’t fight incentives and business models. The incentive structure to publish is strong, and the journals and media know vitamin D studies garner attention – which is their currency.
I am not a nihilist and believe strongly that we must continue to teach critical appraisal and numerical literacy.
In fact, I would speculate that decades of poor critical appraisal by the medical profession have fostered outsized hope and created erroneous norms.
Imagine a counter-factual world in which clinicians have taught society that the human body is unlike an engine that can be repaired by fixing one part (i.e., the vitamin D level), that magic bullets (insulin) are rare, that most treatments fail, or that you can’t rely on association studies to prove efficacy.
In this world, people would be immune from spin and hype.
The norm would be that pills, supplements, and procedures are not what delivers good health. What delivers health is an amalgam of good luck, healthy habits, and lots of time spent outside playing in the sun.
Dr. Mandrola practices cardiac electrophysiology in Louisville, Ky., and is a writer and podcaster for Medscape. He espouses a conservative approach to medical practice. He participates in clinical research and writes often about the state of medical evidence. He has disclosed no relevant financial relationships. A version of this article first appeared on Medscape.com.
