User login
ORLANDO – What doctors think they know to be true in medicine has changed dramatically in the past several decades and will be different again in the decades to come, leaving them with a dilemma, according to Kevin T. Powell, MD, PhD, a pediatric hospitalist in St. Louis. If half of what doctors teach or know in medicine today will ultimately end up not being true, how do they know what to believe or accept?
While there is not a single satisfactory answer to that question, researchers can select research that gets doctors closer to reliable findings and steer them away from the barrage of poor-quality research that emerges from the current publish-or-perish system, Dr. Powell told his colleagues at the annual meeting of the American Academy of Pediatrics.
During his talk, Dr. Powell discussed the challenges and flaws with medical research as it is currently conducted, citing Doug Altman’s writings on these problems as early as 1994.
“The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent effort to find a solution,” wrote Mr. Altman, an English medical statistician (BMJ. 1994;308:283).
“We need less research, better research, and research done for the right reasons,” Mr. Altman concluded. “Abandoning using the number of publications as a measure of ability would be a start.”
In an interview, Dr. Powell described an unfortunate consequence of the publish-or-perish pressure in academic medicine: A glut of short-term, small studies with little clinical utility that researchers can complete in 1 or 2 years rather than the large, multicenter studies that take several years – and produce higher-quality findings – but cannot be turned into as many publications.
“We’re generating a lot of medical research findings that end up being false,” he said. “It’s a random walk in terms of getting to the truth rather than having an accurate process of getting to truth through evidence-based medicine.”
But he was hopeful, not cynical, about the way forward. By persuading people that medical research has changed for the worse over time and can change into something better, Dr. Powell saw potential for future research resulting in the same sort of public health achievements that research produced in the past, such as big reductions in smoking or sudden infant death syndrome.
Dr. Powell concluded his talk with a riff on Martin Luther’s 95 Theses, the 9.5 Theses, for a reformation of evidence-based medicine that together address the various shortcomings he discussed.
1. Recognize academic promotion as a bias, just like drug money.
2. Don’t confound statistically significant and clinically significant.
3. Use only significant figures.
4. Use the phrase “we did not DETECT a difference” and include power calculations.
5. Use confidence intervals instead of P values.
6. Use number needed to harm and number needed to treat instead of relative risk.
7. Absence of proof is not proof of absence. When there is insufficient randomized, controlled trial evidence, have an independent party estimate an effect based on non-RCT articles.
8. Any article implying clinical practice should change must include a counterpoint and a benefit cost analysis. Consider both effectiveness and safety.
9. Use postmarketing peer review.
9.5. Beware of research based on surveys.
Dr. Powell reported no relevant financial disclosures.
ORLANDO – What doctors think they know to be true in medicine has changed dramatically in the past several decades and will be different again in the decades to come, leaving them with a dilemma, according to Kevin T. Powell, MD, PhD, a pediatric hospitalist in St. Louis. If half of what doctors teach or know in medicine today will ultimately end up not being true, how do they know what to believe or accept?
While there is not a single satisfactory answer to that question, researchers can select research that gets doctors closer to reliable findings and steer them away from the barrage of poor-quality research that emerges from the current publish-or-perish system, Dr. Powell told his colleagues at the annual meeting of the American Academy of Pediatrics.
During his talk, Dr. Powell discussed the challenges and flaws with medical research as it is currently conducted, citing Doug Altman’s writings on these problems as early as 1994.
“The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent effort to find a solution,” wrote Mr. Altman, an English medical statistician (BMJ. 1994;308:283).
“We need less research, better research, and research done for the right reasons,” Mr. Altman concluded. “Abandoning using the number of publications as a measure of ability would be a start.”
In an interview, Dr. Powell described an unfortunate consequence of the publish-or-perish pressure in academic medicine: A glut of short-term, small studies with little clinical utility that researchers can complete in 1 or 2 years rather than the large, multicenter studies that take several years – and produce higher-quality findings – but cannot be turned into as many publications.
“We’re generating a lot of medical research findings that end up being false,” he said. “It’s a random walk in terms of getting to the truth rather than having an accurate process of getting to truth through evidence-based medicine.”
But he was hopeful, not cynical, about the way forward. By persuading people that medical research has changed for the worse over time and can change into something better, Dr. Powell saw potential for future research resulting in the same sort of public health achievements that research produced in the past, such as big reductions in smoking or sudden infant death syndrome.
Dr. Powell concluded his talk with a riff on Martin Luther’s 95 Theses, the 9.5 Theses, for a reformation of evidence-based medicine that together address the various shortcomings he discussed.
1. Recognize academic promotion as a bias, just like drug money.
2. Don’t confound statistically significant and clinically significant.
3. Use only significant figures.
4. Use the phrase “we did not DETECT a difference” and include power calculations.
5. Use confidence intervals instead of P values.
6. Use number needed to harm and number needed to treat instead of relative risk.
7. Absence of proof is not proof of absence. When there is insufficient randomized, controlled trial evidence, have an independent party estimate an effect based on non-RCT articles.
8. Any article implying clinical practice should change must include a counterpoint and a benefit cost analysis. Consider both effectiveness and safety.
9. Use postmarketing peer review.
9.5. Beware of research based on surveys.
Dr. Powell reported no relevant financial disclosures.
ORLANDO – What doctors think they know to be true in medicine has changed dramatically in the past several decades and will be different again in the decades to come, leaving them with a dilemma, according to Kevin T. Powell, MD, PhD, a pediatric hospitalist in St. Louis. If half of what doctors teach or know in medicine today will ultimately end up not being true, how do they know what to believe or accept?
While there is not a single satisfactory answer to that question, researchers can select research that gets doctors closer to reliable findings and steer them away from the barrage of poor-quality research that emerges from the current publish-or-perish system, Dr. Powell told his colleagues at the annual meeting of the American Academy of Pediatrics.
During his talk, Dr. Powell discussed the challenges and flaws with medical research as it is currently conducted, citing Doug Altman’s writings on these problems as early as 1994.
“The poor quality of much medical research is widely acknowledged, yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent effort to find a solution,” wrote Mr. Altman, an English medical statistician (BMJ. 1994;308:283).
“We need less research, better research, and research done for the right reasons,” Mr. Altman concluded. “Abandoning using the number of publications as a measure of ability would be a start.”
In an interview, Dr. Powell described an unfortunate consequence of the publish-or-perish pressure in academic medicine: A glut of short-term, small studies with little clinical utility that researchers can complete in 1 or 2 years rather than the large, multicenter studies that take several years – and produce higher-quality findings – but cannot be turned into as many publications.
“We’re generating a lot of medical research findings that end up being false,” he said. “It’s a random walk in terms of getting to the truth rather than having an accurate process of getting to truth through evidence-based medicine.”
But he was hopeful, not cynical, about the way forward. By persuading people that medical research has changed for the worse over time and can change into something better, Dr. Powell saw potential for future research resulting in the same sort of public health achievements that research produced in the past, such as big reductions in smoking or sudden infant death syndrome.
Dr. Powell concluded his talk with a riff on Martin Luther’s 95 Theses, the 9.5 Theses, for a reformation of evidence-based medicine that together address the various shortcomings he discussed.
1. Recognize academic promotion as a bias, just like drug money.
2. Don’t confound statistically significant and clinically significant.
3. Use only significant figures.
4. Use the phrase “we did not DETECT a difference” and include power calculations.
5. Use confidence intervals instead of P values.
6. Use number needed to harm and number needed to treat instead of relative risk.
7. Absence of proof is not proof of absence. When there is insufficient randomized, controlled trial evidence, have an independent party estimate an effect based on non-RCT articles.
8. Any article implying clinical practice should change must include a counterpoint and a benefit cost analysis. Consider both effectiveness and safety.
9. Use postmarketing peer review.
9.5. Beware of research based on surveys.
Dr. Powell reported no relevant financial disclosures.
EXPERT ANALYSIS FROM AAP 18
