In his thought provoking keynote address at the annual Association of VA Hematology and Oncology meeting in Denver on September 15, 2017, John P.A. Ioannidis, MD, DSc, of Stanford University challenged the current reliance on clinical trial research and meta-analyses. Ioannidis has published numerous examinations of medical literature and found significant biases and potential limitations, and as he noted, few health care providers are aware of the poor quality of the research and its limitations.
Ioannidis’ studies contend that in most medical research “there is no evidence to build our decision making,” although he acknowledged that hematology and oncology research is a little better than other parts of the medical research. In a study of indexed articles, for example, just 19% had at least one outcome with high quality evidence. He attributes this to a number of types of flaws, including bias, an increase in research conducted in regions without a strong tradition, and degrees of freedom to conduct research, even in clinical trials.
Complicating matters even further, Ioannidis pointed out that many of the most highly cited studies were later refuted in other studies. In a re-examination of study data made available by researchers publishing in BMJ and PLoS Medicine, Ioannidis and his colleagues found that they could reproduce a fraction of the results. Moreover, he argued, “even when we can trust the data, most clinical research is not useful for making decisions.”
To measure the usefulness of a study, Ioannidis and his colleagues noted 8 different quality measures:
- Problem base;
- Context placement;
- Information gain;
- Pragmatism;
- Patient-centeredness;
- Value for money;
- Feasibility; and
- Transparency.
According to Ioannidis, few trials or studies hit 5 of 8 measures, and he has yet to find a study that incorporates all 8. Even articles in the most prestigious journals tend to score fairly low, he noted.
Meta-analyses are not immune to these structural problems, according to Ioannidis. Close to 100,000 meta-analysis have been published, with nearly 1,000 new studies published each month. Some, he noted, have been “hijacked to serve other agendas,” while many others simply summarized biased information.
Despite these concerns, Ioannidis remains optimistic. “We need to maximize diversity, minimize co-occurrence and incoherence” in the medical literature he argued.
