Preprints During the COVID-19 Pandemic: Public Health Emergencies and Medical Literature

Basic science and clinical research are the hallmarks of progress in biomedicine. Scientists rely on timely access to research findings to accelerate and strengthen their work, and clinicians depend on the latest data to ensure that the highest level of care reaches each patient’s bedside. Historically, academic journals have served as the gatekeepers of this knowledge, using expert peer review to cull the bad science from the good and ensure a meticulous standard of reporting before sharing information with the public. While robust and effective, the peer review process can, at times, be slow and cumbersome. During widespread emergencies, such as the current COVID-19 pandemic, delays in publication may handicap our ability to meet the urgent demands of the global scientific and medical communities. Indeed, academic journals initially struggled to manage the deluge of COVID-19–­related submissions, with potential reviewers similarly occupied on the clinical front lines and unable to promptly evaluate pending submissions. This impasse necessarily hindered the dissemination of relevant clinical data, which left physicians operatingwith limited evidence in some settings and, in turn, may have led to potentially avoidable harm.¹ Although many journals have since expedited their review processes in light of current pressing circumstances, these measures are not necessarily sustainable or scalable in the face of an increasingly expansive biomedical enterprise that will continue to face challenges of increasing urgency.² Moreover, it remains unclear to what extent quality has been sacrificed in exchange for this temporary expedience.

Scientific progress demands access to the rapid dissemination of robust data, and preprint servers are uniquely positioned to meet this need. Preprints are manuscripts released to the public before formal peer review and publication in an “official” indexed journal. Long used in mathematics and the physical sciences, preprint servers for the biomedical community such as medRxiv and bioRxiv have previously had limited traction because many have cited the risks of circulating information that may later be disputed or, worse, invalidated.^3-6 The risk-benefit calculus, however, must be carefully considered. Preprints provide a fast and wide-reaching means for sharing new discoveries. Submissions often undergo a brief screening process to ensure appropriateness, but otherwise largely forego scientific review before being posted online where the data become freely and widely available to the public.

The enthusiasm for preprints in the current era has demonstrated both the promise and peril of a free and wide distribution strategy.⁵ Early in the COVID-19 pandemic, Western hospitals were flooded with critically ill patients and relied on reports from providers in China, where the disease had struck first, to define the basic pathophysiology. Guan et al shared the clinical symptoms, laboratory abnormalities, and radiologic findings of 1,099 patients with COVID-19 through preprint servers in early February 2020, well before many American clinicians had gained direct experience with SARS-CoV-2.⁷ Their findings were published in the New England Journal of Medicine 1 month later,⁸ but the initial preprint provided an early window into the largest threats that COVID-19 would pose for patients and the health system and corroborated that the increasing number of patients with acute respiratory distress syndrome was on pace to dwarf the number of available ventilators around the world. Physicians responded in kind and used preprints as a mechanism to share their early experience with awake prone positioning and shared ventilation, which were critical components of the global strategy to contend with the limited ventilator supply during the height of the pandemic.^9-12

Despite these undisputed triumphs, hazards abound. Rapidly disseminating new findings via preprint servers neither implies shoddy science nor absolves investigators of the need for critical review, yet it provides opportunities for both. As an example, Gautret et al first shared their open-label study examining the efficacy of hydroxychloroquine and azithromycin for COVID-19 by using preprint publication.¹³ The study did not meet a priori sample size requirements, it incorporated a trial arm that was not prespecified, and it was promptly contradicted by a second trial, which raised concern about the validity of the findings.¹⁴ While the study was ultimately published in a journal, preprint allowed these often-misquoted data to circulate far longer than would have been possible were expert peer review to have requested strengthening of the findings.¹⁵ Under ideal circumstances, peer review serves to capture and address these types of methodologic errors in order to avoid the publication of misleading or incomplete results. By foregoing the peer review process when posting a preprint manuscript, investigators have an equal opportunity to share good and bad science with a community that may lack the expertise to distinguish between the two. Indeed, the results posted by Gautret et al were immediately amplified by media and policy makers alike, who touted hydroxychloroquine as a “game-changing” panacea despite the preliminary nature of the findings.¹⁶ Irrational exuberance then prompted drug hoarding and supply issues before more robust studies alerted providers to the potential adverse effects of this regimen and the limited evidence of any efficacy.^17,18

Ultimately, both preprints and perfunctory peer review afford minimal safeguards to prevent the adoption of incomplete or misinterpreted results. While envisioned as a tool for scientific collaboration, preprints do have a broader readership that may be unaware of fundamental differences between a preprint manuscript and one reviewed by a rigorous academic journal. Considering the reliability of findings from these different domains as equivalent could ultimately cause public harm.

To be sure, there are ways to enhance the current system and limit opportunities for misguided enthusiasm. Firstly, preprint servers can be difficult to navigate. Limited indexing in disparate silos that are distinct from the rest of the literature (ie, the U.S. National Library of Medicine’s PubMed) make relevant articles challenging to identify and, in some instances, relegate the curation of new papers to social media platforms. Resources to aggregate and query the growing database of submissions would improve our ability to identify appropriate articles and use this preliminary evidence base.

Secondly, once an article has been unearthed, few tools exist to help nonexpert readers evaluate the quality of the research. Many consumers, inclusive of other scientists, may not share the investigators’ expertise. Preprint platforms might aid readers by compiling metrics to indicate study quality. For example, a voting and commenting function to permit a form of crowd-sourced peer review, while imperfect, would allow subject matter experts to communicate the value of a submission and point out errors. Weighting of votes by the h-index or institution of each “reviewer” might further enhance the value of this crowd-sourced evaluation. Additionally, the site could indicate when there is broad agreement on a particular critique by alerting readers to an established limitation of the study in question. Ultimately, numerous such mechanisms might be considered, but all share the overarching goal of guiding readers to exercise appropriate caution in interpreting a study in order to avoid unfettered acceptance of flawed research.

Thirdly, preprint servers can minimize the circulation of outdated research by highlighting manuscripts whose findings have subsequently been disproven. There are certainly complexities in distinguishing between a scientific difference of opinion and an invalidated research finding, but rather than avoid these challenging topics, systems must acknowledge this critical nuance and address it transparently. Indeed, the more prominent preprint servers have already begun to limit the dissemination of clearly misleading research in acknowledgment of this responsibility.^1,19 The biomedical community must continue to engage in open dialogue to determine where the filter is set between blocking harmful pseudoscience and honest efforts to evaluate research validity.

Lastly, while prominent preprint platforms continue to limit the dissemination of opinion pieces, clinical recommendations, and review articles, these submissions are among the most urgently useful content during a pandemic, as evidenced by the ongoing stream of published consensus statements and clinical guidelines. Moreover, these pieces are often invited unilaterally by journal editors and are less likely to undergo peer review before formal publication. Clinicians hunger for practical insights during this pandemic, and allowing guidelines and reviews to be posted rapidly—and to be flagged accordingly as “nonoriginal” research—could spark timely dialogue that might ultimately accelerate science.

Preprint servers do not obviate the need for critical scientific appraisal of their content; however, their risks are not an excuse to limit their adoption as an effective and practical data sharing platform. By embracing the rapid and transparent dissemination of data afforded by preprints, and thoughtfully navigating the caveats of applying new research (non–peer-­reviewed manuscripts or otherwise), we will have added a powerful instrument to the biomedical armamentarium with lasting implications beyond the current crisis.

Dr Guterman reported receipt of grants from the National Institute of Neurological Disorders and Stroke (1K23NS116128-01), the National Institute on Aging (5R01AG056715), the American Academy of Neurology, as well as consulting fees from Marinus, Inc, that are outside the submitted work. Dr Braunstein reported no potential conflicts of interest. 

Basic science and clinical research are the hallmarks of progress in biomedicine. Scientists rely on timely access to research findings to accelerate and strengthen their work, and clinicians depend on the latest data to ensure that the highest level of care reaches each patient’s bedside. Historically, academic journals have served as the gatekeepers of this knowledge, using expert peer review to cull the bad science from the good and ensure a meticulous standard of reporting before sharing information with the public. While robust and effective, the peer review process can, at times, be slow and cumbersome. During widespread emergencies, such as the current COVID-19 pandemic, delays in publication may handicap our ability to meet the urgent demands of the global scientific and medical communities. Indeed, academic journals initially struggled to manage the deluge of COVID-19–­related submissions, with potential reviewers similarly occupied on the clinical front lines and unable to promptly evaluate pending submissions. This impasse necessarily hindered the dissemination of relevant clinical data, which left physicians operatingwith limited evidence in some settings and, in turn, may have led to potentially avoidable harm.¹ Although many journals have since expedited their review processes in light of current pressing circumstances, these measures are not necessarily sustainable or scalable in the face of an increasingly expansive biomedical enterprise that will continue to face challenges of increasing urgency.² Moreover, it remains unclear to what extent quality has been sacrificed in exchange for this temporary expedience.

Scientific progress demands access to the rapid dissemination of robust data, and preprint servers are uniquely positioned to meet this need. Preprints are manuscripts released to the public before formal peer review and publication in an “official” indexed journal. Long used in mathematics and the physical sciences, preprint servers for the biomedical community such as medRxiv and bioRxiv have previously had limited traction because many have cited the risks of circulating information that may later be disputed or, worse, invalidated.^3-6 The risk-benefit calculus, however, must be carefully considered. Preprints provide a fast and wide-reaching means for sharing new discoveries. Submissions often undergo a brief screening process to ensure appropriateness, but otherwise largely forego scientific review before being posted online where the data become freely and widely available to the public.

The enthusiasm for preprints in the current era has demonstrated both the promise and peril of a free and wide distribution strategy.⁵ Early in the COVID-19 pandemic, Western hospitals were flooded with critically ill patients and relied on reports from providers in China, where the disease had struck first, to define the basic pathophysiology. Guan et al shared the clinical symptoms, laboratory abnormalities, and radiologic findings of 1,099 patients with COVID-19 through preprint servers in early February 2020, well before many American clinicians had gained direct experience with SARS-CoV-2.⁷ Their findings were published in the New England Journal of Medicine 1 month later,⁸ but the initial preprint provided an early window into the largest threats that COVID-19 would pose for patients and the health system and corroborated that the increasing number of patients with acute respiratory distress syndrome was on pace to dwarf the number of available ventilators around the world. Physicians responded in kind and used preprints as a mechanism to share their early experience with awake prone positioning and shared ventilation, which were critical components of the global strategy to contend with the limited ventilator supply during the height of the pandemic.^9-12

Despite these undisputed triumphs, hazards abound. Rapidly disseminating new findings via preprint servers neither implies shoddy science nor absolves investigators of the need for critical review, yet it provides opportunities for both. As an example, Gautret et al first shared their open-label study examining the efficacy of hydroxychloroquine and azithromycin for COVID-19 by using preprint publication.¹³ The study did not meet a priori sample size requirements, it incorporated a trial arm that was not prespecified, and it was promptly contradicted by a second trial, which raised concern about the validity of the findings.¹⁴ While the study was ultimately published in a journal, preprint allowed these often-misquoted data to circulate far longer than would have been possible were expert peer review to have requested strengthening of the findings.¹⁵ Under ideal circumstances, peer review serves to capture and address these types of methodologic errors in order to avoid the publication of misleading or incomplete results. By foregoing the peer review process when posting a preprint manuscript, investigators have an equal opportunity to share good and bad science with a community that may lack the expertise to distinguish between the two. Indeed, the results posted by Gautret et al were immediately amplified by media and policy makers alike, who touted hydroxychloroquine as a “game-changing” panacea despite the preliminary nature of the findings.¹⁶ Irrational exuberance then prompted drug hoarding and supply issues before more robust studies alerted providers to the potential adverse effects of this regimen and the limited evidence of any efficacy.^17,18

Ultimately, both preprints and perfunctory peer review afford minimal safeguards to prevent the adoption of incomplete or misinterpreted results. While envisioned as a tool for scientific collaboration, preprints do have a broader readership that may be unaware of fundamental differences between a preprint manuscript and one reviewed by a rigorous academic journal. Considering the reliability of findings from these different domains as equivalent could ultimately cause public harm.

To be sure, there are ways to enhance the current system and limit opportunities for misguided enthusiasm. Firstly, preprint servers can be difficult to navigate. Limited indexing in disparate silos that are distinct from the rest of the literature (ie, the U.S. National Library of Medicine’s PubMed) make relevant articles challenging to identify and, in some instances, relegate the curation of new papers to social media platforms. Resources to aggregate and query the growing database of submissions would improve our ability to identify appropriate articles and use this preliminary evidence base.

Secondly, once an article has been unearthed, few tools exist to help nonexpert readers evaluate the quality of the research. Many consumers, inclusive of other scientists, may not share the investigators’ expertise. Preprint platforms might aid readers by compiling metrics to indicate study quality. For example, a voting and commenting function to permit a form of crowd-sourced peer review, while imperfect, would allow subject matter experts to communicate the value of a submission and point out errors. Weighting of votes by the h-index or institution of each “reviewer” might further enhance the value of this crowd-sourced evaluation. Additionally, the site could indicate when there is broad agreement on a particular critique by alerting readers to an established limitation of the study in question. Ultimately, numerous such mechanisms might be considered, but all share the overarching goal of guiding readers to exercise appropriate caution in interpreting a study in order to avoid unfettered acceptance of flawed research.

Thirdly, preprint servers can minimize the circulation of outdated research by highlighting manuscripts whose findings have subsequently been disproven. There are certainly complexities in distinguishing between a scientific difference of opinion and an invalidated research finding, but rather than avoid these challenging topics, systems must acknowledge this critical nuance and address it transparently. Indeed, the more prominent preprint servers have already begun to limit the dissemination of clearly misleading research in acknowledgment of this responsibility.^1,19 The biomedical community must continue to engage in open dialogue to determine where the filter is set between blocking harmful pseudoscience and honest efforts to evaluate research validity.

Lastly, while prominent preprint platforms continue to limit the dissemination of opinion pieces, clinical recommendations, and review articles, these submissions are among the most urgently useful content during a pandemic, as evidenced by the ongoing stream of published consensus statements and clinical guidelines. Moreover, these pieces are often invited unilaterally by journal editors and are less likely to undergo peer review before formal publication. Clinicians hunger for practical insights during this pandemic, and allowing guidelines and reviews to be posted rapidly—and to be flagged accordingly as “nonoriginal” research—could spark timely dialogue that might ultimately accelerate science.

Preprint servers do not obviate the need for critical scientific appraisal of their content; however, their risks are not an excuse to limit their adoption as an effective and practical data sharing platform. By embracing the rapid and transparent dissemination of data afforded by preprints, and thoughtfully navigating the caveats of applying new research (non–peer-­reviewed manuscripts or otherwise), we will have added a powerful instrument to the biomedical armamentarium with lasting implications beyond the current crisis.

Dr Guterman reported receipt of grants from the National Institute of Neurological Disorders and Stroke (1K23NS116128-01), the National Institute on Aging (5R01AG056715), the American Academy of Neurology, as well as consulting fees from Marinus, Inc, that are outside the submitted work. Dr Braunstein reported no potential conflicts of interest. 

Basic science and clinical research are the hallmarks of progress in biomedicine. Scientists rely on timely access to research findings to accelerate and strengthen their work, and clinicians depend on the latest data to ensure that the highest level of care reaches each patient’s bedside. Historically, academic journals have served as the gatekeepers of this knowledge, using expert peer review to cull the bad science from the good and ensure a meticulous standard of reporting before sharing information with the public. While robust and effective, the peer review process can, at times, be slow and cumbersome. During widespread emergencies, such as the current COVID-19 pandemic, delays in publication may handicap our ability to meet the urgent demands of the global scientific and medical communities. Indeed, academic journals initially struggled to manage the deluge of COVID-19–­related submissions, with potential reviewers similarly occupied on the clinical front lines and unable to promptly evaluate pending submissions. This impasse necessarily hindered the dissemination of relevant clinical data, which left physicians operatingwith limited evidence in some settings and, in turn, may have led to potentially avoidable harm.¹ Although many journals have since expedited their review processes in light of current pressing circumstances, these measures are not necessarily sustainable or scalable in the face of an increasingly expansive biomedical enterprise that will continue to face challenges of increasing urgency.² Moreover, it remains unclear to what extent quality has been sacrificed in exchange for this temporary expedience.

Scientific progress demands access to the rapid dissemination of robust data, and preprint servers are uniquely positioned to meet this need. Preprints are manuscripts released to the public before formal peer review and publication in an “official” indexed journal. Long used in mathematics and the physical sciences, preprint servers for the biomedical community such as medRxiv and bioRxiv have previously had limited traction because many have cited the risks of circulating information that may later be disputed or, worse, invalidated.^3-6 The risk-benefit calculus, however, must be carefully considered. Preprints provide a fast and wide-reaching means for sharing new discoveries. Submissions often undergo a brief screening process to ensure appropriateness, but otherwise largely forego scientific review before being posted online where the data become freely and widely available to the public.

The enthusiasm for preprints in the current era has demonstrated both the promise and peril of a free and wide distribution strategy.⁵ Early in the COVID-19 pandemic, Western hospitals were flooded with critically ill patients and relied on reports from providers in China, where the disease had struck first, to define the basic pathophysiology. Guan et al shared the clinical symptoms, laboratory abnormalities, and radiologic findings of 1,099 patients with COVID-19 through preprint servers in early February 2020, well before many American clinicians had gained direct experience with SARS-CoV-2.⁷ Their findings were published in the New England Journal of Medicine 1 month later,⁸ but the initial preprint provided an early window into the largest threats that COVID-19 would pose for patients and the health system and corroborated that the increasing number of patients with acute respiratory distress syndrome was on pace to dwarf the number of available ventilators around the world. Physicians responded in kind and used preprints as a mechanism to share their early experience with awake prone positioning and shared ventilation, which were critical components of the global strategy to contend with the limited ventilator supply during the height of the pandemic.^9-12

Despite these undisputed triumphs, hazards abound. Rapidly disseminating new findings via preprint servers neither implies shoddy science nor absolves investigators of the need for critical review, yet it provides opportunities for both. As an example, Gautret et al first shared their open-label study examining the efficacy of hydroxychloroquine and azithromycin for COVID-19 by using preprint publication.¹³ The study did not meet a priori sample size requirements, it incorporated a trial arm that was not prespecified, and it was promptly contradicted by a second trial, which raised concern about the validity of the findings.¹⁴ While the study was ultimately published in a journal, preprint allowed these often-misquoted data to circulate far longer than would have been possible were expert peer review to have requested strengthening of the findings.¹⁵ Under ideal circumstances, peer review serves to capture and address these types of methodologic errors in order to avoid the publication of misleading or incomplete results. By foregoing the peer review process when posting a preprint manuscript, investigators have an equal opportunity to share good and bad science with a community that may lack the expertise to distinguish between the two. Indeed, the results posted by Gautret et al were immediately amplified by media and policy makers alike, who touted hydroxychloroquine as a “game-changing” panacea despite the preliminary nature of the findings.¹⁶ Irrational exuberance then prompted drug hoarding and supply issues before more robust studies alerted providers to the potential adverse effects of this regimen and the limited evidence of any efficacy.^17,18

Ultimately, both preprints and perfunctory peer review afford minimal safeguards to prevent the adoption of incomplete or misinterpreted results. While envisioned as a tool for scientific collaboration, preprints do have a broader readership that may be unaware of fundamental differences between a preprint manuscript and one reviewed by a rigorous academic journal. Considering the reliability of findings from these different domains as equivalent could ultimately cause public harm.

To be sure, there are ways to enhance the current system and limit opportunities for misguided enthusiasm. Firstly, preprint servers can be difficult to navigate. Limited indexing in disparate silos that are distinct from the rest of the literature (ie, the U.S. National Library of Medicine’s PubMed) make relevant articles challenging to identify and, in some instances, relegate the curation of new papers to social media platforms. Resources to aggregate and query the growing database of submissions would improve our ability to identify appropriate articles and use this preliminary evidence base.

Secondly, once an article has been unearthed, few tools exist to help nonexpert readers evaluate the quality of the research. Many consumers, inclusive of other scientists, may not share the investigators’ expertise. Preprint platforms might aid readers by compiling metrics to indicate study quality. For example, a voting and commenting function to permit a form of crowd-sourced peer review, while imperfect, would allow subject matter experts to communicate the value of a submission and point out errors. Weighting of votes by the h-index or institution of each “reviewer” might further enhance the value of this crowd-sourced evaluation. Additionally, the site could indicate when there is broad agreement on a particular critique by alerting readers to an established limitation of the study in question. Ultimately, numerous such mechanisms might be considered, but all share the overarching goal of guiding readers to exercise appropriate caution in interpreting a study in order to avoid unfettered acceptance of flawed research.

Thirdly, preprint servers can minimize the circulation of outdated research by highlighting manuscripts whose findings have subsequently been disproven. There are certainly complexities in distinguishing between a scientific difference of opinion and an invalidated research finding, but rather than avoid these challenging topics, systems must acknowledge this critical nuance and address it transparently. Indeed, the more prominent preprint servers have already begun to limit the dissemination of clearly misleading research in acknowledgment of this responsibility.^1,19 The biomedical community must continue to engage in open dialogue to determine where the filter is set between blocking harmful pseudoscience and honest efforts to evaluate research validity.

Lastly, while prominent preprint platforms continue to limit the dissemination of opinion pieces, clinical recommendations, and review articles, these submissions are among the most urgently useful content during a pandemic, as evidenced by the ongoing stream of published consensus statements and clinical guidelines. Moreover, these pieces are often invited unilaterally by journal editors and are less likely to undergo peer review before formal publication. Clinicians hunger for practical insights during this pandemic, and allowing guidelines and reviews to be posted rapidly—and to be flagged accordingly as “nonoriginal” research—could spark timely dialogue that might ultimately accelerate science.

Preprint servers do not obviate the need for critical scientific appraisal of their content; however, their risks are not an excuse to limit their adoption as an effective and practical data sharing platform. By embracing the rapid and transparent dissemination of data afforded by preprints, and thoughtfully navigating the caveats of applying new research (non–peer-­reviewed manuscripts or otherwise), we will have added a powerful instrument to the biomedical armamentarium with lasting implications beyond the current crisis.

Dr Guterman reported receipt of grants from the National Institute of Neurological Disorders and Stroke (1K23NS116128-01), the National Institute on Aging (5R01AG056715), the American Academy of Neurology, as well as consulting fees from Marinus, Inc, that are outside the submitted work. Dr Braunstein reported no potential conflicts of interest.