Lowering the bar in an emergency
In the early days of the pandemic, the FDA, which regulates diagnostic tests, was criticized for not moving quickly enough to make testing widely available. For much of February, the only available test was the CDC’s, which initially had flaws when it was sent out to public health labs. Only on Feb. 29 did the FDA announce a new policy that made it easier for private labs and academic medical centers to make tests available as well.
Since then, the ongoing need for even more testing capacity across the United States has pushed the agency to loosen its typical requirements for manufacturers to prove that their tests are accurate before allowing them onto the market.
Normally, to get FDA approval, diagnostic makers need to run trials to gather evidence on their tests’ performance, a process that can take months or even years. The agency is currently skipping a lot of those steps by issuing emergency use authorizations.
Manufacturers are now required to run their COVID-19 tests on a minimum of 30 positive samples and 30 negative samples. They must demonstrate to the agency that the test has at least a 95% sensitivity, meaning it must correctly identify at least 95% of the positive samples as having the coronavirus, and 100% specificity, meaning that it must accurately identify all the negative samples as not having the coronavirus.
But the manufacturers are demonstrating their diagnostics’ performance with what’s known as “contrived samples,” which are not taken from actual patients. A contrived sample is made by taking coronavirus RNA made in a lab and putting it into a medium that mimics nasal mucus.
“This is supposed to represent a swab specimen, but it’s not a positive sample from a real patient, and that does make a real difference,” said Benjamin Pinsky, medical director of the Clinical Virology Laboratory for Stanford Health Care.
It’s not clear if the concentrations of virus on the simulated samples are representative of the full range of material taken from patients’ bodies in the real world. Pinksy says that it’s reasonable for the FDA to allow the use of contrived samples, because it makes it much faster for a manufacturer to run validation studies, and the need for speed has been pressing.
“But then we need to have studies to compare these assays and see how they perform with real-world samples, and whether some are more or less sensitive and whether some are more or less specific,” Pinsky said. “We don’t know the answer to these questions at this point.”
To compensate for the lower standard up front, experts say the FDA should track data on accuracy to make sure the tests are performing as expected, but this is easier said than done.
“In diagnostic tests in particular, it’s very difficult to know if something is failing,” said Alberto Gutierrez, former director of the FDA’s Office of In Vitro Diagnostics and Radiological Health. “When are you getting more erroneous results than you should? It’s not always easy to figure out.”
Swiss manufacturer Roche, whose test was authorized by the FDA on March 12, told ProPublica it couldn’t give specific numbers about its test’s actual rate of false negatives and false positives, though it said studies have demonstrated its test could detect very low levels of the coronavirus.
“Clinical studies, which take months to run and would be part of a regular (nonemergency) test approval process, are needed to give us an exact percentage of false negatives and false positives,” Roche spokesman Mike Weist wrote in an email. “We will continue to work with the FDA on ongoing studies post-EUA that will allow us to potentially say more in the future.”
Abbott, which makes a rapid COVID-19 test, also said that “performance characteristics, including accuracy data, will continue to be collected in the field.”
Abbott and the testing firms LabCorp and Quest Diagnostics all told ProPublica that tests should be used by physicians along with other information to form a diagnosis.