Predicting Outcomes
Experiments to see whether an AI system can forecast RA disease activity at the next clinic visit are in their early stages.
Dr. Yazdany and colleagues used EHR data from UCSF and Zuckerberg San Francisco General Hospital on patients with two RA diagnostic codes at 30 days apart, who had at least one disease-modifying antirheumatic drug prescription and two Clinical Disease Activity Index (CDAI) scores 30 days apart.
One model, designed to predict CDAI at the next visit by “playing the odds” based on clinical experience, showed that about 60% of patients at UCSF achieved treat-to-target goals, while the remaining 40% did not.
This model performed barely better than pure chance, with an area under the receiver operating characteristic curve (AUC) of 0.54.
A second model that included the patient’s last CDAI score also fared little better than a roll of the dice, with an AUC of 0.55.
However, a neural network or “deep learning” model designed to process data akin to the way that the human brain works performed much better at predicting outcomes at the second visit, with an AUC of 0.91.
Applying the UCSF-trained neural network model to the Zuckerberg San Francisco General Hospital population, with different patient characteristics from those of UCSF, the AUC was 0.74. Although this result was not as good as that seen when applied to UCSF patients, it demonstrated that the model retains some predictive capability across different hospital systems, Dr. Yazdany said.
The next steps, she said, are to build more robust models based on vast and varied patient data pools that will allow the predictive models to be generalized across various healthcare settings.
The Here and Now
In the Q & A following the presentation, an audience member said that the study was “very cool stuff.”
“Is there a way to sort of get ahead and think of the technology that we’re starting to pilot? Hospitals are already using AI scribes, for example, to collect the data that is going to make it much easier to feed it to the predictive analytics that we’re going to use,” she said.
Dr. Yazdany replied that “over the last couple of years, one of the projects that we’ve worked on is to interview rheumatologists who are participating in the RISE registry about the ways that they are collecting [patient-reported outcomes], and it has been fascinating: A vast majority of people are still using paper forms.”
“The challenge is that our patient populations are very diverse. Technology, and especially filling out forms via online platforms, doesn’t work for everybody, and in some ways, filling out the paper forms when you go to the doctor’s office is a great equalizer. So, I think that we have some real challenges, and the solutions have to be embedded in the real world,” she added.
Dr. Yazdany’s research was supported by grants from the Agency for Healthcare Research & Quality and the National Institutes of Health. She disclosed consulting fees and/or research support from AstraZeneca, Aurinia, Bristol Myers Squibb, Gilead, and Pfizer.
A version of this article appeared on Medscape.com.