Smartphone applications ( said U.K. researchers reporting a systematic review.
These apps are providing information that could lead to “potentially life-or-death decisions,” commented co-lead author Hywel C. Williams, MD, from the Centre of Evidence Based Dermatology, University of Nottingham (England).
“The one thing you mustn’t do in a situation where early diagnosis can make a difference between life and death is you mustn’t miss the melanoma,” he said in an interview.
“These apps were missing melanomas, and that’s very worrisome,” he commented.
The review included nine studies of skin cancer smartphone apps, including two apps, SkinScan and SkinVision, that have been given Conformit Europenne (CE) marks, allowing them to be marketed across Europe. These apps are also available in Australia and New Zealand, but not in the United States.
The review found that SkinScan was not able to identify any melanomas in the one study that assessed this app, while SkinVision had a relatively low sensitivity and specificity, with 12% of cancerous or precancerous lesions missed and 21% of benign lesions wrongly identified as cancerous.
This means that among 1,000 people with a melanoma prevalence of 3%, 4 of 30 melanomas would be missed, and 200 people would be incorrectly told that a mole was of high concern, the authors estimated.
The research was published by The BMJ on Feb. 10.
“Although I was broad minded on the potential benefit of apps for diagnosing skin cancer, I am now worried given the results of our study and the overall poor quality of studies used to test these apps,” Dr. Williams commented in a statement.
Coauthor Jac Dinnes, PhD, from the Institute of Applied Health Research at the University of Birmingham (England), added it is “really disappointing that there is not better quality evidence available to judge the efficacy of these apps.”
“It is vital that health care professionals are aware of the current limitations both in the technologies and in their evaluations,” she added.
The results also highlight the limitations of the regulatory system governing smartphone apps in that they are currently not subject to assessment by bodies such as the U.K.’s Medicines and Healthcare Products Regulatory Agency (MHRA), the authors commented.
“Regulators need to become alert to the potential harm that poorly performing algorithm-based diagnostic or risk monitoring apps create,” said co-lead author Jonathan J. Deeks, PhD, also at the Institute of Applied Health Research.
“We rely on the CE mark as a sign of quality, but the current CE mark assessment processes are not fit for protecting the public against the risks that these apps present.”
Speaking in an interview, Williams lamented the poor quality of the research that had been conducted. “These studies were not good enough,” he said, adding that “there’s no excuse for really poor study design and poor reporting.”
He would like to see the regulations tightened around AI apps purporting to inform decision making for the general public and suggests that these devices should be assessed by the MHRA. “I really do think a CE mark is not enough,” he said.
The team noted that the skin cancer apps “all include disclaimers that the results should only be used as a guide and cannot replace health care advice,” through which the manufacturers “attempt to evade any responsibility for negative outcomes experienced by users.”
Nevertheless, the “poor and variable performance” of the apps revealed by their review indicates that they “have not yet shown sufficient promise to recommend their use,” they concluded.
The “official approval” implied by a CE mark “will give consumers the impression that the apps have been assessed as effective and safe,” wrote Ben Goldacre, DataLab director, Nuffield Department of Primary Care, University of Oxford (England), and colleagues in an accompanying editorial.
“The implicit assumption is that apps are similarly low-risk technology” to devices such as sticking plasters and reading glasses, they comment.
“But shortcomings in diagnostic apps can have serious implications,” they warn. The “risks include psychological harm from health anxiety or ‘cyberchondria,’ and physical harm from misdiagnosis or overdiagnosis; for clinicians there is a risk of increased workload, and changes to ethical or legal responsibilities around triage, referral, diagnosis, and treatment.” There is also potential for “inappropriate resource use, and even loss of credibility for digital technology in general.”