Five common variable selection strategies failed to produce a statistical model that accurately predicted ventral hernia recurrence in an investigation published in the Journal of Surgical Research.
The finding matters because those five techniques – expert opinion and various multivariate regression and bootstrapping strategies – have been widely used in previous studies to create risk scores for ventral hernia recurrence. The new study calls the value of existing scoring systems into question (J Surg Res. 2016 Nov;206[1]:159-67. doi: 10.1016/j.jss.2016.07.042).
©Dmitrii Kotin/Thinkstock.com
The lack of external validation in many studies leads to medical findings that often can’t be confirmed by subsequent studies. It’s a problem that has contributed to skepticism about research results in both the medical community and the general public, they said.
“This study demonstrates the importance of true external validation on an external data set. Simply splitting a data set and validating [internally] does not appear to be an adequate assessment of predictive accuracy. … We recommend that future researchers consider using and presenting the results of multiple variable selection strategies [and] focus on presenting predictive accuracy on external data sets to validate their model,” the team concluded.
The original goal of the project was to identify the best predictors of ventral hernia recurrence since suggestions from past studies have varied. The team first used a prospective database of 790 ventral hernia repair patients to identify predictors of recurrence. Of that group, 526 patients – 173 (32.9%) of whom had a recurrence after a median follow-up of 20 months – were used to identify risk variables using expert opinion, selective stepwise regression, liberal stepwise regression, and bootstrapping with both restrictive and liberal internal resampling.
The team used the remaining 264 patients to confirm the findings. As in previous studies, internal validation worked: all five models had a Harrell’s C-statistic of about 0.76, which is considered reasonable, Dr. Holihan and her associates reported.
However, when the investigators applied their models to a second database of 1,225 patients followed for a median of 9 months – with 155 recurrences (12.7%) – they were not much better at predicting recurrence than a coin toss, with C-statistic values of about 0.56.
Some variables made the cut with all five selection techniques, including hernia type, wound class, and albumin levels, which are related to how well the wound heals. Other variables were significant in some selection strategies but not others, including smoking status, open versus laparoscopic approach, and mesh use.
At least for now, clinical intuition remains important for assessing rerupture risk, they said.
The National Institutes of Health funded the work. Author disclosures were not reported.