报告详情
Six Models, Six Winners: Split Variance in Clinical Machine Learning Comparisons
编号：132
访问权限：仅限参会人
更新：2026-11-10 16:51:50 浏览：13次
Online
报告开始：2026年11月21日 16:00（Asia/Kolkata）
报告时间：15min
所在会场：[S5] Data Science and Intelligent Analytics [S5-2] Data Science and Intelligent Analytics
摘要
Clinical machine learning papers routinely compare several models on one train/test split and report the winner. We measured what that winner is worth. On six public clinical datasets we ran six standard classifiers over 200 stratified splits each. On the Cleveland heart disease data all six models won at least one split, and the best-on-average model won only 34% of them. One model's AUROC ranged from 0.832 to 0.961 across splits while the six models' average AUROCs spanned 0.021, so the noise was six times the signal. From the same data one split supports the claim that random forest beats logistic regression by 0.045 and another supports the reverse by 0.052. The chance that one split reverses the published order follows Φ(−z), with z the gap divided by the standard deviation of the paired difference; observed and predicted rates agreed to within 0.04 on four of five non-degenerate datasets. Drawing ten splits and keeping the best gained a median 0.040 AUROC on heart disease, nearly twenty times the real gap between the top two models. Not every benchmark behaves this way: on Framingham, where logistic regression leads by 0.031, it won 99% of splits. Five-fold cross-validation cut the estimator's standard deviation to a median 23% of a single split and repeated five-fold to 14%, lifting winner agreement on heart disease from 0.53 to 0.83. The standardised gap costs nothing to compute, and reporting it would tell a reader whether the ranking means anything.
关键词
model comparison, benchmarking, cross-validation, clinical prediction, evaluation variance, reproducibility, AUROC, machine learning in medicine,model comparison,benchmarking,Cross validation,clinical prediction,evaluation variance,AUROC,machine learning,
报告人
Karan Tejpal
Analytics Engineer Treasure AI稿件作者
全部评论
重要日期
-
会议日期
11月20日
2026
至11月22日
2026
-
10月31日 2026
初稿截稿日期
-
11月07日 2026
提前注册日期
-
11月22日 2026
注册截止日期
主办单位
United Societies of Science
承办单位
Pondicherry University
协办单位
IEEE Section
IEEE Madras Section
IEEE Madras Section
联系方式
- ICon INDIA 2026
- ic******@usssociety.org
发表评论