Abstract: The Controller Area Network (CAN) used inside modern vehicles has no built-in authentication, which exposes it to denial-of-service (DoS) and spoofing attacks. This paper evaluates an Extreme Gradient Boosting (XGBoost) intrusion detection system for binary CAN traffic on the CICIoV2024 dataset, with emphasis on repeated payload patterns and CAN-identifier shortcuts. Using five payload features, training-only SMOTE, and a stratified random 80:20 split of all 1,408,219 records, XGBoost achieved 99.64% accuracy and a 98.60% attack F1-score. The saved pipeline occupied 0.33 MB; batch inference averaged 5.34 µs per record, while single record inference through the Python interface took about 1 ms. However, the dataset contains only 3,575 unique payload patterns, and 99.80% of random-split test records share a payload pattern with training. When payload patterns were isolated using stratified group five-fold cross-validation, mean attack F1 fell to 70.08 ± 24.34%. A full-dataset leave-one-attack-CAN-ID-out evaluation further yielded mean recall of 43.05 ± 36.73% and mean F1 of 52.95 ± 36.33%, with recall ranging from 0% to 100% across the five held-out attack identifiers. The results show that random row-based evaluation overestimates performance on unseen payload patterns and unseen attack identifiers; compactness and batch speed alone do not establish deployment readiness.

Keywords: Internet of Vehicles; CAN bus; intrusion detection; XGBoost; CICIoV2024; SMOTE; repeated payload patterns; evaluation leakage