Recurrent Biases and Fallacies in Dataset-Driven Intrusion Detection Research
Muhammad M (2026)
Publication Type: Conference contribution
Publication year: 2026
Publisher: Institute of Electrical and Electronics Engineers Inc.
Conference Proceedings Title: 2025 Cyber Awareness and Research Symposium, CARS 2025
Event location: Grand Forks, ND
ISBN: 9798331596286
DOI: 10.1109/CARS67163.2025.11337551
Abstract
Intrusion Detection System (IDS) research often relies on benchmark datasets, yet the reasoning behind their selection is prone to error. We apply a 20-item taxonomy of cognitive biases and logical fallacies to IDS studies published between 2020 and 2024. The taxonomy results show that five pitfalls recur consistently: Confirmation Bias (B1), Sampling Bias (B2), Base Rate Fallacy (F1), Cherry Picking (F2), and Hasty Generalization (F3). These appear across studies using NSL-KDD, CICIDS2017, CSE-CIC-IDS2018, UNSW-NB15, CI-CDDoS2019, and IoT datasets, revealing a pattern of dataset dependence and overstated generalizations. We argue that recognizing these recurrent issues can guide toward improved benchmarking and reporting practices, strengthening the validity of IDS research.
Authors with CRIS profile
How to cite
APA:
Muhammad, M. (2026). Recurrent Biases and Fallacies in Dataset-Driven Intrusion Detection Research. In 2025 Cyber Awareness and Research Symposium, CARS 2025. Grand Forks, ND, US: Institute of Electrical and Electronics Engineers Inc..
MLA:
Muhammad, Mamdouh. "Recurrent Biases and Fallacies in Dataset-Driven Intrusion Detection Research." Proceedings of the 2025 Cyber Awareness and Research Symposium, CARS 2025, Grand Forks, ND Institute of Electrical and Electronics Engineers Inc., 2026.
BibTeX: Download