Abstract: Detecting both known and unknown cyberattacks remains a central challenge for intrusion detection systems (IDS). Signature-based IDS are effective for known threats but fail against novel attacks, while anomaly-based IDS can detect unknowns as outliers yet cannot recognize them if they reappear in new forms. Hybrid IDS improve coverage by combining both approaches, but most existing designs still treat unknown attacks as one-time anomalies without capturing their behavioral patterns. This limitation is critical in adaptive threat landscapes where attackers continuously refine their methods to mimic legitimate traffic, making stealthy intrusions especially difficult to detect. In this paper, we propose a Behavior-Based Hybrid IDS (Behavior-HIDS) that integrates (i) a GA-optimized Support Vector Machine (SVM) for supervised detection of known attacks, with (ii) a two-stage anomaly detection pipeline (Isolation Forest and Extended Isolation Forest) for identifying unseen threats. Crucially, our system goes beyond detection by clustering unknown attacks into coherent behavioral groups using autoencoder embeddings and HDBSCAN. These clusters are incorporated into retraining, thereby creating a form of behavioral memory that enables the IDS to recognize future variants of previously unseen attacks. Comprehensive experiments on NSL-KDD, CICIDS2017, and UNSW-NB15 confirm that Behavior-HIDS consistently outperforms classical hybrid IDS approaches. On NSL-KDD, it achieves 92.49% accuracy and 92.30% F1-score, with similar improvements observed on the other datasets. By combining anomaly detection with behavioral learning, our framework advances IDS design toward adaptive and evolving defense mechanisms.

Key words: intrusion detection system (IDS), outlier analysis, hybrid IDS, unknown attack detection, behavioral clustering, autoencoder, extended isolation forest (EIF), support vector machine (SVM)