IL-IDS: an incremental learning approach with confined data streams for intrusion detection

Abstract In cyberspace, intrusion and detection constitute dynamic and continuous game processes, where data streams are generated incrementally during intrusion. To mitigate intrusions and safeguard assets effectively, it is imperative to take prompt actions based on real-time detection and analysis of the currently available data streams. However, existing approaches that rely on complete and clean data struggle to keep pace with the continuous real-time flow of new network data. To address this issue, we introduce IL-IDS (Incremental Learning for Intrusion Detection Systems), a novel intrusion detection approach that utilizes incremental learning to enable accurate and timely detection of intrusions in real-world scenarios, where the need for real-time processing and learning from newly generated traffic data is paramount. IL-IDS performs in scenarios with limited data availability, where it initially transforms textual data streams into vectorized representations and leverages a variation autoencoder (VAE) to compress these vectors, efficiently extracting their latent features. Then a classifier is trained to distinguish attack and normal behaviors, and a three-way decision method is employed to establish a boundary for ambiguous data that pose challenges in direct classification. Concurrently, threat intelligence is integrated into this process to enhance the accuracy of decision-making. We validate the effectiveness and efficiency of IL-IDS with experiments on real-world deployments during an international activity, highlighting its robustness and reliability in intrusion detection applications, especially under conditions of confined data streams. Notably, IL-IDS has exhibited comparable accuracy and recall results, and attains exceptional 99.93% precision and 96.83% F1-score, which demonstrates a notable improvement of 5.27% and 2.59% respectively in comparison to intrusion detection models trained on complete and readily available data.