A Multi-Layer Hybrid Machine Learning Framework for Intrusion Detection Using the UNSW-NB15 Dataset
Keywords:
intrusion detection systems; hybrid machine learning; autoencoder; gradient boosting; UNSW-NB15Abstract
Network intrusion detection increasingly combines rule-based, supervised, and anomaly-based methods on the premise that complementary detectors can increase robustness. In this study, that assumption is tested using the UNSW-NB15 benchmark and a structured three-layer framework comprising interpretable high-confidence rules, supervised machine-learning classifiers, and a normal-only autoencoder. A review of the data revealed considerable repetition among the predictor profiles, so leakage-aware grouped validation was used during model selection to ensure that identical predictor profiles did not appear in both the development and validation subsets. Logistic Regression, Random Forest, and Histogram-based Gradient Boosting were trained under the original class distribution, with balanced class weighting and SMOTENC. The autoencoder achieved a validation ROC-AUC of 0.9190, an attack recall of 0.7396, and a false-positive rate of 0.0757, while class-weighted Gradient Boosting provided the best balance among the supervised methods. Although the initially proposed ungated hierarchy improved sensitivity to attacks, it also led to an accumulation of false positives; therefore, a confidence-gated rule was introduced that allowed the autoencoder to override a supervised normal prediction only if the supervised attack probability was at least 0.45. On the untouched official test set, class-weighted Gradient Boosting achieved an accuracy of 0.9068, a balanced accuracy of 0.9031, an attack precision of 0.8964, an attack recall of 0.9393, and a false-positive rate of 0.1330. The gated hybrid raised recall to 0.9450 but lowered balanced accuracy to 0.8871 and increased the false-positive rate to 0.1708. McNemar's exact test showed significantly different paired error performance (p <.001): the hybrid detected 259 additional attacks but introduced 1,399 extra false positives. On a separate test subset, the gap in balanced accuracy decreased to 0.0019. These results indicate that hybridization is not inherently better; rather, its usefulness depends on leakage-aware evaluation, controlled fusion, category-specific complementarity, and the operational cost of false alarms
References
Ahmad, M., Riaz, Q., Zeeshan, M., Tahir, H., Haider, S. A. and Khan, M. S. (2021) Intrusion detection in the Internet of Things using supervised machine learning based on application and transport layer features using the UNSW-NB15 data set. EURASIP Journal on Wireless Communications and Networking, 2021, article 10. https://doi.org/10.1186/s13638-021-01893-8
Al-Daweri, M. S., Ariffin, K. A. Z., Abdullah, S., & Senan, M. F. E. M. (2020). An analysis of the KDD99 and UNSW-NB15 datasets for the intrusion detection system. Symmetry, 12(10), 1666. https://doi.org/10.3390/sym12101666
Bai, K. Z., & Fossaceca, J. M. (2025). EM-AUC: A novel algorithm for evaluating anomaly-based network intrusion detection systems. Sensors, 25(1), 78. https://doi.org/10.3390/s25010078
Bouke, M. A., & Abdullah, A. (2023). An empirical study of pattern leakage impact during data preprocessing on machine learning-based intrusion detection models reliability. Expert Systems with Applications, 230, 120715. https://doi.org/10.1016/j.eswa.2023.120715
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 321-357. https://doi.org/10.1613/jair.953
Kumar, A., Radhakrishnan, R., Sumithra, M., Kaliyaperumal, P., Balusamy, B., & Benedetto, F. (2025). A scalable hybrid autoencoder-extreme learning machine framework for adaptive intrusion detection in high-dimensional networks. Future Internet, 17(5), 221. https://doi.org/10.3390/fi17050221
Moustafa, N., & Slay, J. (2015). UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set). In 2015 Military Communications and Information Systems Conference (MilCIS) (pp. 1-6). IEEE. https://doi.org/10.1109/MilCIS.2015.7348942
Moustafa, N., & Slay, J. (2016). The evaluation of network anomaly detection systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set. Information Security Journal: A Global Perspective, 25(1-3), 18-31. https://doi.org/10.1080/19393555.2015.1125974
Musthafa, M. B., Huda, S., Kodera, Y., & Ali, M. A. (2024). Optimizing IoT intrusion detection using balanced class distribution, feature selection, and ensemble machine learning techniques. Sensors, 24(13), 4293. https://doi.org/10.3390/s24134293
Park, H., Shin, D., Park, C., Jang, J., & Shin, D. (2025). Unsupervised machine learning methods for anomaly detection in network packets. Electronics, 14(14), 2779. https://doi.org/10.3390/electronics14142779
Pinto, A., Herrera, L.-C., Donoso, Y., & Gutierrez, J. A. (2023). Survey on intrusion detection systems based on machine learning techniques for the protection of critical infrastructure. Sensors, 23(5), 2415. https://doi.org/10.3390/s23052415
Rao, Y. N., & Babu, K. S. (2023). An imbalanced generative adversarial network-based approach for network intrusion detection in an imbalanced dataset. Sensors, 23(1), 550. https://doi.org/10.3390/s23010550
Saputra, A., Ramli, K., Nugroho, A. S., Nugraha, I. G. D., & Pranggono, B. (2026). A novel hybrid IGL1 feature selection method for high-performance intrusion detection on the UNSW-NB15 dataset using multiple machine learning models. Big Data and Cognitive Computing, 10(6), Article 182. https://doi.org/10.3390/bdcc10060182
Sayegh, H. R., Dong, W., & Al-madani, A. M. (2024). Enhanced intrusion detection with LSTM-based model, feature selection, and SMOTE for imbalanced data. Applied Sciences, 14(2), 479. https://doi.org/10.3390/app14020479
Shanmugam, V., Razavi-Far, R., & Hallaji, E. (2025). Addressing class imbalance in intrusion detection: A comprehensive evaluation of machine learning approaches. Electronics, 14(1), 69. https://doi.org/10.3390/electronics14010069
Song, Y., Hyun, S., & Cheong, Y.-G. (2021). Analysis of autoencoders for network intrusion detection. Sensors, 21(13), 4294. https://doi.org/10.3390/s21134294
Sun, D., Zhang, L., Jin, K., Ling, J., & Zheng, X. (2023). An intrusion detection method based on hybrid machine learning and neural network in the industrial control field. Applied Sciences, 13(18), 10455. https://doi.org/10.3390/app131810455
Wang, C., Sun, Y., Wang, W., Liu, H., & Wang, B. (2023). Hybrid intrusion detection system based on combination of random forest and autoencoder. Symmetry, 15(3), 568. https://doi.org/10.3390/sym15030568
Zoghi Z. and Serpen G. (2024) 'Building an intrusion detection system on UNSW-NB15: Reducing the margin of error to deal with data overlap and imbalance', Concurrency and Computation: Practice and Experience, 36(25), e8242. https://doi.org/10.1002/cpe.8242
