Unsupervised log message anomaly detection

Date

2020

Authors

Farzad, Amir
Gulliver, T. Aaron

Journal Title

Journal ISSN

Volume Title

Publisher

ICT Express

Abstract

Log messages are now broadly used in cloud and software systems. They are important for classification and anomaly detection as millions of logs are generated each day. In this paper, an unsupervised model for log message anomaly detection is proposed which employs Isolation Forest and two deep Autoencoder networks. The Autoencoder networks are used for training and feature extraction, and then for anomaly detection, while Isolation Forest is used for positive sample prediction. The proposed model is evaluated using the BGL, Openstack and Thunderbird log message data sets. The results obtained show that the number of negative samples predicted to be positive is low, especially with Isolation Forest and one Autoencoder. Further, the results are better than with other well-known models.

Description

Keywords

Anomaly detection, Classification, Deep learning, Log messages, Unsupervised learning

Citation

Farzad, A., & Gulliver, T. A. (2020). Unsupervised log message anomaly detection. ICT Express, 6(3), 229-237. https://doi.org/10.1016/j.icte.2020.06.003.