Unsupervised log message anomaly detection
Date
2020
Authors
Farzad, Amir
Gulliver, Thomas Aaron
Journal Title
Journal ISSN
Volume Title
Publisher
ICT Express
Abstract
Log messages are now broadly used in cloud and software systems. They are important for classification and anomaly detection as millions of logs are generated each day. In this paper, an unsupervised model for log message anomaly detection is proposed which employs Isolation Forest and two deep Autoencoder networks. The Autoencoder networks are used for training and feature extraction, and then for anomaly detection, while Isolation Forest is used for positive sample prediction. The proposed model is evaluated using the BGL, Openstack and Thunderbird log message data sets. The results obtained show that the number of negative samples predicted to be positive is low, especially with Isolation Forest and one Autoencoder. Further, the results are better than with other well-known models.
Description
Keywords
Anomaly detection, Classification, Deep learning, Log messages, Unsupervised learning
Citation
Farzad, A., & Gulliver, T. A. (2020). Unsupervised log message anomaly detection. ICT Express, 6(3), 229-237. https://doi.org/10.1016/j.icte.2020.06.003.