Anomaly Detection in Large-Scale Monitoring Systems using a Language Model

Large-scale monitoring systems encounter significant challenges in detecting anomalies, which can disrupt operations and degrade overall system performance. This study proposes Anomaly Detection in Large-Scale Monitoring Systems using a Language Model (AD-LM), an innovative approach designed to address these challenges by employing a language model specifically tailored for anomaly detection. Through two main steps, AD-LM first utilizes BERTopic for topic modeling, clustering log entries into meaningful topics that uncover patterns indicative of anomalies. Then, the system  employs various classification models, including tree-based, graph-based, and sequence-based approaches, to predict and diagnose failures. Extensive experiments were conducted on three real-world log datasets: Hadoop Distributed File System (HDFS), BlueGene/L (BGL) supercomputer system, and Thunderbird supercomputer system. The model achieved F1 scores of 0.998, 0.999, and 0.999, respectively, across these datasets, demonstrating its capability to significantly improve anomaly detection performance.