AI-Powered Anomaly Detection
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V3I2P114Keywords:
Anomaly Detection, Autoencoders, LSTM, Temporal CNN, Transformers, Graph Neural Networks, Federated Learning, MLOpsAbstract
Anomaly detection the identification of rare, consequential deviations in data has been reshaped by AI methods that learn expressive representations from high-volume telemetry. Unified around hybrid pipelines that combine powerful preprocessing and feature stores with a combination of reconstruction models (autoencoders, VAEs), sequence predictors (LSTM/TCN/Transformers), graph neural networks of relational data and powerful classical baselines (Isolation Forest, One-Class SVM, gradient-boosted trees). Such models drive applications in the fields of cybersecurity, payments fraud, IoT/industrial monitoring, and healthcare to force capture of non-linear structure, long-range temporal dependencies as well as cross-entity context. Ongoing issues are the extreme class imbalance, concept drift in non-stationary streams, the lack of labels, or even noisy labels, and the necessity of credible and low-latency explanations. Emerging solutions self-supervised and contrastive pretraining, active learning with human-in-the-loop triage, uncertainty-sensitive scoring, privacy-sensitive transfer and federated learning, and calibrated ensembling make less use of labels and are more robust. Assessment lays emphasis on rare event realism through PR-AUC, early time-to-detect, cost sensitive utility and it is complimented with calibration (isotonic/temperature scaling) to bring scores to operational units. AI-driven detectors will reduce false positives, introduce new failure modes, and speed up the remediation process with MLOps data/version control, drift monitoring, safe retraining, and rollback. This paper summarizes the methodology landscape and provides an operational outline of data readiness to deployment of reliable systems of anomaly detection at scale
References
[1] Gudepu, B. K. (2016). AI-Powered Anomaly Detection Systems for Insider Threat Prevention. The Computertech, 1-9.
[2] Munir, M., Chattha, M. A., Dengel, A., & Ahmed, S. (2019, December). A comparative analysis of traditional and deep learning-based anomaly detection methods for streaming data. In 2019 18th IEEE international conference on machine learning and applications (ICMLA) (pp. 561-566). IEEE.
[3] Xu, X., Liu, H., & Yao, M. (2019). Recent progress of anomaly detection. Complexity, 2019(1), 2686378.
[4] Patcha, A., & Park, J. M. (2007). An overview of anomaly detection techniques: Existing solutions and latest technological trends. Computer networks, 51(12), 3448-3470.
[5] Pang, G., Shen, C., Cao, L., & Hengel, A. V. D. (2021). Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2), 1-38.
[6] Chalapathy, R., & Chawla, S. (2019). Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407.
[7] Zhang, W., Yang, Q., & Geng, Y. (2009, January). A survey of anomaly detection methods in networks. In 2009 International Symposium on Computer Network and Multimedia Technology (pp. 1-3). IEEE.
[8] Oprea, S. V., Bâra, A., Puican, F. C., & Radu, I. C. (2021). Anomaly detection with machine learning algorithms and big data in electricity consumption. Sustainability, 13(19), 10963.
[9] Choi, K., Yi, J., Park, C., & Yoon, S. (2021). Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEEE access, 9, 120043-120065.
[10] Demertzis, K., Iliadis, L., Tziritas, N., & Kikiras, P. (2020). Anomaly detection via blockchained deep learning smart contracts in industry 4.0. Neural Computing and Applications, 32(23), 17361-17378.
[11] Ramchandran, A., & Sangaiah, A. K. (2018). Unsupervised anomaly detection for high dimensional data—An exploratory analysis. In Computational intelligence for multimedia big data on the cloud with engineering applications (pp. 233-251). Academic Press.
[12] Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015). Deep learning applications and challenges in big data analytics. Journal of big data, 2(1), 1.
[13] Luengo, J., García-Gil, D., Ramírez-Gallego, S., García, S., & Herrera, F. (2020). Big data preprocessing. Cham: Springer, 1, 1-186.
[14] Castrillon, J., Lieber, M., Klüppelholz, S., Völp, M., Asmussen, N., Assmann, U., ... & Wunderlich, S. (2017). A hardware/software stack for heterogeneous systems. IEEE Transactions on Multi-Scale Computing Systems, 4(3), 243-259.
[15] Huet, A., Navarro, J. M., & Rossi, D. (2022, August). Local evaluation of time series anomaly detection algorithms. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 635-645).
[16] Probst, P., Wright, M. N., & Boulesteix, A. L. (2019). Hyperparameters and tuning strategies for random forest. Wiley Interdisciplinary Reviews: data mining and knowledge discovery, 9(3), e1301.
[17] Ghosh, A., & Kole, A. L. O. K. (2021). A comparative study of enhanced machine learning algorithms for brain tumor detection and classification. TechRxiv. Preprint.
[18] Falcão, F., Zoppi, T., Silva, C. B. V., Santos, A., Fonseca, B., Ceccarelli, A., & Bondavalli, A. (2019, April). Quantitative comparison of unsupervised anomaly detection algorithms for intrusion detection. In Proceedings of the 34th ACM/SIGAPP symposium on applied computing (pp. 318-327).
[19] Bridges, R. A., Collins, J., Ferragut, E. M., Laska, J., & Sullivan, B. D. (2016). A multi-level anomaly detection algorithm for time-varying graph data with interactive visualization. Social Network Analysis and Mining, 6(1), 99.
[20] Gaddam, A., Wilkin, T., Angelova, M., & Gaddam, J. (2020). Detecting sensor faults, anomalies and outliers in the internet of things: A survey on the challenges and solutions. Electronics, 9(3), 511.
