AI-Driven Auto ML for Analytics Workflow Acceleration on Distributed Data Lakes

Abstract

As organizations generate data from numerous sources, which have progressively become larger in scale ranging from IoT devices to cloud applications the typical centralized data warehouse has been replaced by the distributed data lakes. Such repositories are cloud-based and therefore offer flexible and scalable storage for structured and unstructured data. However, distributed data lakes, besides being a perfect solution for storage and access issues, still bring some adverse effects: the capability to execute the analytical workflows has become more complex, resource-consuming, and hard to control when scaling. Launching and implementing machine learning models in these kinds of places requires, most of the time, a great deal of time, high-level professional skills, and good coordination between different teams. AutoML (Automated Machine Learning) is a newly appearing possibility that aims to solve major problems. It does so by carrying out the laborious stages of feature selection, model training, and hyperparameter tuning—that greatly facilitates the implementation of analytics and makes it less time-consuming but it is not a perfect solution. In order to get the utmost benefit from AutoML in distributed environments, AI-driven orchestration is necessary. With the help of intelligent agents, the pipeline gains real-time decision power about power distribution, routing, and failure recovery; thus, static processes become more dynamic and self-optimized. This paper is a discussion of a combination of AI-driven AutoML frameworks with dispersed data lake architectures to make analytics pipelines easier and faster.