Streaming Data Pipelines for AI at the Edge: Architecting for Real-Time Intelligence
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V3I2P108Keywords:
Edge Computing, Streaming Data Pipelines, Real-Time Intelligence, AI at the Edge, Low-Latency Processing, IoT Analytics, Edge Inference, Data Architecture, Real-Time Decision-Making, Distributed Systems, Event Stream Processing, Edge AI Deployment, Smart Devices, Edge-to-Cloud Integration, AI Model OptimizationAbstract
In the fast changing technological environment of the present day, the shift from centralized cloud computing to distributed more edge settings is affecting the functioning of AI applications. Edge devices from smart sensors to self-driving cars create enormous volumes of data, so actual time data processing becomes very necessary. The important role of streaming data pipelines in delivering actual time intelligence at the edge helps companies to make more quick & more informed decisions close to the data source. We investigate the growing relevance of edge computing in modern AI systems as well as its function in reducing problems with dependability, bandwidth, and also delay. We provide basic architectural concepts for building scalable, robust pipelines with actual time AI inference and continuous data flow. We underline the key elements of more effective edge AI pipelines: lightweight data processing architectures, interaction with more cloud-native technologies, and guarantee of safe, fault-tolerant operations. From predictive maintenance in industrial IoT to actual time analytics in smart cities, more practical implementations from which to show the clear benefits of these systems showcase This paper serves as a conceptual framework as well as a useful tool for engineers, architects & decision-makers creating the next generation of artificial intelligence systems meant to run at the edge, where context and time are most important
References
[1] Wu, Yulei. "Cloud-edge orchestration for the Internet of Things: Architecture and AI-powered data processing." IEEE Internet of Things Journal 8.16 (2020): 12792-12805.
[2] Boppiniti, Sai Teja. "Real-time data analytics with ai: Leveraging stream processing for dynamic decision support." International Journal of Management Education for Sustainable Development 4.4 (2021).
[3] Liu, Peng, Bozhao Qi, and Suman Banerjee. "Edgeeye: An edge service framework for real-time intelligent video analytics." Proceedings of the 1st international workshop on edge systems, analytics and networking. 2018.
[4] Dautov, Rustem, et al. "Pushing intelligence to the edge with a stream processing architecture." 2017 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData). IEEE, 2017.
[5] Ali Asghar Mehdi Syed, and Shujat Ali. “Evolution of Backup and Disaster Recovery Solutions in Cloud Computing: Trends, Challenges, and Future Directions”. JOURNAL OF RECENT TRENDS IN COMPUTER SCIENCE AND ENGINEERING ( JRTCSE), vol. 9, no. 2, Sept. 2021, pp. 56-71
[6] González, Germán, and Conor L. Evans. "Biomedical Image Processing with Containers and Deep Learning: An Automated Analysis Pipeline: Data architecture, artificial intelligence, automated processing, containerization, and clusters orchestration ease the transition from data acquisition to insights in medium‐to‐large datasets." BioEssays 41.6 (2019): 1900004.
[7] Prado, Miguel De, et al. "Bonseyes ai pipeline bringing ai to you: End-to-end integration of data, algorithms, and deployment tools." ACM Transactions on Internet of Things 1.4 (2020): 1-25.
[8] Yasodhara Varma Rangineeni. “End-to-End MLOps: Automating Model Training, Deployment, and Monitoring”. JOURNAL OF RECENT TRENDS IN COMPUTER SCIENCE AND ENGINEERING ( JRTCSE), vol. 7, no. 2, Sept. 2019, pp. 60-76
[9] Hernandez, Aitor, Bin Xiao, and Valentin Tudor. "Eraia-enabling intelligence data pipelines for iot-based application systems." 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 2020.
[10] Salehe, Mohammad, et al. "Videopipe: Building video stream processing pipelines at the edge." Proceedings of the 20th international middleware conference industrial track. 2019.
[11] Singu, Santosh Kumar. "Designing scalable data engineering pipelines using Azure and Databricks." ESP Journal of Engineering & Technology Advancements 1.2 (2021): 176-187.
[12] Atluri, Anusha. “Data-Driven Decisions in Engineering Firms: Implementing Advanced OTBI and BI Publisher in Oracle HCM”. American Journal of Autonomous Systems and Robotics Engineering, vol. 1, Apr. 2021, pp. 403-25
[13] Pentyala, Dillep Kumar. "Enhancing the Reliability of Data Pipelines in Cloud Infrastructures Through AI-Driven Solutions." The Computertech (2020): 30-49.
[14] Sankaranarayanan, Suresh, et al. "Data flow and distributed deep neural network based low latency IoT-edge computation model for big data environment." Engineering Applications of Artificial Intelligence 94 (2020): 103785.
[15] Ali Asghar Mehdi Syed. “High Availability Storage Systems in Virtualized Environments: Performance Benchmarking of Modern Storage Solutions”. JOURNAL OF RECENT TRENDS IN COMPUTER SCIENCE AND ENGINEERING ( JRTCSE), vol. 9, no. 1, Apr. 2021, pp. 39-55
[16] Devarasetty, Narendra. "Integrating AI and Data Engineering in IoT Ecosystems: Streaming Data Management for Smart Devices." The Computertech (2021): 61-72.
[17] Kupunarapu, Sujith Kumar. "AI-Enhanced Rail Network Optimization: Dynamic Route Planning and Traffic Flow Management." International Journal of Science And Engineering 7.3 (2021): 87-95.
[18] Yang, Chen, et al. "Big data driven edge-cloud collaboration architecture for cloud manufacturing: a software defined perspective." IEEE access 8 (2020): 45938-45950.
[19] Atluri, Anusha. “Leveraging Oracle HCM REST APIs for Real-Time Data Sync in Tech Organizations”. Essex Journal of AI Ethics and Responsible Innovation, vol. 1, Nov. 2021, pp. 226-4
[20] Xu, Dianlei, et al. "Edge intelligence: Architectures, challenges, and applications." arXiv preprint arXiv:2003.12172 (2020).
[21] Castro, Antonio, et al. "How to build a data architecture to drive innovation today and tomorrow." Technology, McKinsey& Company (2020).
