Evolution of Data Processing and Management: A Comparative Analysis of Traditional and Modern Big Data Architectures
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V3I1P103Keywords:
Edge computing, Quantum computing, Federated learning, Explainable AI, Big data architectures, Data privacy, Scalability, Real-time processing, Machine learning, Ethical considerationsAbstract
The rapid advancement in technology and the exponential growth in data generation have necessitated the evolution of data processing and management systems. This paper provides a comprehensive comparative analysis of traditional and modern big data architectures, highlighting the key differences, advantages, and limitations of each. We delve into the historical context, the technological advancements, and the current trends in data processing and management. The paper also includes a detailed examination of the algorithms and methodologies used in both traditional and modern architectures, supported by empirical data and case studies. Finally, we discuss the future directions and potential research areas in the field of big data
References
[1] Chen, M., Mao, S., & Liu, Y. (2014). Big data: A survey. Mobile Networks and Applications, 19(2), 171-209.
[2] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107-113.
[3] Zaharia, M., et al. (2010). Spark: Cluster computing with working sets. HotCloud'10 Proceedings of the 2nd USENIX conference on Hot topics in cloud computing.
[4] Armbrust, M., et al. (2015). Spark SQL: Relational data processing in Spark. Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data.
[5] Kafka, J., et al. (2011). Apache Kafka: A high-throughput distributed messaging system. Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data.
[6] Abadi, M., et al. (2016). TensorFlow: A system for large-scale machine learning. Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems.
[7] Chen, J., et al. (2018). Data lake: A new paradigm for big data management. Proceedings of the VLDB Endowment, 11(11), 1682-1695.
[8] Gartner. (2020). Gartner's 2020 Hype Cycle for Data Management. Gartner Research.
[9] IBM. (2021). Big Data and Analytics: A Guide for Business Leaders. IBM White Paper.
[10] Microsoft. (2021). Azure Data Lake: A Comprehensive Guide. Microsoft Documentation.
[11] https://www.ewsolutions.com/evolution-of-data-and-data-management/
[12] https://www.atlantis-press.com/article/25858756.pdf
[13] https://ypoint.net/evolution-of-data-management/
[15] https://www.sparkfish.com/the-evolution-of-data-management/
[16] https://itchronicles.com/big-data/the-evolution-of-big-data-solutions/
[17] https://www.linkedin.com/pulse/evolution-data-processing-journey-through-time-abhishant-gautam-n5owc
[18] https://www.purestorage.com/knowledge/big-data/big-data-vs-traditional-data.html
