Predictive Drift Detection and Adaptive Reconciliation in Multi-Cloud Data Environments
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V3I4P119Keywords:
Multi-Cloud Environments, Data Drift Detection, Predictive Analytics, Adaptive Reconciliation, Schema Evolution, Cloud Data Integration, Anomaly Detection, Data Consistency, Machine Learning, Real-Time Monitoring, Data Governance, Distributed SystemAbstract
In the present day's multi-cloud environments, it has become very hard to keep data consistent & reliable because of the unexpected phenomenon of data drift, which is when the basic structure, quality, or meaning of data changes suddenly across these distant settings. This study offers a way to forecast these kinds of drifts before they have an effect on the important systems. Instead of reacting to many problems after they happen, our solution uses ML models based on the previous information, schema history & change patterns to detect too many problems before they happen. Our adaptive reconciliation engine automatically syncs data sources using resolution methods that take into account the context. It also changes to fit the formats and workloads of the cloud. Companies that use hybrid and multi-cloud architectures need to be able to make decisions, audit, and govern in real time more and more. This dual-layered solution is based on that requirement. Our system brings together cloud-native monitoring tools, statistical drift indicators, and proactive repair methods into a single lifecycle. Early research shows that our system can predict schema and distributional deviations with more than 90% accuracy. The reconciliation layer also cuts down on the need for human input by more than 60%. Some of the most important contributions include the latest predictive model for drift predictions, a reconciliation protocol that can be used on a wide range of cloud platforms, and a single dashboard for operational transparency. The goal of this project is to improve the data governance in these organizations, build trust in the analytical outputs & lower their compliance worries. Our technique is a big step forward in how organizations keep data safe in dynamic multi-cloud systems. Instead of waiting for these kinds of problems to happen and then fixing them, we now anticipate and solve them before they happen.
References
[1] Vishwa, R. (2019). A Resilient Cloud Computing Architecture for Fault-Tolerant Data Processing Using AI-Based Error Recovery. American International Journal of Computer Science and Technology, 1(4), 1-10. https://doi.org/10.63282/3117-5481/AIJCST-V1I4P101
[2] Pedda Muntala, P. S. R., & Jangam, S. K. (2021). Real-time Decision-Making in Fusion ERP Using Streaming Data and AI. International Journal of Emerging Research in Engineering and Technology, 2(2), 55-63. https://doi.org/10.63282/3050-922X.IJERET-V2I2P108
[3] Muppaneni, R. K. (2021). Securing the Enterprise: How Dynamics 365 Meets Global Compliance Standards. International Journal of Emerging Research in Engineering and Technology, 2(1), 133-143. https://doi.org/10.63282/3050-922X.IJERET-V2I1P114
[4] Sharma, A. (2019). A Multi-Layered Framework for Secure Distributed Computing in Heterogeneous Cloud–Edge Environments Using Adaptive AI Orchestration. American International Journal of Computer Science and Technology, 1(3), 1-11. https://doi.org/10.63282/3117-5481/AIJCST-V1I3P101
[5] Suryadevara, S. S. K. (2021). AI-Driven Multi-Cloud Orchestration System for Enterprise Digital Experience Delivery. American International Journal of Computer Science and Technology, 3(1), 21-34. https://doi.org/10.63282/3117-5481/AIJCST-V3I1P103
[6] Suryadevara, S. S. K. (2021). AI-Driven Multi-Cloud Orchestration System for Enterprise Digital Experience Delivery. American International Journal of Computer Science and Technology, 3(1), 21-34. https://doi.org/10.63282/3117-5481/AIJCST-V3I1P103
[7] Gaddam, R. R. (2021). Hermetic ML Environments using Conda-Lock and Docker. American International Journal of Computer Science and Technology, 3(4), 22-34. https://doi.org/10.63282/3117-5481/AIJCST-V3I4P103
[8] Sharma , A. (2019). A Multi-Layered Framework for Secure Distributed Computing in Heterogeneous Cloud–Edge Environments Using Adaptive AI Orchestration. American International Journal of Computer Science and Technology, 1(3), 1-11. https://doi.org/10.63282/3117-5481/AIJCST-V1I3P101
[9] Srigadde, B. R. (2021). Future Methods, Most Underrated Apex Features. American International Journal of Computer Science and Technology, 3(1), 35-45. https://doi.org/10.63282/3117-5481/AIJCST-V3I1P104
[10] Essien, I. A., Nwokocha, G. C., Erigha, E. D., Obuse, E., & Olayiwola, A. (2021). Cybersecurity Risk Modeling in Multi-Cloud Environments: A Quantitative Framework. International Journal of Multidisciplinary Research and Growth Evaluation, 2(5), 551-568.
[11] Muppaneni, R. K. (2021). How Enterprises are Achieving 360° Customer Views with Dynamics 365. International Journal of AI, BigData, Computational and Management Studies, 2(2), 129-138. https://doi.org/10.63282/3050-9416.IJAIBDCMS-V2I2P114
[12] Garrison, J., & Nova, K. (2017). Cloud native infrastructure: patterns for scalable infrastructure and applications in a dynamic environment. " O'Reilly Media, Inc.".
[13] Suryadevara, S. S. K. (2021). Generative AI–Powered Authoring Assistant for Enterprise Content Management. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 2(2), 103-113. https://doi.org/10.63282/3050-9262.IJAIDSML-V2I2P111
[14] Arthur, J. D. (2017). Enhanced Prediction of Network Attacks Using Incomplete Data (Doctoral dissertation, Nova Southeastern University).
[15] Muppaneni, K. (2021). HTTP/3 & REST Latency Improvement. International Journal of Emerging Research in Engineering and Technology, 2(1), 122-132. https://doi.org/10.63282/3050-922X.IJERET-V2I1P113
[16] Parepalli, S. (2017). Evolving enterprise reconciliation: From deterministic validation to AI-supported high-integrity data assurance. Journal of Scientific and Engineering Research, 4(6), 242-252.
[17] Kumar Doodala, A. N. (2021). Intelligent EOB/ERA Generation and Validation Framework on Legacy Systems like Mainframes. International Journal of Emerging Research in Engineering and Technology, 2(1), 111-121. https://doi.org/10.63282/3050-922X.IJERET-V2I1P112
[18] Srigadde, B. R. (2021). When Rounding Up Matters: Working with Decimals in Apex. International Journal of AI, BigData, Computational and Management Studies, 2(1), 122-131. https://doi.org/10.63282/3050-9416.IJAIBDCMS-V2I1P113
[19] Zhang, B., Jin, X., Ratnasamy, S., Wawrzynek, J., & Lee, E. A. (2018, August). Awstream: Adaptive wide-area streaming analytics. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (pp. 236-252).
[20] Vppalapati, M. (2021). The Geometry of Redundancy: Why Physically Separate Storage Paths Behave as One System. International Journal of Emerging Research in Engineering and Technology, 2(2), 107-116. https://doi.org/10.63282/3050-922X.IJERET-V2I2P113
[21] Allenki, S. S., & Korutla, R. (2021). Agile Development in Practice: From Intern to Contributor. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 2(3), 91-103. https://doi.org/10.63282/3050-9262.IJAIDSML-V2I3P110
[22] Ang, H. H., Gopalkrishnan, V., Zliobaite, I., Pechenizkiy, M., & Hoi, S. C. (2012). Predictive handling of asynchronous concept drifts in distributed environments. IEEE Transactions on Knowledge and Data Engineering, 25(10), 2343-2355.
[23] Gaddam, R. R. (2021). Vertex AI as a Unified Control Plane for MLOps. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 2(2), 92-102. https://doi.org/10.63282/3050-9262.IJAIDSML-V2I2P110
[24] Karra, S., Shaw, R., Patwardhan, S. C., & Noronha, S. (2008). Adaptive model predictive control of multivariable time-varying systems. Industrial & engineering chemistry research, 47(8), 2708-2720.
[25] Muppaneni, K. (2021). Cross-Browser Debugging Strategies. American International Journal of Computer Science and Technology, 3(5), 25-36. https://doi.org/10.63282/3117-5481/AIJCST-V3I5P103
[26] Shiramalla, R., & Guntupalli, B. . (2021). Cost-Effective Softphone Integration in CRM Platforms Using RESTful APIs: A Salesforce Case Study for Voice-to-Text Sales Enablement. International Journal of Emerging Trends in Computer Science and Information Technology, 2(1), 101-114. https://doi.org/10.63282/3050-9246.IJETCSIT-V2I1P112
[27] Maisenbacher, M., & Weidlich, M. (2017, June). Handling concept drift in predictive process monitoring. In 2017 IEEE International Conference on Services Computing (SCC) (pp. 1-8). IEEE.
[28] Vppalapati, M., & Talasila, P. K. (2021). Failure without Faults: How Enterprise Storage Systems Degrade Long Before They Break. International Journal of Emerging Trends in Computer Science and Information Technology, 2(1), 115-123. https://doi.org/10.63282/3050-9246.IJETCSIT-V2I1P113
[29] Hu, G., Zhang, Z., Chen, J., Zhang, Z., Armaou, A., & Yan, Z. (2021). Elman neural networks combined with extended Kalman filters for data-driven dynamic data reconciliation in nonlinear dynamic process systems. Industrial & Engineering Chemistry Research, 60(42), 15219-15235.
[30] Parakala, A. (2021). Building Analytics-Driven Bots: RPA Meets Business Intelligence. International Journal of Emerging Research in Engineering and Technology, 2(1), 77-87. https://doi.org/10.63282/3050-922X.IJERET-V2I1P109
[31] Cinar, A., & Turksoy, K. (2018). Fault Detection and Data Reconciliation. In Advances in Artificial Pancreas Systems: Adaptive and Multivariable Predictive Control (pp. 89-95). Cham: Springer International Publishing.
