Causal Graph-Based Root Cause Analysis for Distributed Data Integrity Incidents: A Framework for Cross-System Failure Attribution and Resolution
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V5I4P128Keywords:
Causal Graphs, Root Cause Analysis, Data Integrity, Distributed Systems, Microservices, Causal Discovery, Observability, Failure Attribution, Data Lineage, AIopsAbstract
Distributed data platforms increasingly rely on microservices, event streams, replicated storage, automated deployment pipelines, and heterogeneous integration layers. These environments improve scalability and organizational agility, but they also create a difficult class of incidents: cross-system data integrity failures whose visible symptoms occur far away from their originating cause. A duplicate financial transaction, stale customer profile, corrupted analytical feature, or conflicting replicated record may appear in one system while its causal antecedents lie in delayed message propagation, schema drift, compensating transaction failure, weak consistency, deployment misconfiguration, or an upstream machine-learning service. Conventional monitoring and root cause analysis techniques are often optimized for latency, availability, or infrastructure symptoms, and therefore they struggle to distinguish causal faults from correlated anomalies. This paper proposes CG-RCA-DI, a causal graph-based root cause analysis framework for distributed data integrity incidents. The framework combines system topology, event lineage, observability telemetry, data quality constraints, deployment metadata, and domain policies into a multilayer causal graph. It then applies causal discovery, temporal precedence, counterfactual screening, and graph-based attribution to identify likely root causes and recommend resolution actions. The contribution is a complete reference architecture, an incident model, a causal scoring method, and an evaluation protocol for precision, attribution latency, resolution quality, and auditability. The paper argues that data integrity RCA must move beyond service-level symptom localization toward evidence-grounded causal attribution across application, data, process, and governance layers.
References
[1] J. Pearl, Causality: Models, Reasoning, and Inference, 2nd ed. Cambridge, U.K.: Cambridge University Press, 2009.
[2] Gunda SK, Yettapu SDR, Bodakunti S, Bikki SB. Decision Intelligence Methodology for AI-Driven Agile Software Lifecycle Governance and Architecture-Centered Project Management, 2023 Mar. 30;4(1):102-8. https://doi.org/10.63282/3050-9262.IJAIDSML-V4I1P112
[3] L. Lamport, “Time, clocks, and the ordering of events in a distributed system,” Communications of the ACM, vol. 21, no. 7, pp. 558–565, 1978. https://doi.org/10.1145/359545.359563
[4] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels, “Dynamo: Amazon’s highly available key-value store,” in Proceedings of the 21st ACM Symposium on Operating Systems Principles (SOSP ’07), 2007, pp. 205–220. https://doi.org/10.1145/1323293.1294281
[5] Gunda, S. K. G. (2023). The Future of Software Development and the Expanding Role of ML Models. International Journal of Emerging Research in Engineering and Technology, 4(2), 126-129. https://doi.org/10.63282/3050-922X.IJERET-V4I2P113
[6] P. Chen, Y. Qi, P. Zheng, and D. Hou, “CauseInfer: Automatic and distributed performance diagnosis with hierarchical causality graph in large distributed systems,” in IEEE INFOCOM 2014—IEEE Conference on Computer Communications, 2014, pp. 1887–1895. https://doi.org/10.1109/INFOCOM.2014.6848128
[7] S. Gilbert and N. Lynch, “Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services,” ACM SIGACT News, vol. 33, no. 2, pp. 51–59, 2002. https://doi.org/10.1145/564585.564601
[8] Sivva, S. D., Thalakanti, R. R., Bandari, S. S. G., & Yettapu, S. D. R. (2023). AI-Driven Decision Intelligence for Agile Software Lifecycle Governance: An Architecture-Centered Framework Integrating Machine Learning Defect Prediction and Automated Testing. International Journal of Emerging Trends in Computer Science and Information Technology, 4(4), 167-172. https://doi.org/10.63282/3050-9246.IJETCSIT-V4I4P118
[9] M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski, “Conflict-free replicated data types,” in Stabilization, Safety, and Security of Distributed Systems, Lecture Notes in Computer Science, vol. 6976, 2011, pp. 386–400. https://doi.org/10.1007/978-3-642-24550-3_29
[10] M. A. Ikram, S. Chakraborty, S. Mitra, S. Bagchi, and M. K. Reiter, “Root cause analysis of failures in microservices through causal discovery,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 31158–31170.
[11] Balerao, M. (2023). A converged artificial intelligence architecture for innovation, software lifecycle optimization, and cybersecurity risk mitigation. International Journal of Multidisciplinary Futuristic Development, 4(1), 117–120. https://doi.org/10.54660/IJMFD.2023.4.1.117-120
[12] W. Lloyd, M. J. Freedman, M. Kaminsky, and D. G. Andersen, “Don’t settle for eventual: Scalable causal consistency for wide-area storage with COPS,” in Proceedings of the 23rd ACM Symposium on Operating Systems Principles (SOSP ’11), 2011, pp. 401–416. https://doi.org/10.1145/2043556.2043593
[13] B. Li, X. Peng, Q. Xiang, H. Wang, T. Xie, J. Sun, and X. Liu, “Enjoy your observability: An industrial survey of microservice tracing and analysis,” Empirical Software Engineering, vol. 27, no. 1, article 25, 2022. https://doi.org/10.1007/s10664-021-10063-9
[14] S. K. Gunda, “Analyzing Machine Learning Techniques for Software Defect Prediction: A Comprehensive Performance Comparison,” 2024 Asian Conference on Intelligent Technologies (ACOIT), KOLAR, India, 2024, pp. 1-5, https://doi.org/10.1109/ACOIT62457.2024.10939610.
[15] P. Spirtes, C. Glymour, and R. Scheines, Causation, Prediction, and Search, 2nd ed. Cambridge, MA, USA: MIT Press, 2000.
[16] P. Wang, J. Xu, M. Ma, W. Lin, D. Pan, Y. Wang, and P. Chen, “CloudRanger: Root cause identification for cloud native systems,” in Proceedings of the 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), 2018, pp. 492–502. https://doi.org/10.1109/CCGRID.2018.00076
[17] D. B. Terry, A. J. Demers, K. Petersen, M. J. Spreitzer, M. M. Theimer, and B. B. Welch, “Session guarantees for weakly consistent replicated data,” in Proceedings of the Third International Conference on Parallel and Distributed Information Systems, 1994, pp. 140–149. https://doi.org/10.1109/PDIS.1994.331722
[18] Sivva, S. D. (2023). An end-to-end AI-based systems engineering paradigm for lifecycle governance, predictive quality assurance, automation economics, and cybersecurity intelligence. Journal of Frontiers in Multidisciplinary Research, 4(1), 600–604. https://doi.org/10.54660/.JFMR.2023.4.1.600-604
[19] P. Bailis, A. Davidson, A. Fekete, A. Ghodsi, J. M. Hellerstein, and I. Stoica, “Highly available transactions: Virtues and limitations,” Proceedings of the VLDB Endowment, vol. 7, no. 3, pp. 181–192, 2013. https://doi.org/10.14778/2732232.2732237
[20] B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag, “Dapper, a large-scale distributed systems tracing infrastructure,” Google Technical Report, 2010.
[21] J. Peters, D. Janzing, and B. Schölkopf, Elements of Causal Inference: Foundations and Learning Algorithms. Cambridge, MA, USA: MIT Press, 2017.
[22] G. Yu, P. Chen, H. Chen, Z. Guan, Z. Huang, L. Jing, T. Weng, X. Sun, and X. Li, “MicroRank: End-to-end latency issue localization with extended spectrum analysis in microservice environments,” in Proceedings of The Web Conference 2021, 2021, pp. 3087–3098. https://doi.org/10.1145/3442381.3449905
[23] Z. Zhao, T. Zhang, Z. Shen, H. Dong, X. Ma, X. Liu, and Y. Yang, “CHASE: A causal heterogeneous graph based framework for root cause analysis in multimodal microservice systems,” arXiv:2406.19711, 2024.
[24] N. Dragoni, S. Giallorenzo, A. L. Lafuente, M. Mazzara, F. Montesi, R. Mustafin, and L. Safina, “Microservices: Yesterday, today, and tomorrow,” in Present and Ulterior Software Engineering, M. Mazzara and B. Meyer, Eds. Cham, Switzerland: Springer, 2017, pp. 195–216. https://doi.org/10.1007/978-3-319-67425-4_12
[25] C.-M. Lin, C. Chang, W.-Y. Wang, K.-D. Wang, and W.-C. Peng, “Root cause analysis in microservice using neural Granger causal discovery,” arXiv: 2402.01140, 2024.
