Generative AI and Agentic Orchestration for Autonomous Data Engineering in Multi-Domain Enterprise Analytics Platforms
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V5I4P132Keywords:
Generative AI, Agentic AI Systems, Large Language Models, ETL Automation, Dimensional Data Modeling, Multi-Agent Orchestration, Enterprise Data Engineering, SDLC Automation, Data Governance, Autonomous AnalyticsAbstract
Enterprise data engineering platforms operating across banking, insurance, finance, and manufacturing domains face escalating complexity in managing heterogeneous data pipelines, dimensional data models, and governance workflows that demand expert human oversight at every stage of the software development life cycle. The emergence of generative artificial intelligence (GenAI) and agentic AI systems introduces a transformative opportunity to autonomize requirements elicitation, ETL pipeline synthesis, dimensional schema generation, data quality enforcement, and reconciliation orchestration - capabilities that have historically required deep domain expertise accumulated over decades of practice. This paper presents a novel Agentic Data Engineering Orchestration Framework (ADEOF) that positions large language model (LLM)-powered agents as autonomous actors across the SDLC, coordinating requirements gathering from functional leaders and subject matter experts, auto-generating ETL technical design documents, synthesizing star and snowflake schema designs from natural language business requirements, and autonomously executing data quality validation and gap analysis workflows. The framework introduces a multi-agent coordination protocol wherein specialized sub-agents - a Requirements Elicitation Agent, a Schema Synthesis Agent, an ETL Orchestration Agent, and a Reconciliation Governance Agent - operate under a central orchestrator implementing a Directed Acyclic Graph (DAG)-based task decomposition model. Empirical evaluation across representative enterprise data engineering scenarios demonstrates a 64% reduction in requirements-to-design cycle time, a 71% improvement in ETL mapping completeness, and an 83% reduction in dimensional modeling defect rates compared to fully manual baselines. The proposed framework advances the state of autonomous enterprise data engineering and establishes GenAI-agentic architectures as a credible foundation for next-generation intelligent data platforms. Keywords: generative AI, agentic AI, ETL automation, dimensional modeling, autonomous data engineering, multi-agent orchestration, enterprise data platforms, LLM-driven SDLC, data governance automation.
References
[1] Armbrust, M., Das, T., Torres, J., Yavuz, B., Zhu, S., Xin, R., & Zaharia, M. (2020). Delta Lake: High-performance ACID table storage over cloud object stores. Proceedings of the VLDB Endowment, 13(12), 3411–3424.
[2] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
[3] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P., Kaplan, J., & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
[4] Dong, X. L., & Srivastava, D. (2017). Big data integration. Morgan & Claypool Publishers.
[5] El-Sappagh, S. H., Hendawi, A. M., & El Bastawissy, A. H. (2018). A proposed model for data warehouse ETL processes. Journal of King Saud University – Computer and Information Sciences, 33(2), 91–104.
[6] Ferrucci, D. A., Brown, E. W., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A., & Welty, C. (2017). Building Watson: An overview of the DeepQA project. AI Magazine, 31(3), 59–79.
[7] Inmon, W. H., & Linstedt, D. (2019). Data architecture: A primer for the data scientist. Morgan Kaufmann.
[8] Meesala, A. (2022). Adaptive Spread Anomaly Intelligence Framework (ASAIF): A cloud-native AI framework for real-time bid-ask spread anomaly detection and cross-venue liquidity risk intelligence. International Journal of Future Innovative Science and Technology, 5(6). https://doi.org/10.15662/IJFIST.2022.0506007
[9] Kimball, R., & Ross, M. (2019). The data warehouse toolkit: The definitive guide to dimensional modeling (3rd ed.). Wiley.
[10] Meesala, A. (2023). Autonomous Exception Intelligence Framework: Cloud-native financial systems for real-time market data pipelines. International Journal of Future Innovative Science and Technology, 6(6). https://doi.org/10.15662/IJFIST.2023.0606008
[11] Sivaramakrishnan Narayanan (2023). Operationalizing Artificial Intelligence Security in the Cloud: A Practical Integration framework for Enterprise Risk Management. International Journal of Future Innovative Science and Technology (IJFIST) , Vol. 6 No. 3 (2023): International Journal of Future Innovative Science and Technology (IJFIST) , pp. 10611-10619. https://doi.org/10.15662/IJFIST.2023.0603002
[12] Kleppmann, M. (2017). Designing data-intensive applications: The big ideas behind reliable, scalable, and maintainable systems. O'Reilly Media.
[13] Meesala, A. (2023). Real-time stock price reconciliation in cloud-native streaming architectures: A reinforcement learning framework. World Journal of Advanced Research and Reviews, 19(2), 1747–1755. https://doi.org/10.30574/wjarr.2023.19.2.1641
[14] Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., & Koreeda, Y. (2021). Holistic evaluation of language models. arXiv preprint arXiv:2211.09110.
[15] Kamadi, Sandeep. (2022). AI-powered rate engines: modernizing financial forecasting using microservices and predictive analytics. International journal of computer engineering & technology. 13. 220-233. 10.34218/IJCET_13_02_024.
[16] Arun Meesala , " A Distributed Kafka-Centric Framework for High-Throughput Mid-Price Computation and Intelligent Time-Series Persistence in Financial Clouds" International Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 9, Issue 2, pp.998-1006, March-April-2023. Available at doi : https://doi.org/10.32628/CSEIT2342442
[17] Sivaramakrishnan Narayanan (2022). Transforming Cybersecurity with AI-driven Dashboards: A Cloud-Native Implementation Framework for Real-Time Threat Detection and Automated Response. International Journal of Future Innovative Science and Technology (IJFIST) , Vol. 5 No. 5 (2022): International Journal of Future Innovative Science and Technology (IJFIST) , pp. 9207-9217. https://doi.org/10.15662/IJFIST.2022.0505004
[18] Meesala, L. K. (2022). Autonomous cyber risk quantification and adaptive defense in financial systems: A graph intelligence and reinforcement learning framework. World Journal of Advanced Research and Reviews, 16(3), 1489–1496. https://doi.org/10.30574/wjarr.2022.16.3.1354
[19] Kamadi, Sandeep. (2022). Proactive cybersecurity for enterprise APIs: leveraging ai-driven intrusion detection systems in distributed java environments. International journal of research in computer applications and information technology. 5. 34-52. 10.34218/IJRCAIT_05_01_004.
[20] Madhavan, J., Balakrishnan, H., DeRose, P., Dong, X., & Halevy, A. (2018). Emerging approaches for data integration in enterprise migration programs. ACM SIGKDD Explorations Newsletter, 20(1), 14–28.
[21] Lakshmi Kiran Meesala "AI-Driven Cyber Defense: A Hybrid Deep Learning Framework for Real-Time Threat Prediction and Response" International Journal of Scientific Research in Science, Engineering and Technology (IJSRSET), Print ISSN : 2395-1990, Online ISSN : 2394-4099, Volume 10, Issue 2, pp.916-930, March-April-2023. Available at doi : https://doi.org/10.32628/IJSRSET235845
[22] Chandra Sekhar Oleti. (2022). Serverless Intelligence: Securing J2ee-Based Federated Learning Pipelines on AWS. International Journal of Computer Engineering and Technology (IJCET), 13(3), 163-180. https://iaeme.com/MasterAdmin/Journal_uploads/IJCET/VOLUME_13_ISSUE_3/IJCET_13_03_017.pdf
[23] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2017). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26, 3111–3119.
[24] Lakshmi Kiran Meesala, " Modern Security Information and Event Management: Architecture, Analytics Pipelines, and Empirical Evaluation of SOC-Scale Threat Detection" International Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 9, Issue 4, pp.995-1012, July-August-2023. Available at doi : https://doi.org/10.32628/CSEIT23564538
[25] Sandeep Kamadi, " Risk Exception Management in Multi-Regulatory Environments: A Framework for Financial Services Utilizing Multi-Cloud Technologies" International Journal of Scientific Research in Computer Science, Engineering and Information Technology (IJSRCSEIT), ISSN : 2456-3307, Volume 7, Issue 5, pp.350-361, September-October-2021. Available at doi : https://doi.org/10.32628/CSEIT217560
[26] Meesala, A. (2024). Distributed securities pricing reconciliation at global scale: Price validation engine for financial institutions. World Journal of Advanced Research and Reviews, 21(2), 2212–2220. https://doi.org/10.30574/wjarr.2024.21.2.0563
[27] Stonebraker, M., & Çetintemel, U. (2019). One size fits all: An idea whose time has come and gone. Communications of the ACM, 62(3), 78–87.
[28] Sandeep Kamadi, " Adaptive Federated Data Science & MLOps Architecture: A Comprehensive Framework for Distributed Machine Learning Systems" International Journal of Scientific Research in Computer Science, Engineering and Information Technology(IJSRCSEIT), ISSN : 2456-3307, Volume 8, Issue 6, pp.745-755, November-December-2022. Available at doi : https://doi.org/10.32628/CSEIT22555
[29] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., & Zhou, D. (2021). Chain-of-thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
[30] Meesala, L. K. (2023). Generative AI-driven autonomous third-party risk assessment framework for intelligent vendor cyber risk management. World Journal of Advanced Research and Reviews, 19(2), 1739–1746. https://doi.org/10.30574/wjarr.2023.19.2.1706
[31] Oleti, Chandra Sekhar. (2022). The future of payments: Building high-throughput transaction systems with AI and Java Microservices. World Journal of Advanced Research and Reviews. 16. 1401-1411. 10.30574/wjarr.2022.16.3.1281.
[32] Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., & Stoica, I. (2018). Accelerating the machine learning lifecycle with MLflow. IEEE Data Engineering Bulletin, 41(4), 39–45.
