A Safety-Constrained Reinforcement Learning Framework for Scheduling with Latency-Tail Guarantees in Industrial URLLC
DOI:
https://doi.org/10.63282/3050-9262.IJAIDSML-V6I3P120Keywords:
5G-Advanced, Industrial URLLC, Safety-Constrained Reinforcement Learning, Latency-Tail Guarantees, Deterministic Scheduling, AI-Driven RAN, Constrained Optimization, Time-Critical Communications, RAN Intelligence, Risk-Sensitive Optimization, Queue-Aware Scheduling, Lyapunov Optimization, Industrial Wireless NetworksAbstract
In industrial wireless networks, Ultra-Reliable Low-Latency Communications (URLLC) require strict end-to-end latency requirements with the reliability level of higher than 99.999% especially in time-based control, robotics and safety systems. Although current schedulers used in 5G-Advanced networks are based on minimizing the average latency or maximizing the throughput, they tend to lack a guarantee on strict latency-tails in bursty traffic and in dynamically changing channel scenario. Deterministic scheduling methods are not adaptable to stochastic variations in networks, and reinforcement learning (RL) based schedulers usually seek long-term available levels, but do not have enforceable limits on safety and tail-risk creating the risk of deadline breaches in industrial challenges that are mission-critical. To overcome these shortcomings, a Safety-Constrained Reinforcement Learning (SCRL) framework to latency-sensitive scheduling on Industrial URLLC settings is proposed in this paper. The scheduling is formulated based on a Constrained Markov Decision Process (CMDP) that has clear latency-tail and reliability constraints. An embedded Lyapunov-based virtual queue mechanism is utilized to guarantee the presence of queue stability and hard deadline guarantee, whereas a risk-sensitive competitive objective, which is grounded on Conditional Value-at-Risk (CVaR), is minimized to reduce extreme lateny events. The suggested framework also uses the concept of queue-aware executing state representations in helping to be responsive in burst traffic conditions. Realistic industrial 5G-Advanced simulations show that the given method yields 99.999% reduction in the over standardized queue stability and reliability topology as compared to deterministic earliest deadline first and classic RL schedulers. The scheme delivers close deterministic performance at the expense of flexibility to offer a scalable, and safety guaranteed scheduling system to AI-controlled Radio Access Networks (RAN) in industrial wireless systems.
References
[1] Zhang, W., Derakhshani, M., Zheng, G., & Lambotharan, S. (2024). Constrained risk-sensitive deep reinforcement learning for eMBB-URLLC joint scheduling. IEEE Transactions on Wireless Communications, 23(9), 10608-10624.
[2] Li, J., & Zhang, X. (2020). Deep reinforcement learning-based joint scheduling of eMBB and URLLC in 5G networks. IEEE Wireless Communications Letters, 9(9), 1543-1546.
[3] Li, Q., Chen, J., Cheffena, M., & Shen, X. (2023). Channel-aware latency tail taming in industrial IoT. IEEE Transactions on Wireless Communications, 22(9), 6107-6123.
[4] Khalifa, N. B., Assaad, M., & Debbah, M. (2019, April). Risk-sensitive reinforcement learning for URLLC traffic in wireless networks. In 2019 IEEE Wireless Communications and Networking Conference (WCNC) (pp. 1-7). IEEE.
[5] Wang, X., Yao, H., Mai, T., Guo, S., & Liu, Y. (2023). Reinforcement learning-based particle swarm optimization for end-to-end traffic scheduling in TSN-5G networks. IEEE/ACM Transactions on Networking, 31(6), 3254-3268.
[6] Destounis, A., Paschos, G. S., Arnau, J., & Kountouris, M. (2018, May). Scheduling URLLC users with reliable latency guarantees. In 2018 16th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt) (pp. 1-8). IEEE.
[7] Upadhyay, D., Soni, M., Gupta, S., Sharma, R., & Venu, N. (2025, March). Latency-Aware Network Slicing for 5G URLLC Applications: Design and Optimization Strategies. In 2025 3rd International Conference on Device Intelligence, Computing and Communication Technologies (DICCT) (pp. 113-118). IEEE.
[8] Praveen, S., Khan, J., & Jacob, L. (2021, July). Reinforcement learning based link adaptation in 5G URLLC. In 2021 8th International Conference on Smart Computing and Communications (ICSCC) (pp. 159-163). IEEE.
[9] Alsenwi, M., Tran, N. H., Bennis, M., Pandey, S. R., Bairagi, A. K., & Hong, C. S. (2021). Intelligent resource slicing for eMBB and URLLC coexistence in 5G and beyond: A deep reinforcement learning based approach. IEEE Transactions on Wireless Communications, 20(7), 4585-4600.
[10] Shi, W., Ganjalizadeh, M., Ghadikolaei, H. S., & Petrova, M. (2023, September). Communication-efficient orchestrations for urllc service via hierarchical reinforcement learning. In 2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) (pp. 1-6). IEEE.
[11] Azari, A., Ozger, M., & Cavdar, C. (2019). Risk-aware resource allocation for URLLC: Challenges and strategies with machine learning. IEEE Communications Magazine, 57(3), 42-48.
[12] Haque, M. E., Tariq, F., Khandaker, M. R., Wong, K. K., & Zhang, Y. (2023). A survey of scheduling in 5G URLLC and outlook for emerging 6G systems. IEEE access, 11, 34372-34396.
[13] Sohaib, R. M., Onireti, O., Sambo, Y., Swash, R., Ansari, S., & Imran, M. A. (2023). Intelligent resource management for eMBB and URLLC in 5G and beyond wireless networks. IEEE access, 11, 65205-65221.
[14] Liang, F., Yu, W., Liu, X., Griffith, D., & Golmie, N. (2021). Toward deep Q-network-based resource allocation in industrial Internet of Things. IEEE internet of things journal, 9(12), 9138-9150.
[15] Shaik, R. B., Nagaradjane, P., Ioannou, I., Sittakul, V., Vasiliou, V., & Pitsillides, A. (2024). AI/ML-aided capacity maximization strategies for URLLC in 5G/6G wireless systems: A survey. Computer Networks, 249, 110506.
[16] Salh, A., Ngah, R., Hussain, G. A., Alhartomi, M., Boubkar, S., Shah, N. S. M., ... & Alzahrani, S. (2024). Bandwidth allocation of URLLC for real-time packet traffic in B5G: A Deep-RL framework. ICT Express, 10(2), 270-276.
[17] Wang, J., Zheng, Y., Wang, J., Shen, Z., Tong, L., Jing, Y., ... & Liao, Y. (2023). Ultra-reliable deep-reinforcement-learning-based intelligent downlink scheduling for 5G new radio-vehicle to infrastructure scenarios. Sensors, 23(20), 8454.
[18] Neely, M. (2010). Stochastic network optimization with application to communication and queueing systems. Morgan & Claypool Publishers.
[19] Al-Saadeh, O., Wikstrom, G., Sachs, J., Thibault, I., & Lister, D. (2018, December). End-to-end latency and reliability performance of 5G in London. In 2018 IEEE Global Communications Conference (GLOBECOM) (pp. 1-6). IEEE.
[20] Dahlman, E., Parkvall, S., & Skold, J. (2023). 5G/5G-advanced: the new generation wireless access technology. Elsevier.
[21] Khoshnevisan, M., Joseph, V., Gupta, P., Meshkati, F., Prakash, R., & Tinnakornsrisuphap, P. (2019). 5G industrial networks with CoMP for URLLC and time sensitive network architecture. IEEE Journal on Selected Areas in Communications, 37(4), 947-959.
[22] Hamidi-Sepehr, F., Sajadieh, M., Panteleev, S., Islam, T., Karls, I., Chatterjee, D., & Ansari, J. (2021). 5G URLLC: Evolution of high-performance wireless networking for industrial automation. IEEE Communications Standards Magazine, 5(2), 132-140.
[23] Khan, B. S., Jangsher, S., Ahmed, A., & Al-Dweik, A. (2022). URLLC and eMBB in 5G industrial IoT: A survey. IEEE Open Journal of the Communications Society, 3, 1134-1163.
[24] Anand, A., De Veciana, G., & Shakkottai, S. (2020). Joint scheduling of URLLC and eMBB traffic in 5G wireless networks. IEEE/ACM Transactions On Networking, 28(2), 477-490.
[25] Yoshizawa, T., Baskaran, S. B. M., & Kunz, A. (2019, October). Overview of 5G URLLC system and security aspects in 3GPP. In 2019 IEEE Conference on Standards for Communications and Networking (CSCN) (pp. 1-5). IEEE.
