International Journal of Wireless and Microwave Technologies (IJWMT)
IJWMT Vol. 16, No. 3, 8 Jun. 2026
Cover page and Table of Contents: PDF (size: 1632KB)
Intelligent Load Balancing Framework for Distributed Big Data Processing System
PDF (1632KB), PP.72-87
Views: 0 Downloads: 0
Author(s)
Ivan Dychka
1
Liubov Oleshchenko
1,*
Dmytro Vovchenko
1
Zhengbing Hu
2
Index Terms
Intelligent Load Balancing, Distributed Software System, Big Data Processing, Adaptive Algorithms, Machine Learning, Resource Allocation, Multi-Node Architecture, Dynamic Weighting
Abstract
This paper proposes an intelligent load balancing framework for distributed big data processing systems that integrates machine learning techniques with adaptive weight-based decision mechanisms. The study addresses limitations of traditional static load balancing methods, which do not account for dynamic workload variations and heterogeneous request characteristics, leading to inefficient resource utilization and bottlenecks in multi-node environments. The proposed approach combines an online learning model for real-time estimation of request complexity with multi-parameter evaluation of node states, including CPU utilization, memory consumption, queue length, response latency, and cache efficiency. A dynamic weighting strategy is used to construct an integrated load indicator for adaptive request distribution across nodes. The framework is deployed within a multi-layer distributed architecture consisting of clustered application servers, distributed databases, caching subsystems, and monitoring components, ensuring scalable and fault-tolerant processing. For evaluation, a three-node simulation environment was used with 10,000 heterogeneous requests, followed by extended testing on semi-realistic workload traces derived from web traffic patterns and database query logs. The dataset included over 1.2 million requests, capturing bursty arrivals, skewed distributions, and heterogeneous complexity. Experimental results show that the proposed method improves load distribution uniformity to 6%, reduces average response time to 210 ms, and increases throughput up to 13,800 requests per second. Statistical validation using confidence intervals and hypothesis testing confirms a 47% (±3.2% at 95% confidence level) reduction in mean response time and throughput improvement up to 14,200 requests per second under realistic workloads.
Cite This Paper
Ivan Dychka, Liubov Oleshchenko, Dmytro Vovchenko, Zhengbing Hu, "Intelligent Load Balancing Framework for Distributed Big Data Processing System", International Journal of Wireless and Microwave Technologies(IJWMT), Vol.16, No.3, pp. 72-87, 2026. DOI:10.5815/ijwmt.2026.03.05
Reference
[1]Y. M. Teo and R. Ayani, “Comparison of load balancing strategies on cluster-based web servers,” Simulation, vol. 77, no. 5–6, pp. 185–195, 2001. https://doi.org/10.1177/003754970107700504
[2]M. R. Belgaum, S. Musa, M. M. Alam, and M. M. Su’ud, “A systematic review of load balancing techniques in software-defined networking,” IEEE Access, vol. 8, pp. 98612–98636, 2020. https://doi.org/10.1109/ACCESS.2020.2995849
[3]Z. Pan and J. Jiangxing, “Load balancing algorithm for web server based on weighted minimal connections,” Journal of Web Systems and Applications, vol. 1, pp. 1–8, 2017. https://doi.org/10.23977/jwsa.2017.11001
[4]P. Jiang, L. Ma, Y. Song, and Y. Huang, “A client proximity based load balance algorithm in web server cluster,” in Proceedings of the 2nd International Conference on Wireless Communication and Network Engineering, 2017, pp. 317–322.
[5]M. Omori and H. Nishi, “Request distribution for heterogeneous database server clusters with processing time estimation,” in IEEE International Conference on Industrial Informatics (INDIN), 2018, pp. 278–283. https://doi.org/10.1109/INDIN.2018.8471931
[6]H. Okano, F. Yamaguchi, K. Takagiwa, and H. Nishi, “Traffic-based flow cache port separate mechanism for network processor,” IEICE Technical Report, vol. 114, no. 18, pp. 69–74, 2014.
[7]H. Mao, M. Alizadeh, I. Menache, and S. Kandula, “Resource management with deep reinforcement learning,” in Proceedings of the ACM HotNets Conference, 2016, pp. 50–56. https://doi.org/10.1145/3005745.3005750
[8]Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo, and J. Zhang, “Edge intelligence: paving the last mile of artificial intelligence with edge computing,” Proceedings of the IEEE, vol. 107, no. 8, pp. 1738–1762, 2021. https://doi.org/10.1109/JPROC.2019.2918951
[9]D. Wu, J. Li, A. Ferini, Y. T. Xu, M. Jenkin, S. Jang, X. Liu, and G. Dudek, “Reinforcement learning for communication load balancing: approaches and challenges,” Frontiers in Computer Science, vol. 5, p. 1156064, 2023. https://doi.org/10.3389/fcomp.2023.1156064
[10]A. R. Khan, “Dynamic load balancing in cloud computing: optimized RL-based clustering with multi-objective optimized task scheduling,” Processes, vol. 12, no. 3, p. 519, 2024. https://doi.org/10.3390/pr12030519
[11]M. E. Esmaeili, A. Khonsari, V. Sohrabi, and A. Dadlani, “Reinforcement learning-based dynamic load balancing in edge computing networks,” Computer Communications, vol. 222, pp. 188–197, 2024. https://doi.org/10.1016/j.comcom.2024.04.009
[12]Y. Zhang et al., “Reinforcement learning-based load balancing for heavy traffic Internet of Things,” Pervasive and Mobile Computing, vol. 99, p. 101891, 2024. https://doi.org/10.1016/j.pmcj.2024.101891
[13]X. Liao, B. Bao, W. Cui, and D. Liu, “Load balancing and topology dynamic adjustment strategy based on deep reinforcement learning,” Frontiers in Energy Research, vol. 11, p. 1342854, 2024. https://doi.org/10.3389/fenrg.2023.1342854
[14]M. Shahakar, S. Mahajan, and L. Patil, “Load balancing in distributed cloud computing using reinforcement learning algorithms,” International Journal on Recent and Innovation Trends in Computing and Communication, vol. 11, no. 2, 2023. https://doi.org/10.17762/ijritcc.v11i2.6130
[15]S. Vollem, “Intelligent load balancing for cloud-native systems: A multi-layered framework integrating predictive analytics and distributed control,” Journal of Artificial Intelligence, Machine Learning & Data Science, vol. 7, no. 2, pp. 3350–3358, 2024. https://doi.org/10.51219/JAIMLD/shekar-vollem/671
[16]H. Kuchuk, Y. Husieva, S. Novoselov, D. Lysytsia, and H. Krykhovetskyi, “Load balancing of the layers IoT fog-cloud support network,” Advanced Information Systems, vol. 9, no. 1, pp. 91–98, 2025. https://doi.org/10.20998/2522-9052.2025.1.11