Yahia Mohsen Abu Saqer; Khalil Mohammed Eslayyeh; Nasser Majed Abudalu; Aiman A. Abusamra

ACO-QL: Enhancing ACO Algorithm for Routing in MANETs Using Reinforcement Learning

PDF (757KB), PP.31-45

Views: 0 Downloads: 0

Author(s)

Yahia Mohsen Abu Saqer ^1,* Khalil Mohammed Eslayyeh ¹ Nasser Majed Abudalu ¹ Aiman A. Abusamra ²

1. Islamic University of Gaza/Faculty of IT, Gaza, P850, Palestine

2. Islamic University of Gaza/Faculty of Engineering, Gaza, P850, Palestine

* Corresponding author.

DOI: https://doi.org/10.5815/ijem.2025.05.03

Received: 17 Feb. 2025 / Revised: 14 Mar. 2025 / Accepted: 17 Apr. 2025 / Published: 8 Oct. 2025

Index Terms

MANET, Machine Learning, Reinforcement Learning, Deep Learning, Q-Learning, ACO, Deep Q-Learning

Abstract

ACO-based routing protocols like AntHocNet have emerged as a solution for adaptive routing in MANETS. Likewise, deep Q-learning based protocols are suitable for complex and dynamic environments like MANETs and utilizing real time data for better decision-making. However, there is lack of studies in enhancing ACO-based protocols using Q-learning in a new hybrid protocol, and comparing it with the established ACO-based protocol AntHocNet.
By combining ACO’s strengths (eg. Multi-agent pathfinding and historical data creatd by pheromones) and combine it with key components of Q-learning, then we have a promising protocol ready to be compared with AntHocNet. Previous studies have explored integrating ML with MANET routing, but few of them, if any, have explored enhancing ACO using ML techniques. Therefore, we propose two new protocols: ACO-QL and ACO-DQN.
One uses Q-learning and the latter uses deep Q-learning. After conducting many experiments by running implementations of ACO-DQN, ACO-QL, and AntHocNet on a MANET simulation, we found out that AntHocNet is superior to ACO-DQN in terms of execution time, end-to-end delay, and path cost in most cases, but on the other hand ACO-DQN achieved better packet delivery ratio and throughput results. Meanwhile, ACO-QL consistently achieved lower packet delivery ratios than AntHocNet, and mostly matching AntHocNet’s performance in terms and of other metrics, making it a valid lightweight and faster alternative.

Cite This Paper

Yahia Mohsen Abu Saqer, Khalil Mohammed Eslayyeh, Nasser Majed Abudalu, Aiman A. Abusamra, "ACO-QL: Enhancing ACO Algorithm for Routing in MANETs Using Reinforcement Learning", International Journal of Engineering and Manufacturing (IJEM), Vol.15, No.5, pp. 31-45, 2025. DOI:10.5815/ijem.2025.05.03

Reference

[1]Khan, B. U. I., Olanrewaju, R. F., Anwar, F., Najeeb, A. R., & Yaacob, M. (n.d.). A Survey on MANETs: Architecture, Evolution, Applications, Security Issues and Solutions. Department of Electrical & Computer Engineering, Kulliyyah of Engineering, IIUM Malaysia. Retrieved Dec 28, 2024 from: https://www.researchgate.net/profile/Burhan-Khan-6/publication/327201905_A_Survey_on_MANETs_Architecture_Evolution_Applications_Security_Issues_and_Solutions/links
/5b7fe70292851c1e122ec1a8/A-Survey-on-MANETs-Architecture-Evolution-Applications-Security-Issues-
and-Solutions.pdf?__cf_chl_tk=Pr6L1X3eTs5QlqptyQl55poNAfqOqzkYW5M_Hs3IlCM-1735389667-1.0.1.1-uQAQHkvKs6I.APol5v8.yh3_VZ.Wv6_aWkGZLdH_yRc
[2]Goyal, P., Parmar, V., & Rishi, R. (n.d.). MANET: Vulnerabilities, Challenges, Attacks, Application. Department of Computer Science and Engineering, Technological Institute of Textile and Science, Bhiwani, Haryana, India. Retrieved Feb 1, 2025 from: https://www.researchgate.net/publication/289675441_Manet_Vulnerabilities_challenges_attacks_application
[3]M. Dorigo and L. M. Gambardella, "Ant colony system: a cooperative learning approach to the traveling salesman problem," in IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 53-66, April 1997, doi: 10.1109/4235.585892. Retrieved Dec 28, 2024 from: https://ieeexplore.ieee.org/document/585892/authors
[4]Di Caro, G., Ducatelle, F. and Gambardella, L. M. (2004). AntHocNet: an Ant-Based Hybrid Routing Algorithm for Mobile Ad Hoc Networks. Proceedings of the 8th International Conference on Parallel Problem Solving from Nature (PPSN VIII). Retrieved Jan 10, 2025 from: https://people.idsia.ch/~frederick/anthocnet/anthocnet.html
[5]Uchhula, V., & Bhatt, B. (2010). Comparison of different Ant Colony Based Routing Algorithms. Dharamsinh Desai University, Nadiad, Gujarat, India. Retrieved March 28, 2025 from: https://www.researchgate.net/profile/Brijesh-Bhatt/publication/46122442_Comparison_of_different_Ant_Colony_Based_Routing_Algorithms/links/542bad320cf277d58e8a2281
/Comparison-of-different-Ant-Colony-Based-Routing-Algorithms.pdf?origin=publication_detail&_tp=eyJjb250ZXh0Ijp7ImZpcnN0UGFnZSI6Il9kaXJlY3QiLCJwYWdlIjoicHVibGljYXRpb25Eb3dubG9hZCIsInBy
ZXZpb3VzUGFnZSI6InB1YmxpY2F0aW9uIn19&__cf_chl_tk=0KKuhoOjrqibUl2SCaiisfDutUllODe4HwvlNaxwwSI-1743783981-1.0.1.1-UOwGQ4vwuqXHeY8wy_u3Ru3Gi7GvfC9y8vi3R2cWtAk
[6]Hao, Z.-F., Cai, R.-C., & Huang, H. (n.d.). An Adaptive Parameter Control Strategy for ACO. College of Computer Science and Engineering, South China University of Technology, Guangzhou, P. R. China; College of Mathematical Science, South China University of Technology, Guangzhou, P. R. China. Retrieved Feb 1, 2025 from: https://www.google.co.il/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.researchgate.net/profile/Ruichu-Cai/publication/220776315_A_Novel_ACO_Algorithm_with_Adaptive_Parameter/links/0046352b24ede8caa2000000/A-Novel-ACO-Algorithm-with-Adaptive-Parameter.pdf&ved=2ahUKEwjtxK_c3KKLAxVPcKQEHb_aIaEQFnoECBEQAQ&usg=AOvVaw3le1goNLW0Wkgkqx5OYNUo
[7]S. Kaushik, K. Tripathi, R. Gupta and P. Mahajan, "Futuristic Analysis of Machine Learning Based Routing Protocols in Wireless Ad Hoc Networks," 2021 Fourth International Conference on Computational Intelligence and Communication Technologies (CCICT), Sonepat, India, 2021, pp. 324-329, doi: 10.1109/CCICT53244.2021.00067. Retrieved Nov 29, 2024 from: https://www.researchgate.net/publication/354110638_Futuristic_Analysis_of_Machine_Learning_Based_Routing_Protocols_in_Wireless
_Ad_Hoc_Networks
[8]Arif, Mohammad & Bhargavi, K & Swaroopa, K & Karuturi, Satish & Balamurugan, A. (2024). Machine learning-Based Energy Efficient and Enhancing Communication Reliability for MANETs of Balanced Less Loss Routing Protocol. Journal of Electrical Systems. 20. 777-783. 10.52783/jes.1670. Retrieved Dec 3, 2024 from: https://www.proquest.com/openview/46e5efcf373e81cd1265b606b3c3ab04/1?pq-origsite=gscholar&cbl=4433095
[9]Akhilesh Bijalwan,Iqram Hussain, Kamlesh Chandra Purohit, and M. Anand Kumar. (2023). Enhanced Ant Colony Optimization for Vehicular Ad Hoc Networks Using Fittest Node Clustering. Retrieved Dec 19, 2024 from: https://www.mdpi.com/2071-1050/15/22/15903
[10]Zahid Khan, Sangsha Fang, Anis Koubaa, Pingzhi Fan, Fakhar Abbas, Haleem Farman, (2020) Street-centric routing scheme using ant colony optimization-based clustering for bus-based vehicular ad-hoc network, Computers & Electrical Engineering, Volume 86. Retrieved Dec 19, 2024 from: https://www.sciencedirect.com/science/article/abs/pii/S0045790620305917
[11]Z. Mammeri, "Reinforcement Learning Based Routing in Networks: Review and Classification of Approaches," in IEEE Access, vol. 7, pp. 55916-55950, 2019, doi: 10.1109/ACCESS.2019.2913776. Retrieved Nov 29, 2024 from: https://ieeexplore.ieee.org/abstract/document/8701570
[12]Saeed Kaviani, Bo Ryu, Ejaz Ahmed, Kevin A. Larson, Anh Le, Alex Yahja, Jae H. Kim. (2021). Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for MANETs. Retrieved Dec 19, 2024 from: https://arxiv.org/abs/2101.03273
[13]H. -K. Hsin, E. -J. Chang, K. -Y. Su and A. -Y. Wu, "Ant Colony Optimization-Based Adaptive Network-on-Chip Routing Framework Using Network Information Region," in IEEE Transactions on Computers, vol. 64, no. 8, pp. 2119-2131, 1 Aug. 2015, doi: 10.1109/TC.2014.2366768. Retrieved Dec 19, 2024 from: https://ieeexplore.ieee.org/document/6945251
[14]R. Amin, E. Rojas, A. Aqdus, S. Ramzan, D. Casillas-Perez and J. M. Arco, "A Survey on Machine Learning Techniques for Routing Optimization in SDN," in IEEE Access, vol. 9, pp. 104582-104611, 2021, doi: 10.1109/ACCESS.2021.3099092. Retrieved Nov 29, 2024 from: https://ieeexplore.ieee.org/abstract/document/9493245
[15]Davi Ribeiro Militan, Hermes Pimenta de Moraes, Renata Lopes Rosa, Lunchakorn Wuttisittikulkij, Miguel Arjona Ramírez, and Demóstenes Zegarra Rodríguez. (2021). Enhanced Routing Algorithm Based on Reinforcement Machine Learning—A Case of VoIP Service. Retrieved Nov 29, 2024 from: https://www.mdpi.com/1424-8220/21/2/504
[16]S. Nimmala, M. Ramchander, M. Mahendar, P. Manasa, D. D. Bhavani and K. Raghavendar, "Dynamic RL-ACO: Reinforcement Learning-based Ant Colony Optimization for Load Balancing in Cloud Networks," 2024 5th International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2024, pp. 475-480, doi: 10.1109/ICOSEC61587.2024.10722410. Retrieved Dec 19, 2024 from: https://ieeexplore.ieee.org/abstract/document/10722410
[17]Ghanshyam Prasad Dubey, Shalini Stalin, Omar Alqahtani, Areej Alasiry, Madhu Sharma, Aliya Aleryani, Piyush Kumar Shukla, M. Turki-Hadj Alouane, 2023, “Optimal path selection using reinforcement learning based ant colony optimization algorithm in IoT-Based wireless sensor networks with 5G technology”, Computer Communications, Volume 212. Retrieved Dec 19, 2024 from: https://www.sciencedirect.com/science/article/abs/pii/S0140366423003250
[18]S. Mande, N. Ramachandran, S. Salma Asiya Begum and F. Moreira, "Optimized Reinforcement Learning for Resource Allocation in Vehicular Ad Hoc Networks," in IEEE Access, vol. 12, pp. 167040-167048, 2024, doi: 10.1109/ACCESS.2024.3489395. Retrieved Dec 19, 2024 from: https://ieeexplore.ieee.org/abstract/document/10740275
[19]Johnson, David B., and David A. Maltz. “Dynamic Source Routing in Ad Hoc Wireless Networks.” Computer Science Department, Carnegie Mellon University, (1996). Retrieved Jan 10, 2025 from: https://link.springer.com/chapter/10.1007/978-0-585-29603-6_5#:~:text=This%20paper%20presents%20a%20protocol,which%20hosts%20move%20less%20frequently.
[20]Bentley, J. L. (1975). Multidimensional Binary Search Trees Used for Associative Searching. Communications of the ACM, 18(9), 509–517. https://doi.org/10.1145/361002.361007. Retrieved Jan 10, 2025 from https://dl.acm.org/doi/10.1145/361002.361007
[21]Kurose, J. F., & Ross, K. W. (2017).“Computer Networking: A Top-Down Approach” (7th ed.). Pearson. Retreived Jan 29, 2025 from: https://www.google.co.il/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.ucg.ac.me/skladiste/blog_44233/objava_64433/fajlovi/
Computer%2520Networking%2520_%2520A%2520Top%2520Down%2520Approach,%25207th,%2520converted.pdf&ved=
2ahUKEwjC6ZCG5qKLAxW7TKQEHdE9Gz0QFnoECCUQAQ&usg=
AOvVaw0nYJN2FwHGPeoye6BigjF5
[22]C. E. Perkins and E. M. Royer, "Ad-hoc on-demand distance vector routing," Proceedings WMCSA'99. Second IEEE Workshop on Mobile Computing Systems and Applications, New Orleans, LA, USA, 1999, pp. 90-100, doi: 10.1109/MCSA.1999.749281. Retrieved Mar 26, 2025 from: https://ieeexplore.ieee.org/document/749281

International Journal of Engineering and Manufacturing (IJEM)