International Journal of Computer Network and Information Security (IJCNIS)
IJCNIS Vol. 18, No. 5, 8 Oct. 2026
Cover page and Table of Contents: PDF (size: 1636KB)
A Multi-Modal Deep Learning Framework for Robust Detection of Obfuscated API Calls in Malware Analysis
PDF (1636KB), PP.119-142
Views: 0 Downloads: 0
Author(s)
Index Terms
Malware Detection, API Obfuscation, Multi-Modal Deep Learning, Graph Attention Networks (GATs), Behavioural Malware Analysis, API Call Graph Analysis, Contrastive Learning.
Abstract
Malware developers employ advanced API obfuscation techniques, such as name randomization, dynamic resolution, call stack manipulation, parameter spoofing, and API chaining, to bypass detection. Existing unimodal analysis systems frequently fail to identify these threats due to their limited scope. To address this, we introduce a novel multi-modal deep learning framework that combines temporal, structural, and parametric analysis for malware detection. This multi‑modal method uses a temporal encoder with self‑attention to detect name obfuscation and API chaining, Graph attention networks analyze call graphs for call stack tampering and dynamic API resolution, while a contrastive learning module identifies anomalous parameter usage. A key novel approach is the dynamic fusion mechanism, which uses attention-based weighting to combine features, enhancing accuracy and interpretability. Additionally, adversarial training ensures robustness against evasion attempts, with theoretical guarantees on performance under variations. Evaluated on 29,505 real‑world malware samples, the proposed multi‑modal framework achieves a 94.2% F1‑score (an 18% improvement over unimodal baselines) and a 98.1% AUC‑ROC. The framework notably maintains 82% robustness against adversarial variations, significantly outperforming conventional LSTM‑based approaches (45%). Beyond detection, the proposed method provides explainable attention maps for forensic analysis and low latency (<1ms/sample), making it suitable for real-time security deployment. These results suggest that multi-modal fusion is critical for next-generation endpoint protection.
Cite This Paper
Nayankumar M. Mali, Narendrasinh C. Chauhan, "A Multi-Modal Deep Learning Framework for Robust Detection of Obfuscated API Calls in Malware Analysis", International Journal of Computer Network and Information Security(IJCNIS), Vol.18, No.5, pp. 119-142, 2026. DOI:10.5815/ijcnis.2026.05.08
Reference
[1]J. Singh and J. Singh, “Challenge of Malware Analysis: Malware obfuscation Techniques,” International Journal of Information Security Science, vol. 7, no. 3, pp. 100–110, Sept. 2018.
[2]Asghar, H.J., Zhao, B.Z.H., Ikram, M. et al. Use of cryptography in malware obfuscation. J Comput Virol Hack Tech 20, 135–152 (2024). https://doi.org/10.1007/s11416-023-00504-y
[3]Li, Yang, Fei Kang, Hui Shu, Xiaobing Xiong, Yuntian Zhao, and Rongbo Sun. 2023. "APIASO: A Novel API Call Obfuscation Technique Based on Address Space Obscurity" Applied Sciences 13, no. 16: 9056. https://doi.org/10.3390/app13169056
[4]Z. Chen, E. Brophy, and T. E. Ward, "Malware Classification Using Static Disassembly and Machine Learning," TechRxiv, preprint, Dec. 2021, doi: 10.36227/techrxiv.17259806.v1. Available: arXiv:2201.07649.
[5]V. E. Kotov and M. Wojnowicz, "Towards Generic Deobfuscation of Windows API Calls," arXiv preprint arXiv:1802.04466, Feb. 2018. doi: 10.48550/arXiv.1802.04466. Available: https://arxiv.org/abs/1802.04466
[6]A. A. Alhashmi et al., “Hybrid Malware Variant Detection Model with Extreme Gradient Boosting and Artificial Neural Network Classifiers,” Comput. Mater. Contin., vol. 76, no. 3, pp. 3483–3498, 2023. https://doi.org/10.32604/cmc.2023.041038
[7]Brezinski, Kenneth, Ferens, Ken, Metamorphic Malware and Obfuscation: A Survey of Techniques, Variants, and Generation Kits, Security and Communication Networks, 2023, 8227751, 41 pages, 2023. https://doi.org/10.1155/2023/8227751
[8]X. Wei, C. Li, Q. Lv, N. Li, D. Sun, and Y. Wang, "Mitigating the Impact of Malware Evolution on API Sequence-based Windows Malware Detector," arXiv preprint arXiv:2408.01661, Aug. 2024, revised Nov. 2025. doi: 10.48550/arXiv.2408.01661.
[9]E. Amer, S. El-Sappagh, and J. W. Hu, "Contextual Identification of Windows Malware through Semantic Interpretation of API Call Sequence," Applied Sciences, vol. 10, no. 21, Art. no. 7673, 2020, doi: 10.3390/app10217673.
[10]Song, Y., Zhang, D., Wang, J. et al. Application of deep learning in malware detection: a review. J Big Data 12, 99 (2025). https://doi.org/10.1186/s40537-025-01157-y
[11]P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, "API-MalDetect: Automated malware detection framework for windows based on API calls and deep learning techniques," Journal of Network and Computer Applications, vol. 218, Art. no. 103704, 2023, doi: 10.1016/j.jnca.2023.103704.
[12]A. Afianian, S. Niksefat, B. Sadeghiyan, and D. Baptiste, "Malware Dynamic Analysis Evasion Techniques: A Survey," ACM Computing Surveys, vol. 52, no. 6, Art. no. 126, Nov. 2019, doi: 10.1145/3365001.
[13]M. F. Zolkipli and A. Jantan, "A Framework for Defining Malware Behavior Using Run Time Analysis and Resource Monitoring," in Digital Forensics and Cyber Crime, Communications in Computer and Information Science (CCIS), vol. 179, K. Lui, C. K. Yeo, and K. S. Yap, Eds. Berlin, Germany: Springer, 2011, pp. 199–209, doi: 10.1007/978-3-642-22170-5_18.
[14]I. Santos, J. Devesa, F. Brezo, J. Nieves, and P. G. Bringas, "OPEM: A Static-Dynamic Approach for Machine-Learning-Based Malware Detection," in Computational Intelligence in Security for Information Systems (CISIS 2012), Advances in Intelligent Systems and Computing, vol. 189. Berlin, Germany: Springer, 2013, pp. 271–280, doi: 10.1007/978-3-642-33018-6_28.
[15]M. F. Rafique, M. Ali, A. S. Qureshi, A. Khan, and A. M. Mirza, "Malware Classification Using Deep Learning Based Feature Extraction and Wrapper Based Feature Selection Technique," arXiv preprint arXiv:1910.10958, Oct. 2019, revised Dec. 2020. doi: 10.48550/arXiv.1910.10958.
[16]D. Gibert, C. Mateu, and J. Planes, "The rise of machine learning for detection and classification of malware: Research developments, trends and challenges," Journal of Network and Computer Applications, vol. 153, Art. no. 102526, 2020, doi: 10.1016/j.jnca.2019.102526.
[17]S. Saad, W. Briguglio, and H. Elmiligi, "The Curious Case of Machine Learning in Malware Detection," in Proc. 5th International Conference on Information Systems Security and Privacy (ICISSP), Prague, Czech Republic, 2019, pp. 85–94, doi: 10.5220/0007677600850094.
[18]MITRE, “Obfuscated Files or Information: Dynamic API Resolution (T1027.007),” MITRE ATT&CK, 2022. [Online]. Available: https://attack.mitre.org/techniques/T1027/007/
[19]C. Lim, Y. S. Kotualubun, S. Suryadi, and K. Ramli, "Mal-Xtract: Hidden Code Extraction Using Memory Analysis," Journal of Physics: Conference Series, vol. 801, no. 1, Art. no. 012058, 2017, doi: 10.1088/1742-6596/801/1/012058.
[20]J. Ferdous, R. Islam, M. Bhattacharya, and M. Z. Islam, "Malware Resistant Data Protection in Hyper-connected Networks: A Survey," arXiv preprint arXiv:2307.13164, Jul. 2023, doi: 10.48550/arXiv.2307.13164.
[21]G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, V. G. Sujadevi, and P. Poornachandran, "CNN-LSTM Hybrid Model for Enhanced Malware Analysis and Detection," Procedia Computer Science, vol. 233, pp. 492–503, 2024, doi: 10.1016/j.procs.2024.03.239.
[22]Y. Kawakoya, E. Shioji, M. Iwamura, and J. Miyoshi, "API Chaser: Taint-Assisted Sandbox for Evasive Malware Analysis," Journal of Information Processing, vol. 27, pp. 297–314, 2019, doi: 10.2197/ipsjjip.27.297.
[23]H. Yin, B. Lou, and P. Reiher, "A Method for Summarizing and Classifying Evasive Malware," in Proc. 26th International Symposium on Research in Attacks, Intrusions and Defenses (RAID), 2023, pp. 455–470, doi: 10.1145/3607199.3607207.
[24]H. Aghakhani, F. Gritti, F. Mecca, M. Lindorfer, S. Ortolani, D. Balzarotti, G. Vigna, and C. Kruegel, "When Malware is Packin' Heat: Limits of Machine Learning Classifiers Based on Static Analysis Features," in Proc. Network and Distributed System Security Symposium (NDSS), San Diego, CA, USA, Feb. 2020, doi: 10.14722/ndss.2020.24310
[25]B. Ndibanje, K. H. Kim, Y. J. Kang, H. H. Kim, T. Y. Kim, and H. J. Lee, "Cross-Method-Based Analysis and Classification of Malicious Behavior by API Calls Extraction," Applied Sciences, vol. 9, no. 2, Art. no. 239, 2019, doi: 10.3390/app9020239.
[26]O. A. Madamidola, F. Ngobigha, and A. Ez-zizi, "Detecting New Obfuscated Malware Variants: A Lightweight and Interpretable Machine Learning Approach," Intelligent Systems with Applications, vol. 25, Art. no. 200472, 2025, doi: 10.1016/j.iswa.2024.200472.
[27]S. Chandran, S. R. Syam, S. Sankaran, T. Pandey, and K. Achuthan, "From Static to AI-Driven Detection: A Comprehensive Review of Obfuscated Malware Techniques," IEEE Access, vol. 13, pp. 74335–74358, 2025, doi: 10.1109/ACCESS.2025.3550781.
[28]M. Naseer, F. Ullah, S. Ijaz, H. Naeem, A. Alsirhani, G. N. Alwakid, and A. Alomari, "Obfuscated Malware Detection and Classification in Network Traffic Leveraging Hybrid Large Language Models and Synthetic Data," Sensors, vol. 25, no. 1, Art. no. 202, 2025, doi: 10.3390/s25010202.
[29]S. P. Sharmila, A. Tiwari, and N. S. Chaudhari, "Obfuscated Memory Malware Detection," arXiv preprint arXiv:2408.12866, Aug. 2024, doi: 10.48550/arXiv.2408.12866.
[30]T. Carrier, P. Victor, A. Tekeoglu, and A. H. Lashkari, "Detecting Obfuscated Malware Using Memory Feature Engineering," in Proc. 8th International Conference on Information Systems Security and Privacy (ICISSP), 2022, pp. 177–188, doi: 10.5220/0010908200003120.
[31]R. H. Mahdi and H. Trabelsi, “Effective obfuscated malware detection leveraging cutting-edge machine and deep learning approaches,” Int. J. Intell. Eng. Syst., vol. 18, no. 1, pp. 1045–1057, 2025, doi: 10.22266/ijies2025.0229.75.
[32]H. S. Anderson and P. Roth, "EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models," arXiv preprint arXiv:1804.04637, Apr. 2018, doi: 10.48550/arXiv.1804.04637.
[33]L. Yang, A. Ciptadi, I. Laziuk, A. Ahmadzadeh, and G. Wang, "BODMAS: An Open Dataset for Learning Based Temporal Analysis of PE Malware," in Proc. IEEE Symposium on Security and Privacy Workshops (SPW), 2021, pp. 78–84, doi: 10.1109/SPW53761.2021.00020.
[34]Tristan Carrier, Princy Victor, Ali Tekeoglu, Arash Habibi Lashkari,” Detecting Obfuscated Malware using Memory Feature Engineering”, The 8th International Conference on Information Systems Security and Privacy (ICISSP), 2022
[35]X. Liu, J. Zhang, Y. Lin, and H. Li, "ATMPA: Attacking Machine Learning-Based Malware Visualization Detection Methods via Adversarial Examples," in Proc. IEEE/ACM 27th International Symposium on Quality of Service (IWQoS), Phoenix, AZ, USA, Jun. 2019, Art. no. 38, doi: 10.1145/3326285.3329073.
[36]P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, "API-MalDetect: Automated Malware Detection Framework for Windows Based on API Calls and Deep Learning Techniques," Journal of Network and Computer Applications, vol. 218, Art. no. 103704, 2023, doi: 10.1016/j.jnca.2023.103704.
[37]S. Zhang, M. Gao, L. Wang, S. Xu, W. Shao, and R. Kuang, "A Malware-Detection Method Using Deep Learning to Fully Extract API Sequence Features," Electronics, vol. 14, no. 1, Art. no. 167, 2025, doi: 10.3390/electronics14010167.
[38]G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, V. G. Sujadevi, and P. Poornachandran, "CNN-LSTM Hybrid Model for Enhanced Malware Analysis and Detection," Procedia Computer Science, vol. 233, pp. 492–503, 2024, doi: 10.1016/j.procs.2024.03.239.
[39]C. K. Patanaik, F. A. Barbhuiya, and S. Nandi, "Obfuscated Malware Detection Using API Call Dependency," in Proc. 1st International Conference on Security of Internet of Things (SecurIT), Kollam, India, 2012, pp. 185–193, doi: 10.1145/2490428.2490454.
[40]C. K. . Yuk and C. J. . Seo, “Static Analysis and Machine Learning-based Malware Detection System using PE Header Feature Values”, ijirss, vol. 5, no. 4, pp. 281–288, Oct. 2022.