International Journal of Intelligent Systems and Applications (IJISA)
IJISA Vol. 18, No. 2, 8 Apr. 2026
Cover page and Table of Contents: PDF (size: 662KB)
Sentence Classification in Medical Abstracts Using Quantized Transformer and BiLSTM Architecture
PDF (662KB), PP.156-166
Views: 0 Downloads: 0
Author(s)
Ahmed Abdal Shafi Rasel
1,*
Md. Towhidul Islam Robin
2
Md. Samiul Islam
3
Mehedi Hasan
2
Index Terms
Sentence Classification, Medical Journal, Abstract, Transformer Block, Attention, LSTM, Word Vectors, Word2vec, Contextual Modeling
Abstract
Automatically classifying abstract sentences into significant categories such as - background, methods, objective, result, and conclusions - is an essential support tool for scientific medical database querying that assists in searching and summarizing relevant literature works and writing new abstracts. This paper presents a memory-efficient deep learning model for sentence role classification in medical scientific abstracts, achieved by integrating quantized Transformer layers with a Bidirectional Long Short-Term Memory (BiLSTM) network. While the core components are recognized, our contribution is demonstrated in the successful application of quantization to this hybrid architecture, significantly reducing model size (from ~75MB to ~25MB) without a meaningful drop in classification performance on a subset of the PubMed 200k RCT dataset. This makes our approach distinctively practical for deployment in resource-constrained environments, offering an effective tool for automated literature analysis.
Cite This Paper
Ahmed Abdal Shafi Rasel, Md. Towhidul Islam Robin, Md. Samiul Islam, Mehedi Hasan, "Sentence Classification in Medical Abstracts Using Quantized Transformer and BiLSTM Architecture", International Journal of Intelligent Systems and Applications(IJISA), Vol.18, No.2, pp.156-166, 2026. DOI:10.5815/ijisa.2026.02.11
Reference
[1]Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, Jeffrey Dean, "Distributed Representations of Words and Phrases and their Compositionality", Advances in Neural Information Processing Systems, 2013.
[2]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding", Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[3]Haibo He, Edwardo A. Garcia, "Learning from Imbalanced Data", IEEE Transactions on Knowledge and Data Engineering, Vol.21, No.9, pp.1263-1284, 2009. DOI:10.1109/TKDE.2008.239
[4]X. Zhao, Y. Liu, L. Li, C. Zhang, "A Novel Learning Strategy Based on Dynamic Resampling for Imbalanced Data", IEEE Access, Vol.9, pp.121509-121523, 2021. DOI:10.1109/ACCESS.2021.3108099
[5]J. Li, Y. Xu, H. Shi, "Bidirectional LSTM with Hierarchical Attention for Text Classification", 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Vol.1, pp.456-459, 2019.
[6]Y. Guo, H. Zhao, Z. Peng, L. Wang, "Attention-based BiLSTM Model for Medical Text Classification", IEEE Access, Vol.10, pp.53604-53614, 2022. DOI:10.1109/ACCESS.2022.3171067
[7]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, "Attention Is All You Need", Advances in Neural Information Processing Systems, 2017.
[8]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI Blog, Vol.1, No.8, 2019.
[9]Franck Dernoncourt, Ji Young Lee, Peter Szolovits, "Neural Networks for Joint Sentence Classification in Medical Paper Abstracts", CoRR, Vol.abs/1612.05251, 2016.
[10]X. Yu, Z. Huang, X. Li, J. Zhang, "Joint Learning for Medical Text Classification and Summarization", IEEE Access, Vol.9, pp.144654-144664, 2021. DOI:10.1109/ACCESS.2021.3122826
[11]Di Jin, Peter Szolovits, "Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts", CoRR, Vol.abs/1808.06161, 2018.
[12]W. Huang, Y. Huang, M. Gu, X. Lian, "Sentence-level Text Classification Using Attention-based Hierarchical Recurrent Neural Network", IEEE Access, Vol.9, pp.97277-97289, 2021. DOI:10.1109/ACCESS.2021.3093139
[13]S. Gonçalves, P. Cortez, S. Moro, "A Deep Learning Classifier for Sentence Classification in Biomedical and Computer Science Abstracts", Neural Computing and Applications, Vol.32, pp.6793-6807, 2019.
[14]X. Yuan, Y. Zhang, Y. Sun, "A BERT-based Deep Learning Model for Sentence Classification in Medical Texts", Neural Computing and Applications, Vol.34, pp.10591-10603, 2022.
[15]Kyunghyun Cho, Bart van Merrienboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio, "Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation", CoRR, Vol.abs/1406.1078, 2014.
[16]H. Song, C. Tao, Z. Fan, H. Wang, "Learning Cross-lingual Phrase Representations Using Multilingual Transformers", IEEE Access, Vol.9, pp.71247-71256, 2021. DOI:10.1109/ACCESS.2021.3080486
[17]Steven Bird, "NLTK: The Natural Language Toolkit", Proceedings of the ACL Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, 2002.
[18]Radim Řehůřek, Petr Sojka, "Gensim–Python Framework for Vector Space Modelling", NLP Centre, Faculty of Informatics, Masaryk University, Vol.3, No.2, 2011.
[19]Y. Choi, M. Kang, S. Yoon, "Quantization Techniques for Deep Learning Models: A Survey", Journal of Machine Learning Research, Vol.21, No.102, pp.1-38, 2020.