Yih-Chang Chen; Chia-Ching Lin; Sedat Agan

A Teacher-Student BERT Architecture for Semi-Supervised Learning on Unstructured Social Work Texts: Identifying Service Gaps and Needs

PDF (698KB), PP.213-230

Views: 0 Downloads: 0

Author(s)

Yih-Chang Chen ^1,* Chia-Ching Lin ² Sedat Agan ³

1. Department of Social Work, Chang Jung Christian University, Tainan 711, Taiwan, China

2. Department of Finance, Chang Jung Christian University, Tainan 711, Taiwan, China

3. Department of Physics, Kirikkale University, Kirikkale 71450, Turkey

* Corresponding author.

DOI: https://doi.org/10.5815/ijisa.2026.05.11

Received: 10 Apr. 2026 / Revised: 7 Jun. 2026 / Accepted: 14 Jul. 2026 / Published: 8 Oct. 2026

Index Terms

Semi-Supervised Learning, Natural Language Processing (NLP), Social Work, Service Gap Analysis, Topic Modeling, Knowledge Distillation, Data Augmentation

Abstract

To address the “information gap” in social work arising from unstructured data and limited labeled instances, this study proposes a semi-supervised learning framework based on a Teacher-Student BERT architecture. Based on an analysis of 50,000 case records spanning 2019 to 2024, the model incorporates Latent Dirichlet Allocation for topic extraction alongside a multidimensional gap analysis. The proposed method attained a state-of-the-art F1 score of 0.913, markedly surpassing baseline BERT models. Notable findings include a 67.6% increase in mental health-related discourse and the quantification of significant systemic deficiencies, such as inadequate coverage in elderly care (45.8%) and substantial unmet needs in medical assistance (46.4%). Furthermore, to address the profound challenges of class imbalance and the data-hungry nature of transformer models, this study integrated generative artificial intelligence (AI) data augmentation techniques. This approach produced synthetically varied case narratives that preserved the original socio-economic context while expanding lexical and structural diversity, thereby significantly enhancing the classification accuracy of minority classes. Additionally, the use of a Mean Teacher denoising framework and knowledge distillation drastically reduced the computational inference time, rendering the architecture highly suitable for deployment in resource-constrained social welfare environments. Prior to analysis, all case records underwent rigorous automated Personally Identifiable Information (PII) scrubbing utilizing the Microsoft Presidio framework to guarantee data privacy. To ensure reproducibility, the Teacher-Student training codebase, prompt architectures, and anonymized synthetic data samples will be made publicly available upon request. This research effectively transforms administrative textual data into actionable strategic intelligence, offering a scalable and evidence-based tool to enhance resource allocation and inform policy development.

Cite This Paper

Yih-Chang Chen, Chia-Ching Lin, Sedat Agan, "A Teacher-Student BERT Architecture for Semi-Supervised Learning on Unstructured Social Work Texts: Identifying Service Gaps and Needs", International Journal of Intelligent Systems and Applications (IJISA), Vol.18, No.5, pp.213-230, 2026. DOI: 10.5815/ijisa.2026.05.11

Reference

[1]A. T. Bako, H. L. Taylor, K. Wiley Jr, J. Zheng, H. Walter-McCabe, S. N. Kasthurirathne, and J. R. Vest, “Using natural language processing to classify social work interventions,” Am. J. Manag. Care, vol. 27, no. 1, pp. e24-e31, 2021. doi: 10.37765/ajmc.2021.88580
[2]B. G. Victor, B. E. Perron, R. L. Sokol, L. Fedina, and J. P. Ryan, “Automated identification of domestic violence in written child welfare records: Leveraging text mining and machine learning to enhance social work research and evaluation,” J. Soc. Social Work Res., vol. 12, no. 4, pp. 631-655, 2021. doi: 10.1086/712734
[3]S. Sun, T. Zack, C. Y. Williams, M. Sushil, and A. J. Butte, “Topic modeling on clinical social work notes for exploring social determinants of health factors,” JAMIA Open, vol. 7, no. 1, p. ooad112, 2024. doi: 10.1093/jamiaopen/ooad112
[4]H. Allam, L. Makubvure, B. Gyamfi, K. N. Graham, and K. Akinwolere, “Text classification: How machine learning is revolutionizing text categorization,” Information, vol. 16, no. 2, p. 130, 2025. doi: 10.3390/info16020130
[5]K. Taha, P. D. Yoo, C. Yeun, D. Homouz, and A. Taha, “A comprehensive survey of text classification techniques and their research applications: Observational and experimental insights,” Comput. Sci. Rev., vol. 54, p. 100664, 2024. doi: 10.1016/j.cosrev.2024.100664
[6]J. Wang, C. Qi, X. Luo, S. Deng, and Q. Lei, “Dynamic feature extraction and semi-supervised soft sensor model based on SCINet for industrial and transportation processes,” Appl. Syst. Innov., vol. 8, no. 3, p. 73, 2025. doi: 10.3390/asi8030073
[7]K. Denecke, R. May, and O. Rivera-Romero, “Transformer models in healthcare: A survey and thematic analysis of potentials, shortcomings and risks,” J. Med. Syst., vol. 48, art. 23, 2024. doi: 10.1007/s10916-024-02043-5
[8]G. Casalino, G. Castellano, O. Hryniewicz, D. Leite, K. Opara, W. Radziszewska, and K. Kaczmarek-Majer, “Semi-supervised vs. supervised learning for mental health monitoring: A case study on bipolar disorder,” Int. J. Appl. Math. Comput. Sci., vol. 33, no. 3, pp. 419-428, 2023. doi: 10.34768/amcs-2023-0030
[9]N. Farruque, R. Goebel, S. Sivapalan, R. Greiner, O. R. Zaïane, V. Masrani, and A. Hindle, “Depression symptoms modelling from social media text: An LLM driven semi-supervised learning approach,” Lang. Resour. Eval., vol. 58, pp. 1013-1041, 2024. doi: 10.1007/s10579-024-09720-4
[10]Y. Kou, Z. Chen, Y. Cao, and Q. Gu, “How does semi-supervised learning with pseudo-labelers work? A case study,” in Int. Conf. Learn. Representations, 2023.
[11]S. Khan, M. A. Khawer, R. Qureshi, M. Nawaz, M. Asim, and W. Chen, “Semi-supervised knee cartilage segmentation with successive eigen noise-assisted mean teacher knowledge distillation,” IEEE Trans. Med. Imag., vol. 44, no. 7, pp. 3051-3063, July 2025. doi: 10.1109/TMI.2025.3556870
[12]Z. Li, L. Shi, Y. Zhou, and J. Wang, “Towards student behaviour simulation: A decision transformer based approach,” in Augmented Intelligence and Intelligent Tutoring Systems. ITS 2023, Lecture Notes in Computer Science, vol. 13891, C. Frasson, P. Mylonas, and C. Troussas, Eds. Cham: Springer, 2023. doi: 10.1007/978-3-031-32883-1_49
[13]D. Dorr, C. A. Bejan, C. Pizzimenti, S. Singh, M. Storer, and A. Quinones, “Identifying patients with significant problems related to social determinants of health with natural language processing,” Stud. Health Technol. Inform., vol. 264, pp. 1456-1457, 2019. doi: 10.3233/SHTI190482
[14]M. Conway, S. Keyhani, L. Christensen, B. R. South, M. Vali, D. C. Obbold, D. L. Mowery, and W. W. Chapman, “Moonstone: A novel natural language processing system for inferring social risk from clinical narratives,” J. Biomed. Semant., vol. 10, p. 6, 2019. doi: 10.1186/s13326-019-0198-0
[15]Y. Zhang, B. Lv, L. Xue, W. Zhang, Y. Liu, Y. Fu, and Y. Qi, “SemiSAM+: Rethinking semi-supervised medical image segmentation in the era of foundation models,” arXiv preprint arXiv:2502.20749, 2025. doi: 10.48550/arXiv.2502.20749
[16]J. M. Duarte and L. Berton, “A review of semi-supervised learning for text classification,” Artif. Intell. Rev., vol. 56, pp. 9401-9469, 2023. doi: 10.1007/s10462-023-10393-8
[17]M. H. Rahman, M. A. Uddin, Z. F. Ria, and R. M. Rahman, "Optimizing BERT for Bengali Emotion Classification: Evaluating Knowledge Distillation, Pruning, and Quantization," Computer Modeling in Engineering & Sciences, vol. 142, no. 2, pp. 1637-1666, 2025. doi:10.32604/cmes.2024.058329.
[18]X. Wang, L. Weissweiler, H. Schütze, and B. Plank, “How to distill your BERT: An empirical study on the impact of weight initialisation and distillation objectives,” in Proc. 61st Annu. Meet. Assoc. Comput. Linguist. (Volume 2: Short Papers), Toronto, Canada, July 2023, pp. 1843-1852. doi: 10.18653/v1/2023.acl-short.157
[19]K. Howell, J. Wang, A. Hazare, J. Bradley, C. Brew, X. Chen, M. Dunn, B. Hockey, A. Maurer, and D. Widdows, “Domain-specific knowledge distillation yields smaller and better models for conversational commerce,” in Proc. Fifth Workshop e-Commerce NLP (ECNLP 5), Dublin, Ireland, May 2022, pp. 151-160. doi: 10.18653/v1/2022.ecnlp-1.18
[20]A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, et al., “Mistral 7B,” arXiv preprint arXiv:2310.06825, Oct. 2023. doi: 10.48550/arXiv.2310.06825
[21]J. Harte, W. Zorgdrager, P. Louridas, A. Katsifodimos, D. Jannach, and M. Fragkoulis, “Leveraging Large Language Models for Sequential Recommendation,” in Proc. 17th ACM Conf. Recommender Syst. (RecSys ‘23), New York, NY, USA: ACM, 2023, pp. 1096-1102. doi: 10.1145/3604915.3610639
[22]Y. Wu and J. Wan, “A survey of text classification based on pre-trained language model,” Neurocomputing, vol. 616, pp. 128921, 2025. doi: 10.1016/j.neucom.2024.128921
[23]E. G. Gado, T. Martorella, L. Zunino, P. Mejia-Domenzain, V. Swamy, J. Frej, and T. Käser, “Student answer forecasting: Transformer-driven answer choice prediction for language learning,” in Proc. 17th Int. Conf. Educ. Data Mining, B. Paaßen and C. D. Epp, Eds. Atlanta, GA, USA, July 2024, pp. 634-642. doi: 10.5281/zenodo.12729904
[24]A. O. Arık, G. Parlayandemir, and S. Çelik, “LLM-based data augmentation for text classification on imbalanced datasets: A case study on fake news detection,” Egypt. Inform. J., vol. 33, p. 100886, 2026. doi: 10.1016/j.eij.2026.100886
[25]H. J. Nelson, A. Munns, B. Angus, E. Arbuckle, and S. K. Burns, “Facilitators and barriers of accessing community health services for children in the early years: An Australian qualitative study,” J. Pediatr. Nurs., vol. 81, pp. 1-7, 2025. doi: 10.1016/j.pedn.2025.01.009
[26]G. A. Wasihun, M. Addise, A. Nega, A. Kifle, G. Taye, and A. Y. Gebrekidan, “Gap analysis of service quality and associated factors at the oncology center of Tikur Anbessa Specialized Hospital, Addis Ababa, Ethiopia, 2022: a cross-sectional study,” BMJ Open, vol. 14, no. 1, p. e078239, 2024. doi: 10.1136/bmjopen-2023-078239
[27]I. R. Joosse, H. A. van den Ham, A. K. Mantel-Teeuwisse, H. G. M. Leufkens, and A. M. van den Berg, “A proposed analytical framework for qualitative evaluation of access to medicines from a health systems perspective,” BMC Res. Notes, vol. 17, p. 159, 2024. doi: 10.1186/s13104-024-06764-1
[28]Z. Wang, Y. Ma, Y. Song, Y. Huang, G. Liang, and X. Zhong, “The utilization of natural language processing for analyzing social media data in nursing research: A scoping review,” J. Nurs. Manag., vol. 2024, p. 2857497, 2024. doi: 10.1155/jonm/2857497
[29]Y. Yang, H. Shi, Y. Tao, Y. Ma, B. Song, and S. Tan, “A semi-supervised feature contrast convolutional neural network for processes fault diagnosis,” J. Taiwan Inst. Chem. Eng., vol. 151, p. 105098, 2023. doi: 10.1016/j.jtice.2023.105098
[30]S. Althabiti, C. Chen, C. Wu, and V. Shanker, “Predicting Behavioral Determinants of Health from Clinical Text Using Transformer Models and BiLSTM,” medRxiv, Oct. 2025. doi: 10.1101/2025.10.13.25337944
[31]M. Samardžić-Petrović, M. Kovačević, B. Bajat, and S. Dragićević, “Machine learning techniques for modelling short term land-use change,” ISPRS Int. J. Geo-Inf., vol. 6, no. 12, p. 387, Dec. 2017. doi: 10.3390/ijgi6120387
[32]F. Chen, G. Zhang, Y. Fang, Y. Peng, and C. Weng, “Semi-supervised learning from small annotated data and large unlabeled data for fine-grained Participants, Intervention, Comparison, and Outcomes entity recognition,” J. Am. Med. Inform. Assoc., vol. 32, no. 3, pp. 555-565, March 2025. doi: 10.1093/jamia/ocae326
[33]Q. Ye, M. Khabsa, M. Lewis, S. Wang, X. Ren, and A. Jaech, “Sparse distillation: Speeding up text classification by using bigger models,” arXiv, Oct. 2021, Art. no. 2110.08536. doi: 10.48550/arXiv.2110.08536
[34]S. Wyllie, I. Shumailov, and N. Papernot, “Fairness feedback loops: Training on synthetic data amplifies bias,” in Proc. 2024 ACM Conf. Fairness, Accountability, and Transparency (FAccT ‘24), June 2024, pp. 1-35. doi: 10.1145/3630106.3659029
[35]A. Benayas, S. Miguel-Ángel, and M. Mora-Cantallops, “Enhancing intent classifier training with large language model-generated data,” Appl. Artif. Intell., vol. 38, no. 1, 2024. doi: 10.1080/08839514.2024.2414483
[36]F. Reamer, “Artificial intelligence in social work: Emerging ethical issues,” Int. J. Soc. Work Values Ethics, vol. 20, no. 2, pp. 52-71, 2023. doi: 10.55521/10-020-205
[37]M. Fareed, M. Fatima, J. Uddin, A. Ahmed, and M. A. Sattar, “A systematic review of ethical considerations of large language models in healthcare and medicine,” Front. Digit. Health, vol. 7, Art. no. 1653631, Sept. 2025. doi: 10.3389/fdgth.2025.1653631
[38]I. S. Schwartz, K. E. Link, R. Daneshjou, and N. Cortés-Penfield, “Black box warning: Large language models and the future of infectious diseases consultation,” Clin. Infect. Dis., vol. 78, no. 4, pp. 860-866, April 2024. doi: 10.1093/cid/ciad633

International Journal of Intelligent Systems and Applications (IJISA)