January F. Naga; Julieto E. Perez

Learner Engagement State Typologies in AI Tutoring: A Clustering Analysis of Dialogue Behaviors in Introductory Programming Sessions

PDF (992KB), PP.152-166

Views: 0 Downloads: 0

Author(s)

January F. Naga ^1,* Julieto E. Perez ²

1. Department of Information Technology, College of Computer Studies, MSU-Iligan Institute of Technology, Philippines

2. Department of Computer Science, College of Computer Studies, MSU-Iligan Institute of Technology, Philippines

* Corresponding author.

DOI: https://doi.org/10.5815/ijmecs.2026.03.10

Received: 5 Jul. 2025 / Revised: 17 Jan. 2026 / Accepted: 14 Feb. 2026 / Published: 8 Jun. 2026

Index Terms

Large Language Models, Intelligent Tutoring Systems, Engagement State Modeling, Self-Regulated Learning, Metacognition

Abstract

As Large Language Models (LLMs) become increasingly embedded in intelligent tutoring systems (ITSs), a growing need exists to understand how learners engage with these tools, especially in cognitively demanding domains such as computer programming. While prior research has focused mainly on LLM-generated scaffolding, less attention has been paid to student-side engagement, including how learners think, respond, and regulate their learning during natural language tutoring sessions. This study addresses that gap by identifying learner engagement state profiles based on behavioral and metacognitive patterns in student dialogue. Drawing on data from 36 recorded LLM-mediated C introductory programming tutorials, 1,046 dual-annotated student utterances were analyzed using K-means clustering. The analysis revealed three distinct learner engagement state profiles: Passive Reactors, Clarification Seekers, and Reflective Performers. These engagement states differed in response accuracy, metacognitive expression, and interaction style. Passive Reactors showed low initiative and limited self-regulation; Clarification Seekers demonstrated moderate accuracy and reactive help-seeking; Reflective Performers exhibited strategic engagement and high metacognitive activity. This study introduces an exploratory, scalable approach to learner profiling through natural language dialogue, advancing the design of adaptive, learner-aware LLM tutoring systems. The findings support the development of real-time learner modeling techniques that move beyond correctness, offering actionable insights for delivering more personalized and effective AI-assisted instruction in programming education.

Cite This Paper

January F. Naga, Julieto E. Perez, "Learner Engagement State Typologies in AI Tutoring: A Clustering Analysis of Dialogue Behaviors in Introductory Programming Sessions", International Journal of Modern Education and Computer Science(IJMECS), Vol.18, No.3, pp. 152-166, 2026. DOI:10.5815/ijmecs.2026.03.10

Reference

[1]S. Steinert, K. E. Avila, S. Ruzika, J. Kuhn, and S. Küchemann, "Harnessing large language models to develop research-based learning assistants for formative feedback," Smart Learning Environments, vol. 11, no. 1, p. 62, 2024/12/19 2024, doi: https://doi.org/10.1186/s40561-024-00354-1.
[2]W. Xing, N. Nixon, S. Crossley, P. Denny, A. Lan, J. Stamper, and Z. Yu, "The Use of Large Language Models in Education," International Journal of Artificial Intelligence in Education, vol. 35, no. 2, pp. 439-443, 2025/06/01 2025, doi: https://doi.org/10.1007/s40593-025-00457-x.
[3]G. Jošt, V. Taneski, and S. Karakatič, "The Impact of Large Language Models on Programming Education and Student Learning Outcomes," Applied Sciences, vol. 14, no. 10, p. 4115, 2024, doi: https://doi.org/10.3390/app14104115.
[4]M. Giannakos et al., "The promise and challenges of generative AI in education," Behaviour & Information Technology, pp. 1-27, doi: https://doi.org/10.1080/0144929X.2024.2394886.
[5]A. Lubbe, E. Marais, and D. Kruger, "Cultivating independent thinkers: The triad of artificial intelligence, Bloom’s taxonomy and critical thinking in assessment pedagogy," Education and Information Technologies, 2025/03/12 2025, doi: https://doi.org/10.1007/s10639-025-13476-x.
[6]E. Kasneci et al., "ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education," Learning and Individual Differences, vol. 103, p. 102274, 01/30 2023, doi: https://doi.org/10.1016/j.lindif.2023.102274.
[7]Z. Pardos and S. Bhandari, "ChatGPT-generated help produces learning gains equivalent to human tutor-authored help on mathematics skills," PLOS ONE, vol. 19, 05/24 2024, doi: https://doi.org/10.1371/journal.pone.0304013.
[8]P. Cui and M. Sachan, Investigating the Zone of Proximal Development of Language Models for In-Context Learning. 2025, pp. 6470-6483.
[9]O. Zawacki-Richter, V. I. Marín, M. Bond, and F. Gouverneur, "Systematic review of research on artificial intelligence applications in higher education – where are the educators?" International Journal of Educational Technology in Higher Education, vol. 16, no. 1, p. 39, 2019/10/28 2019, doi: 10.1186/s41239-019-0171-0.
[10]D. Gašević, S. Dawson, T. Rogers, and D. Gasevic, "Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success," The Internet and Higher Education, vol. 28, pp. 68-84, 2016/01/01/ 2016, doi: https://doi.org/10.1016/j.iheduc.2015.10.002.
[11]E. Lahtinen, K. Ala-Mutka, and H.-M. Järvinen, A study of the difficulties of novice programmers. 2005, pp. 14-18.
[12]R. Lister et al., "A Multi-National Study of Reading and Tracing Skills in Novice Programmers," ACM SIGCSE Bulletin, vol. 36, pp. 119-150, 01/01 2005, doi: 10.1145/1044550.1041673.
[13]A. Robins, R. Janet, and N. and Rountree, "Learning and Teaching Programming: A Review and Discussion," Computer Science Education, vol. 13, no. 2, pp. 137-172, 2003/06/01 2003, doi: https://doi.org/10.1076/csed.13.2.137.14200.
[14]P. Kinnunen and L. Malmi, "Why students drop out CS1 course?," presented at the Proceedings of the second international workshop on Computing education research, Canterbury, United Kingdom, 2006. [Online]. Available: https://doi.org/10.1145/1151588.1151604.
[15]E. Panadero, "A Review of Self-regulated Learning: Six Models and Four Directions for Research," Frontiers in Psychology, vol. 8, 2017, doi: https://doi.org/10.3389/fpsyg.2017.00422.
[16]B. J. Zimmerman, "Chapter 2 - Attaining Self-Regulation: A Social Cognitive Perspective," in Handbook of Self-Regulation, M. Boekaerts, P. R. Pintrich, and M. Zeidner Eds. San Diego: Academic Press, 2000, pp. 13-39.
[17]C. A. Brenner, "Self-regulated learning, self-determination theory and teacher candidates’ development of competency-based teaching practices," Smart Learning Environments, vol. 9, no. 1, p. 3, 2022/01/06 2022, doi: https://doi.org/10.1186/s40561-021-00184-5.
[18]F. Arianto and M. Hanif, "Evaluating metacognitive strategies and self-regulated learning to predict primary school students self-efficacy and problem-solving skills in science learning," Journal of Pedagogical Research, 2024, doi: https://doi.org/10.33902/jpr.202428575.
[19]R. Azevedo et al., "Lessons Learned and Future Directions of MetaTutor: Leveraging Multichannel Data to Scaffold Self-Regulated Learning With an Intelligent Tutoring System," (in English), Frontiers in Psychology, Review vol. Volume 13 - 2022, 2022-June-14 2022, doi: 10.3389/fpsyg.2022.813632.
[20]D. Sachan and K. Saroha, "A Review of Adaptive and Intelligent Online Learning Systems," Singapore, 2022: Springer Nature Singapore, in ICT Analysis and Applications, pp. 251-262, doi: https://doi.org/10.1007/978-981-16-5655-2_24.
[21]Y. Fang, A. Lippert, Z. Cai, S. Chen, J. C. Frijters, D. Greenberg, and A. C. Graesser, "Patterns of Adults with Low Literacy Skills Interacting with an Intelligent Tutoring System," International Journal of Artificial Intelligence in Education, vol. 32, no. 2, pp. 297-322, 2022/06/01 2022, doi: https://doi.org/10.1007/s40593-021-00266-y.
[22]P. H. Winne and A. F. Hadwin, "Studying as self-regulated learning," in Metacognition in educational theory and practice., (The educational psychology series. Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers, 1998, pp. 277-304.
[23]K. Muldner and W. Burleson, "Utilizing sensor data to model students’ creativity in a digital environment," Computers in Human Behavior, vol. 42, pp. 127-137, 2015/01/01/ 2015, doi: https://doi.org/10.1016/j.chb.2013.10.060.
[24]S.-H. Jin, K. Im, M. Yoo, I. Roll, and K. Seo, "Supporting students’ self-regulated learning in online learning using artificial intelligence applications," International Journal of Educational Technology in Higher Education, vol. 20, no. 1, p. 37, 2023/06/26 2023, doi: 10.1186/s41239-023-00406-5.
[25]L. Huang, L. Dias, E. Nelson, L. Liang, S. P. Lajoie, and E. G. Poitras, "The Role of Self-Improving Tutoring Systems in Fostering Pre-Service Teacher Self-Regulated Learning," (in English), Frontiers in Artificial Intelligence, Original Research vol. Volume 4 - 2021, 2022-January-03 2022, doi: 10.3389/frai.2021.769455.
[26]A. J. Carroll and J. Borycz, "Integrating large language models and generative artificial intelligence tools into information literacy instruction," The Journal of Academic Librarianship, vol. 50, no. 4, p. 102899, 2024/07/01/ 2024, doi: https://doi.org/10.1016/j.acalib.2024.102899.
[27]Y.-M. Yan, C.-Q. Chen, Y.-B. Hu, and X.-D. Ye, "LLM-based collaborative programming: impact on students’ computational thinking and self-efficacy," Humanities and Social Sciences Communications, vol. 12, no. 1, p. 149, 2025/02/07 2025, doi: 10.1057/s41599-025-04471-1.
[28]A. Gaeta, F. Orciuoli, A. Pascuzzo, and A. Peduto, "Enhancing traditional ITS architectures with large language models for generating motivational feedback," Computers and Education: Artificial Intelligence, vol. 9, p. 100433, 2025/12/01/ 2025, doi: https://doi.org/10.1016/j.caeai.2025.100433.
[29]C.-H. Lai and C.-Y. Lin, "Analysis of Learning Behaviors and Outcomes for Students with Different Knowledge Levels: A Case Study of Intelligent Tutoring System for Coding and Learning (ITS-CAL)," Applied Sciences, vol. 15, no. 4, p. 1922, 2025. [Online]. Available: https://www.mdpi.com/2076-3417/15/4/1922.
[30]E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, "ChatGPT for good? On opportunities and challenges of large language models for education," Learning and Individual Differences, vol. 103, Art. no. 102274, 03/01 2023, doi: 10.1016/j.lindif.2023.102274.
[31]J. Han, M. Kamber, and J. Pei, Data Mining: Concepts and Techniques, 4th ed. Morgan Kaufmann, 2022.
[32]V. Kovanovic, S. Joksimovic, D. Gasevic, G. Siemens, and M. Hatala, "What public media reveals about MOOCs: A systematic analysis of news reports," British Journal of Educational Technology, vol. 46, pp. 510-527, 04/01 2015, doi: 10.1111/bjet.12277.
[33]J. A. Larusson and R. Alterman, "Wikis to support the “collaborative” part of collaborative learning," International Journal of Computer-Supported Collaborative Learning, vol. 4, no. 4, pp. 371-402, 2009/12/01 2009, doi: 10.1007/s11412-009-9076-6.
[34]R. Davies, G. Allen, C. Albrecht, N. Bakir, and N. Ball, "Using Educational Data Mining to Identify and Analyze Student Learning Strategies in an Online Flipped Classroom," Education Sciences, vol. 11, no. 11, p. 668, 2021. [Online]. Available: https://www.mdpi.com/2227-7102/11/11/668.
[35]A. Létourneau, M. Deslandes Martineau, P. Charland, J. A. Karran, J. Boasen, and P. M. Léger, "A systematic review of AI-driven intelligent tutoring systems (ITS) in K-12 education," npj Science of Learning, vol. 10, no. 1, p. 29, 2025/05/14 2025, doi: 10.1038/s41539-025-00320-7.
[36]C. Conati and S. Kardan, "Student Modeling: Supporting Personalized Instruction, from Problem Solving to Exploratory Open-Ended Activities," AI Magazine, vol. 34, pp. 13-26, 09/01 2013, doi: 10.1609/aimag.v34i3.2483.
[37]J. van der Graaf et al., "How to design and evaluate personalized scaffolds for self-regulated learning," Metacognition and Learning, vol. 18, no. 3, pp. 783-810, 2023/12/01 2023, doi: 10.1007/s11409-023-09361-y.
[38]A. M. Ikotun, A. E. Ezugwu, L. Abualigah, B. Abuhaija, and J. Heming, "K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big data," Information Sciences, vol. 622, pp. 178-210, 2023/04/01/ 2023, doi: https://doi.org/10.1016/j.ins.2022.11.139.
[39]J. R. Landis and G. G. Koch, "The Measurement of Observer Agreement for Categorical Data," Biometrics, vol. 33, no. 1, pp. 159-174, 1977, doi: https://doi.org/10.2307/2529310.
[40]P. J. Rousseeuw, "Silhouettes: A graphical aid to the interpretation and validation of cluster analysis," Journal of Computational and Applied Mathematics, vol. 20, pp. 53-65, 1987/11/01/ 1987, doi: https://doi.org/10.1016/0377-0427(87)90125-7.
[41]K. VanLehn, "The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems," Educational Psychologist, vol. 46, no. 4, pp. 197-221, 2011, doi: https://doi.org/10.1080/00461520.2011.611369.
[42]M. T. H. Chi, "Active-Constructive-Interactive: A Conceptual Framework for Differentiating Learning Activities," Topics in Cognitive Science, vol. 1, no. 1, pp. 73-105, 2009, doi: https://doi.org/10.1111/j.1756-8765.2008.01005.x.

International Journal of Modern Education and Computer Science (IJMECS)