International Journal of Modern Education and Computer Science (IJMECS)
IJMECS Vol. 18, No. 4, 8 Aug. 2026
Cover page and Table of Contents: PDF (size: 921KB)
Comparative Performance Analysis of Generative AI Applications in PLC: An Industrial Electrical Engineering Subject
PDF (921KB), PP.25-39
Views: 0 Downloads: 0
Author(s)
Index Terms
Engineering Education, Programmable Logic Controller, Large Language Model, Statistical Analysis
Abstract
This study evaluates the performance of six generative artificial intelligence (AI) systems in solving Thai-language multiple-choice examinations in the subject of Programmable Logic Controllers (PLC), This study evaluates the performance of six generative artificial intelligence (AI) systems in solving Thai-language multiple-choice examinations in the subject of Programmable Logic Controllers (PLC), a core component of Industrial Electrical Engineering education. Six large language models (LLMs), including ChatGPT, Claude, DeepSeek, Gemini, Copilot, and Grok, were tested using fifteen sets of PLC examination questions. Statistical analysis was conducted using one-way ANOVA and two-sample t-Tests with Bonferroni correction to examine performance differences. In addition, effect size measures, including Eta-squared (η²) and Cohen’s d, were calculated to assess the magnitude of the observed differences. The results show that ChatGPT achieved the highest mean score (77.27%), while DeepSeek followed closely (76.73%) and demonstrated the lowest standard deviation (±1.83%), indicating the most consistent performance across test sets. Claude also performed strongly (74.80%), whereas Gemini, Copilot, and Grok obtained similar mid-tier scores ranging from 72.40% to 72.73%. Although all LLMs achieved scores within the passing grade range, ANOVA confirmed statistically significant differences among systems (p-value = 0.0002). However, after applying the Bonferroni correction, only a subset of pairwise differences remained statistically significant, particularly between DeepSeek and several mid-tier LLMs, while the differences among ChatGPT, Claude, and DeepSeek were not statistically significant under the adjusted threshold. Effect size analysis further indicates that some of these differences represent meaningful practical variation in LLM performance. These findings indicate that contemporary LLMs demonstrate baseline comprehension of PLC concepts and can achieve passing-level performance in technical examinations conducted in a non-English language. The study contributes empirical evidence on AI performance in Thai-language technical assessments and highlights the potential role of generative AI as a complementary learning support tool in vocational and engineering education.
Cite This Paper
Amnaj Prajong, Therdpong Daengsi, "Comparative Performance Analysis of Generative AI Applications in PLC: An Industrial Electrical Engineering Subject", International Journal of Modern Education and Computer Science(IJMECS), Vol.18, No.4, pp. 25-39, 2026. DOI:10.5815/ijmecs.2026.04.02
Reference
[1]M. Koçak, A. K. Oğuz, and Z. Akçalı, “The role of artificial intelligence in medical education: an evaluation of Large Language Models (LLMs) on the Turkish Medical Specialty Training Entrance Exam,” BMC Med. Educ., vol. 25: 609, 2025. doi: 10.1186/s12909-025-07148-0
[2]F. Guillen-Grima et al., “Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine,” Clin. Pract., vol. 13(6): 1460–1487, 2023. doi: 10.3390/clinpract13060130
[3]P. A. Massey, C. Montgomery, and A. S. Zhang, “Comparison of ChatGPT–3.5, ChatGPT-4, and Orthopaedic Resident Performance on Orthopaedic Assessment Examinations,” J. Am. Acad. Orthop. Surg., vol. 31(23): 1173–1179, 2023. doi: 10.5435/JAAOS-D-23-00396
[4]C. J. Chen, V. K. Bilolikar, D. VanNest, J. Raphael, and G. Shaffer, “Artificial intelligence in orthopaedic education: A comparative analysis of ChatGPT and Bing AI’s Orthopaedic In‐Training Examination performance,” Medicine Advances, vol. 2(3): 284–290, 2024. doi: 10.1002/med4.77
[5]S. Stankovski, G. Ostojić, S. Tegeltija, M. Stanojević, M. Babić and X. Zhang, "Generative AI Applications and Tools in Engineering Education," 2024 23rd International Symposium INFOTEH-JAHORINA (INFOTEH), 1-4, 2024. doi: 10.1109/INFOTEH60418.2024.10495941.
[6]S. M. Ismail, D. R. Rahul, I. Patra, and E. Rezvani, “The impact of formative assessment on learners’ achievement and self-regulation,” Language Testing in Asia, vol. 12, art. no. 40, 2022, doi: 10.1186/s40468-022-00191-4.
[7]B. C. Wesolowski, “Validity, reliability, and fairness in classroom tests,” in Applying Model Cornerstone Assessments in K–12 Music: A Research-Supported Approach, F. Burrack and K. A. Parkes, Eds. Lanham, MD: Rowman & Littlefield, 2018, ch. 5, pp. 83–102.
[8]G. T. L. Brown, “Products, processes, psychology, and technology: Quo vadis educational assessment?” Frontiers in Education, vol. 7, art. no. 1060633, 2022, doi: 10.3389/feduc.2022.1060633.
[9]C. H. Aguayo-Hernández, A. Sánchez Guerrero, and P. Vázquez-Villegas, “The learning assessment process in higher education: A grounded theory approach,” Education Sciences, vol. 14, no. 9, p. 984, 2024, doi: 10.3390/educsci14090984.
[10]T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual BERT?,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996–5001, 2019, doi: 10.18653/v1/p19-1493.
[11]C. J. Chen, V. K. Bilolikar, D. VanNest, J. Raphael, and G. Shaffer, “Artificial intelligence in orthopaedic education: A comparative analysis of ChatGPT and Bing AI's Orthopaedic In-Training Examination performance,” Medicine Advances, vol. 2(2): 284–290, 2024. doi: 10.1002/med4.77.
[12]L. Dolinar, E. Calcina, and E. Novak, “Evaluating Open-Source Large Language Models for Synthetic Non-English Medical Data Generation Using Prompt-Based Techniques,” Informatica, vol. 49(27): 125–134, 2025. doi: 10.31449/inf.v49i27.7763
[13]J. P. d. O. Hansen, S. Hansson and M. Koenig, "Teaching programmable logic controllers: a hands-on and skill-building approach," 2024 IEEE 7th International Conference on Industrial Cyber-Physical Systems (ICPS), 1-6, 2024. doi: 10.1109/ICPS59941.2024.10640000.
[14]J. Lasso, G. Banks, K. Wing, T. Lybeck and J. Shovic, "Flipping the Switch: Leveraging Pinball Machines to Teach PLC Programming and Control Systems," 2025 Intermountain Engineering, Technology and Computing (IETC), 1-6, 2025, doi: 10.1109/IETC64455.2025.11039489.
[15]M. M. Sakanti, A. Efendi, A. Surjanto, M. A. Falahuddin, N. Khakim and C. B. D. Kuncoro, "Development of PLC Trainer Kit as a Teaching Aid for Electric Motor Practice," 2023 10th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), 467-474, 2023. doi: 10.1109/EECSI59885.2023.10295587.
[16]S. Lu, L. Wang, T. Yang and Y. Wu, "An Engineer-Oriented Teaching Practice of Industrial Process Control Systems," 2025 IEEE Global Engineering Education Conference (EDUCON), London, United Kingdom, 1-9, 2025. doi: 10.1109/EDUCON62633.2025.11016402.
[17]P. Kim, V. Chhoeum, S. Heng and H. Yahoui, "Accreditation of PLC course in Royal University of Phnom Penh to meet Factori 4.0," 2023 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON), 360-363, 2023. doi: 10.1109/ECTIDAMTNCON57770.2023.10139494.
[18]S. Pratumsuwan and W. Pongaen, “An embedded PLC development for teaching in mechatronics education,” International Journal of Engineering Education, vol. 27, no. 3, pp. 567–574, 2011.
[19]V. Bencheva, S. Rüßmann, M. Tanev, and D. Iliev, “Capacity development through intensive training of higher education teachers for implementation of PLC curricula in South-East Asia,” International Journal of Engineering Pedagogy (iJEP), vol. 14, no. 2, pp. 34–49, 2024.
[20]M. A. Mohd Khairudin, N. A. Ahmad, and M. S. A. Rahman, “Design of portable industrial automation education training kit compatible for IR 4.0,” International Journal of Engineering and Technology, vol. 8, no. 1.9, pp. 221–226, 2019.
[21]K. Kheowsakul, P. Senajit, and S. Maneechay, “Designing and building a basic programming logic controller (PLC) kit for teaching in the programmable logic controller course,” Journal of Technical Education and Training, vol. 14, no. 2, pp. 45–54, 2022.
[22]R. Sakanti, A. H. Putra, and C. Y. Lin, “Development of PLC trainer kit as a teaching aid for electric motor practice,” International Journal of Engineering Research and Technology, vol. 12, no. 5, pp. 620–628, 2023.
[23]B. Vogel-Heuser, C. Legat, and D. Rinker, “Evaluation of a UML-based versus an IEC 61131-3-based software engineering approach for teaching PLC programming,” IEEE Transactions on Education, vol. 56, no. 3, pp. 360–366, Aug. 2013.
[24]B. Vogel-Heuser, C. Legat, and D. Rinker, “Usability evaluation on teaching and applying model-driven object-oriented approaches for PLC software,” IEEE Transactions on Industrial Informatics, vol. 8, no. 4, pp. 898–907, Nov. 2012.
[25]R. DeGuglielmo, M. A. Dempsey, and J. S. Marshall, “Introduce ladder logic and programmable logic controller (PLC) modules for engineering curriculum integration,” ASEE Annual Conference and Exposition Proceedings, pp. 1–12, 2020.
[26]N. Abdullah, M. R. Ismail, and A. A. Rahman, “Programmable logic controller (PLC) automation control learning kit programmed using the CX-programmer software,” International Journal of Interactive Mobile Technologies (iJIM), vol. 15, no. 7, pp. 144–156, 2021.
[27]S. Chumchuen, T. Pankan, and P. Akatimagool, “Experimental-based learning package of automatic control systems combining programmable logic controller and cloud SCADA system,” International Journal of Interactive Learning Environments, 2025, doi: (as listed in article).
[28]Q. T. Tran, H. Yahoui, and R. Siauve, “An interactive approach to teach automation in the training of the industry 4.0,” International Journal of Engineering Pedagogy (iJEP), vol. 9, no. 3, pp. 50–62, 2019.
[29]L. Meng and Y. Dong, “Teaching reform of PLC course based on outcome-based education in smart manufacturing context,” International Journal of Emerging Technologies in Learning (iJET), vol. 16, no. 18, pp. 135–148, 2021.
[30]P. Khieovongphachanh, S. Phommahaxay, and B. Vongphachanh, “Implement of curricula in Industry 4.0 context: Faculty of Engineering, National University of Laos,” International Journal of Engineering Pedagogy (iJEP), vol. 13, no. 1, pp. 89–101, 2023.
[31]B. Mesnard et al., “Artificial Intelligence: Ready To Pass the European Board Examinations in Urology?,” Eur. Urol. Open Sci., vol. 60: 44–46, 2024. doi: 10.1016/j.euros.2024.01.002
[32]M. Mousavi, S. Shaheei, J. M. Harley, et al., “Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada,” Fam. Med. Com. Health, vol. 12: e002626, 2024. doi: 10.1136/fmch-2023-002626
[33]N. Kadoya et al., “Assessing knowledge about medical physics in language-generative AI with large language model: using the medical physicist exam,” Radiol. Phys. Technol., vol. 17: 929–937, 2024. doi: 10.1007/s12194-024-00838-2
[34]A. Sood, N. Mansoor, C. Memmi, M. Lynch, and J. Lynch, “Generative pretrained transformer-4, an artificial intelligence text predictive model, has a high capability for passing novel written radiology exam questions,” Int. J. Comput. Assist. Radiol. Surg., vol. 19: 645–653, 2024. doi: 10.1007/s11548-024-03071-9
[35]B. Sorenson and K. Hanson, “Identifying Generative Artificial Intelligence Chatbot Use on Multiple-Choice, General Chemistry Exams Using Rasch Analysis,” J. Chem. Educ., vol. 101: 3216–3223, 2024. doi: 10.1021/acs.jchemed.4c00165
[36]J. Metsämuuronen, “Generalized Cohen’s d for multiple means and polytomous settings,” Applied Psychological Measurement, 2026, doi: 10.1177/01466216261416025.