International Journal of Engineering and Manufacturing (IJEM)
IJEM Vol. 16, No. 3, 8 Jun. 2026
Cover page and Table of Contents: PDF (size: 369KB)
Comparative Evaluation of Evolutionary Hyperparameter Optimization for Gradient Boosting Ensembles in Clinical Multi-Class Classification
PDF (369KB), PP.1-10
Views: 0 Downloads: 0
Author(s)
Fayzullo Nazarov
1
Shokhrukh Sariyev
1,*
Mekhriddin Nurmamatov
1
Islom Yalgoshev
2
Index Terms
Hyperparameter optimization, gradient boosting, ensemble learning, genetic algorithm, differential evolution, CMA-ES, DEHB, multi-class classification, clinical decision support
Abstract
In this study, the hyperparameters of Stochastic Gradient Boosting, LightGBM, and Histogram-based Gradient Boosting models were optimized using evolutionary algorithms. The main goal of this research is to find the most effective combination of hyperparameters for each model. Their goal is to increase accuracy and improve computational efficiency. The study was conducted on the basis of real clinical data. The data were obtained from the Samarkand City Endocrinology Center, using a dataset of diabetes-related and clinical indicators. The data were initially cleaned and processed using normalization, imputation, and 5-point cross-validation. We used five evolutionary strategies to fine-tune the main hyperparameters: Differential Evolution (DE), Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Covariance Matrix Adaptation Evolution Strategy (CMA-ES), and Dynamic Ensemble with Hyperband (DEHB).In the optimization, the F1-macro was chosen as the main fitness function. This allows for a balanced accuracy assessment for multi-class classification. Across all three prototypes, evolutionary tuning consistently led to better results than Grid Search and Random Search. SGB+DE accuracy 0.9098, F1-macro 0.9096 gave the best results for the single model, whereas DEHB and CMA-ES worked better for LightGBM and HGB, respectively. The absolute increases above the unadjusted baselines ranged from 0.3 to 0.6 percentage points. They were small but could be repeated reliably across all evaluation criteria. During this study, the models were evaluated on the metrics of accuracy, precision, recall, and F1-macro.
Cite This Paper
Fayzullo Nazarov, Shokhrukh Sariyev, Mekhriddin Nurmamatov, Islom Yalgoshev, "Comparative Evaluation of Evolutionary Hyperparameter Optimization for Gradient Boosting Ensembles in Clinical Multi-Class Classification", International Journal of Engineering and Manufacturing (IJEM), Vol.16, No.3, pp.1-10, 2026. DOI:10.5815/ijem.2026.03.01
Reference
[1]Z.He, D. Lin, T. Lau, and M. Wu, "Gradient Boosting Machine: A Survey," arXiv preprint arXiv:1908.06951, 2019.
[2]O. Bakhteev, "Comprehensive Analysis of Gradient-Based Hyperparameter Optimization," Machine Learning Research Reports, 2017.
[3]C. Meaney, X. Wang, J. Guan, and T. A. Stukel, "Comparison of Methods for Tuning Machine Learning Model Hyper-Parameters: With Application to Predicting High-Need High-Cost Health Care Users," BMC Medical Research Methodology, vol. 25, no. 3, pp. 112–128, 2025, doi: 10.1186/s12874-024-02389-5.
[4]A. A. Rustamovich and N. F. Makhmadiyarovich, "Development of Control Models of Non-Stationary Systems in Limited Conditions," 2020 International Conference on Information Science and Communications Technologies (ICISCT), Tashkent, Uzbekistan, 2020, pp. 1-4, doi: 10.1109/ICISCT50599.2020.9351372
[5]A. Axatov, M. Nurmamatov, F. Nazarov, and Sh. Sariyev, "Genetic Algorithm Application Technology in Multi-Parameter Optimization Problems," AIP Conf. Proc., vol. 3244, art. no. 030025, 2024, doi: 10.1063/5.0242074.
[6]M. Molina Van den Bosch, C. Montalbano, R. Dornberger, and T. Hanne, "Evolutionary, Bayesian, and Quasi-Monte Carlo Hyperparameter Tuning," in Advances in Computational Intelligence, Springer, 2025, pp. 512–524.
[7]N. Fayzullo, S. Sariyev and Y. Sherzodjon, "Analyzing the Effectiveness of Ensemble Methods in Solving Multi-Class Classification Problems," 2025 International Russian Smart Industry Conference (SmartIndustryCon), Sochi, Russian Federation, 2025, pp. 788-793, doi: 10.1109/SmartIndustryCon65166.2025.10986248.
[8]S. Sariyev, I. Yalgoshev, and M. Nuriddinova, "Traditional Methods and Modern Approaches Based on Ensemble Algorithms for Decision-Making in Diagnostics Using Medical Data," in Proc. 2025 Int. Russian Automation Conf. (RusAutoCon), Sochi, Russia, 2025, pp. 971–975, doi: 10.1109/RusAutoCon65989.2025.11177423.
[9]X. Liu and M. Yang, "Simultaneous Curve Registration and Clustering for Functional Data," Comput. Statist. Data Anal., vol. 53, no. 4, pp. 1361–1376, 2019.
[10]L. Ye and E. Keogh, "Time Series Shapelets: A New Primitive for Data Mining," in Proc. 15th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2009, pp. 947–956.
[11]J. Kennedy and R. Eberhart, "Particle Swarm Optimization," in Proc. IEEE Int. Conf. Neural Networks (ICNN), Perth, Australia, 1995, pp. 1942–1948, doi: 10.1109/ICNN.1995.488968.
[12]N. Awad, N. Mallik, and F. Hutter, "DEHB: Evolutionary Hyper-parameter Optimization at Scale," in Proc. 30th Int. Joint Conf. Artificial Intelligence (IJCAI), 2021, pp. 2147–2153, doi: 10.24963/ijcai.2021/296.
[13]D. Maclaurin, D. Duvenaud, and R. Adams, "Gradient-Based Hyperparameter Optimization Through Reversible Learning," in Proc. 32nd Int. Conf. Machine Learning (ICML), vol. 37, 2015, pp. 2113–2122.
[14]H. Yang, G. Qin, Z. Liu, Y. Hu, and Q. Dai, "LightGBM Robust Optimization Algorithm Based on Topological Data Analysis," arXiv preprint arXiv:2406.13300, 2024.
[15]B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas, "Taking the Human Out of the Loop: A Review of Bayesian Optimization," Proc. IEEE, vol. 104, no. 1, pp. 148–175, 2016, doi: 10.1109/JPROC.2015.2494218.
[16]N. Hansen and A. Ostermeier, "Completely Derandomized Self-Adaptation in Evolution Strategies," Evolutionary Computation, vol. 9, no. 2, pp. 159–195, 2001, doi: 10.1162/106365601750190398.
[17]T. Zhang and Y. Zhao, "Hyper-Parameter Tuned LightGBM Using Memetic Firefly Algorithm," Appl. Soft Comput., vol. 110, p. 107698, 2021, doi: 10.1016/j.asoc.2021.107698.
[18]T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System," in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining, 2016, pp. 785–794, doi: 10.1145/2939672.2939785.
[19]S. Akhatkulov, I. Yalgoshev, and J. Haydarov, "Different Warmup and Annealing Strategies for ANN Models to Predict Air Quality Index," in Proc. 2025 Int. Russian Automation Conf. (RusAutoCon), Sochi, Russia, 2025, pp. 491–496, doi: 10.1109/RusAutoCon65989.2025.11177428.
[20]G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, "LightGBM: A Highly Efficient Gradient Boosting Decision Tree," in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017, pp. 3146–3154.
[21]L. Liu and Y. Wang, "Evolutionary Differential Algorithm for Hyperparameter Optimization in Machine Learning Models," Expert Syst. Appl., vol. 223, p. 120902, 2023.
[22]D. Adhikari et al., "Improving Medical Data Classification with Hybrid Evolutionary LightGBM Models," Comput. Biol. Med., vol. 148, p. 105833, 2022.
[23]S. Das and P. N. Suganthan, "Differential Evolution: A Survey of the State-of-the-Art," IEEE Trans. Evol. Comput., vol. 15, no. 1, pp. 4–31, 2011, doi: 10.1109/TEVC.2010.2059031.
[24]T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. Springer, 2009, doi: 10.1007/978-0-387-84858-7.
[25]F. Hutter, L. Kotthoff, and J. Vanschoren, Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019, doi: 10.1007/978-3-030-05318-5.
[26]A. E. Eiben and J. E. Smith, Introduction to Evolutionary Computing, 2nd ed. Springer, 2015, doi: 10.1007/978-3-662-44874-8.
[27]M. Feurer and F. Hutter, "Hyperparameter Optimization," in Automated Machine Learning: Methods, Systems, Challenges, F. Hutter, L. Kotthoff, and J. Vanschoren, Eds. Springer, 2019, pp. 3–33, doi: 10.1007/978-3-030-05318-5_1.