International Journal of Intelligent Systems and Applications (IJISA)
IJISA Vol. 17, No. 4, 8 Aug. 2025
Cover page and Table of Contents: PDF (size: 1251KB)
Leveraging Sensitivity Analysis for Configurable Kafka Clusters: A Multi-objective Model to Minimize Latency
PDF (1251KB), PP.25-39
Views: 0 Downloads: 0
Author(s)
Index Terms
Multi-objective Optimization, Minimum Search Algorithms, Sobol First Index, Morris µ, Directional Local Optimal Response
Abstract
This article presents a new multi-objective model that optimizes Kafka configuration to minimize end-to-end latency while quantifying independent parameter influence, interaction effects and sensitivity to local parameter changes. The proposed model addresses a challenging problem of selecting the configuration to prevent overloading while maintaining high availability and low latency of Kafka cluster. The study proposes an algorithm to implement this model using an adaptive optimization strategy that combines gradient-based and derivative-free search methods. This strategy enables a balance between convergence speed and global search capabilities, which is critical when dealing with the nonlinear parameter space characteristic of large-scale Kafka deployments. Experimental evaluation demonstrates 99% accuracy of the model verified against a trained XGBRegressor model and tested across multiple optimization strategies. The experimental results show that alternative configurations can be selected to meet secondary objectives-such as operational constraints - without significantly impacting latency. In this context, the designed multi-objective model serves as a valuable tool to guide the configuration selection process by quantifying and incorporating such secondary objectives into the optimization landscape. The proposed multi-objective function could be adopted in real time applications as a tool for Kafka performance tuning.
Cite This Paper
Olga Solovei, Tetiana Honcharenko, "Leveraging Sensitivity Analysis for Configurable Kafka Clusters: A Multi-Objective Model to Minimize Latency", International Journal of Intelligent Systems and Applications(IJISA), Vol.17, No.4, pp.25-39, 2025. DOI:10.5815/ijisa.2025.04.03
Reference
[1]T. Raptis, A. Passarella. “A survey on networked data streaming with apache kafka”. IEEE Access, vol. 11, pp. 85333-85350, 2023, doi: 10.1109/ACCESS.2023.3303810.
[2]M. Pacella, A. Papa, G. Papadia, & E. A.Fedeli. “Scalable Framework for Sensor Data Ingestion and Real-Time Processing in Cloud Manufacturing. Algorithms”, vol. 18, iss. 1, article no. 22, 2025, doi: 10.3390/a18010022.
[3]E. Elshoubary, T. Radwan. “Studying the Efficiency of the Apache Kafka System Using the Reduction Method, and Its Effectiveness in Terms of Reliability Metrics Subject to a Copula Approach”. Applied Sciences, vol. 14, iss. 15, article no. 6758,2024, doi: 10.3390/app14156758.
[4]K. Sathupadi, S. Achar, S. Bhaskaran, N. Faruqui, & J. Uddin. “BankNet: Real-Time Big Data Analytics for Secure Internet Banking. Big Data and Cognitive Computing”, vol. 9, iss. 2, article no. 24,2025, doi: 10.3390/bdcc9020024.
[5]M. Ezzeddine, F. Baude, F. Huet, & F. Laaziz. “Latency Aware and Resource-Efficient Bin Pack Autoscaling for Distributed Event Queues: Parameters Impact and Setting”. SN Computer Science,vol. 6, article no. 219, 2025, doi: 10.1007/s42979-025-03740-9.
[6]M. Achouch, M. Dimitrova, R. Ziane, Karganroudi, S. Sattarpanah, R. Dhouib, H. Ibrahim and M. Adda, “On predictive maintenance in industry 4.0: Overview, models, and challenges”. Applied Sciences, 12(16), p.8081, 2022, doi:10.3390/app12168081.
[7]O. Khattach, O. Moussaoui, and M. Hassine. “End-to-End Architecture for Real-Time IoT Analytics and Predictive Maintenance Using Stream Processing and ML Pipelines”. Sensors, 25(9), p.2945, 2025, Doi: 10.3390/s25092945.
[8]M. Dang'ana and H.A. Jacobsen. "Ksurf: Attention Kalman Filter and Principal Component Analysis for Prediction under Highly Variable Cloud Workloads." 11th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI). IEEE, 2024, doi: 10.1109/EECSI63442.2024.10776180.
[9]A. Nouri, C. van Treeck, & J. Frisch. “Sensitivity Assessment of Building Energy Performance Simulations Using MARS Meta-Modeling in Combination with Sobol’Method”. Energies,2024, 17(3), 695,2024, doi: 10.3390/en17030695.
[10]Z. Sadeghi, S. Matwin, “A Review of Global Sensitivity Analysis Methods and a comparative case study on Digit Classification”, arXiv preprint arXiv:2406.16975, 2024.
[11]G. Mazo, L. Tournier. “An inference method for global sensitivity analysis”.Technometrics, pp 1-13, 2025.
[12]M. Kaddoura, G. Majeau-Bettez, B. Amor and M. Margni. “Global sensitivity analysis reduces data collection efforts in LCA: A comparison between two additive manufacturing technologies”. Science of The Total Environment, 975, p.179269,2025, doi: 10.1016/j.scitotenv.2025.179269.
[13]I. H. Sarker. "Machine learning: Algorithms, real-world applications and research directions". SN computer science, vol. 2(3), March 2021, doi: 10.1007/s42979-021-00592-x.
[14]T. Klein, P. Rochet, P. “Efficiency of the averaged rank-based estimator for first order Sobol index inference”. Statistics & Probability Letters, vol. 207, April 2024, doi: 0.1016/j.spl.2023.110015.
[15]J. Morio and M. Balesdent, “Estimation of Rare Event Probabilities in Complex Aerospace and Other Systems”. Elsevier, 2016, doi: 10.1016/C2014-0-02344-1.
[16]S. Howison, “Practical Applied Mathematics: Modelling, Analysis, Approximation”. Cambridge University Press, March 2005, ISBN: 0521603692 (ISBN-13: 978‑0521603690).
[17]E. Temizhan, H. Mirtagioglu, & M. Mendes. "Which correlation coefficient should be used for investigating relations between quantitative variables". American Scientific Research Journal for Engineering, Technology, and Sciences, vol. 85(1), pp. 265-277, 2022.
[18]N. A. AL Ajmi, & M. Shoaib. Optimization Strategies in Quantum Machine Learning: A Performance Analysis. Applied Sciences, vol. 15(8), March 2025, doi: 10.3390/app15084493.
[19]Y. Ma, X. Gao, C. Liu, & J. Li. Improved sqp and slsqp algorithms for feasible path-based process optimisation. Computers & Chemical Engineering, vol. 188, Sep 2024, doi:10.1016/j.compchemeng.2024.108751.
[20]A. L. Marchildon,& D. W. Zingg. "Efficient Gradient-Enhanced Bayesian Optimizer with Comparisons to Quasi-Newton Optimizers for Unconstrained Local Optimization". arXiv preprint arXiv:2504.09375, 2025.
[21]T. Honcharenko, G. Ryzhakova, Ye. Borodavka, D. Ryzhakov, V. Savenko, O. Polosenko. Method for representing spatial information of topological relations based on a multidimensional data model, ARPN Journal of Engineering and Applied Sciences, 16(7), 2021, pp. 802-809.
[22]L. Grippo, M. Sciandrone. "Introduction to methods for nonlinear optimization (1st ed.)", Vol. 152, Springer International Publishing AG, Cham, 2023, doi: 10.1007/978-3-031-26790-1.