Optimizing Latent Dirichlet Allocation using Metaheuristic Technique: A Comparative Study

doi:10.23940/ijpe.25.03.p2.131140

Abstract

Abstract: Community websites offer specialized online platforms for people to connect and share their knowledge about specific topics or objectives, encouraging deep engagement. A significant amount of unstructured text data can be analyzed to uncover valuable insights and trends. Latent Dirichlet Allocation (LDA) is a commonly used technique in topic modeling. However, many people use LDA with default parameters, leading to inaccurate and less cohesive topics. The selection of hyperparameters impacts the effectiveness of LDA models, and we seek to investigate the influence of the varied metaheuristic approaches on enhancing LDA's performance. In this study, we aim to analyze and compare five metaheuristic optimization algorithms for tuning the hyperparameters of the LDA model. We compare the effectiveness of Genetic Algorithms (GA), Particle Swarm Optimization algorithm (PSO), Grey Wolf Optimizer algorithm (GWO), Firefly Algorithm (FA), and Whale Optimization Algorithm (WOA). It has been found that GA is a superior metaheuristic in producing enhanced results from LDA. LDA+GA has achieved an improvement of 12.4 % over the baseline LDA technique. It has demonstrated a notable improvement in perplexity score compared to GA.

Key words: LDA, GA, PSO, GWO, topic modeling

Sneh Prabha and Neetu Sardana. Optimizing Latent Dirichlet Allocation using Metaheuristic Technique: A Comparative Study [J]. Int J Performability Eng, 2025, 21(3): 131-140.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Chen T.H., Thomas S.W., and Hassan A.E., 2016. A survey on the use of topic models when mining software repositories. Empirical Software Engineering, 21(5), pp. 1843-1919.
[2] Boyd-Graber J., Hu Y., and Mimno D., 2017. Applications of topic models. Foundations and Trends® in Information Retrieval, 11(2-3), pp. 143-296.
[3] Lin X., Liu M., and Zhang J., 2020. A top-down binary hierarchical topic model for biomedical literature. IEEE Access, 8, pp. 59870-59882.
[4] Chen H., Wang X., Pan S., and Xiong F., 2019. Identify topic relations in scientific literature using topic modeling. IEEE Transactions on Engineering Management, 68(5), pp. 1232-1244.
[5] Älgå A., Eriksson O., and Nordberg M., 2020. Analysis of scientific publications during the early phase of the COVID-19 pand emic: topic modeling study. Journal of Medical Internet Research, 22(11), e21559.
[6] De Lucia A., Di Penta M., Oliveto R., Panichella A., and Panichella S., 2014. Labeling source code with information retrieval methods: an empirical study. Empirical Software Engineering, 19, pp. 1383-1420.
[7] Blei D.M., Ng A.Y., and Jordan M.I., 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan), pp. 993-1022.
[8] Vayansky I., and Kumar S.A., 2020. A review of topic modeling methods. Information Systems, 94, 101582.
[9] Jelodar H., Wang Y., Yuan C., Feng X., Jiang X., Li Y., and Zhao L., 2019. Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. Multimedia Tools and Applications, 78, pp. 15169-15211.
[10] Melucci M.,2009. Vector-space model. In Encyclopedia of Database Systems. Springer.
[11] Foltz P.W.,1996. Latent semantic analysis for text-based research. Behavior Research Methods, Instruments, & Computers, 28, pp. 197-202.
[12] Thomas H.,1999. Probabilistic latent semantic analysis. Uncertainty in Artificial Intelligence.
[13] Lienhart R., Romberg S., and Hörster E., 2009. Multilayer pLSA for multimodal image retrieval. In Proceedings of the ACM International Conference on Image and Video Retrieval, pp. 1-8.
[14] Bassiou N.K., and Kotropoulos C.L., 2014. Online PLSA: batch updating techniques including out-of-vocabulary words. IEEE Transactions on Neural Networks and Learning Systems, 25(11), pp. 1953-1966.
[15] Wu H., Wang Y., and Cheng X., 2008. Incremental probabilistic latent semantic analysis for automatic question recommendation. In Proceedings of the 2008 ACM Conference on Recommender Systems, pp. 99-106.
[16] Peng Y., Lu Z., and Xiao J., 2009. Semantic concept annotation based on audio PLSA model. In Proceedings of the 17th ACM International Conference on Multimedia, pp. 841-844.
[17] Zhong C., and Miao Z., 2013. Multi-modal GM-plsa and its application to video classification. In 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1-4.
[18] Asmussen C.B., and Møller C., 2019. Smart literature review: a practical topic modelling approach to exploratory literature review. Journal of Big Data, 6(1), pp. 1-18.
[19] Yau C.K., Porter A., Newman N., and Suominen A., 2014. Clustering scientific documents with topic modeling. Scientometrics, 100, pp. 767-786.
[20] Reisinger J., Waters A., Silverthorn B., and Mooney R.J., 2010. Spherical topic models. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 903-910.
[21] Kherwa P., and Bansal P., 2020. Topic modeling: a comprehensive review. EAI Endorsed Trans. Scalable Inf. Syst., 7(24), e2.
[22] Zhao W., Chen J.J., Perkins R., Liu Z., Ge W., Ding Y., and Zou W., 2015. A heuristic approach to determine an appropriate number of topics in topic modeling. In BMC Bioinformatics, 16, pp. 1-10.
[23] Hosseiny Marani A., and Baumer E.P., 2023. A review of stability in topic modeling: metrics for assessing and techniques for improving stability. ACM Computing Surveys, 56(5), pp. 1-32.
[24] Mantyla M.V., Claes M., and Farooq U., 2018. Measuring LDA topic stability from clusters of replicated runs. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1-4.
[25] Grant S., and Cordy J.R., 2010. Estimating the optimal number of latent concepts in source code analysis. In 2010 10th IEEE Working Conference on Source Code Analysis and Manipulation, pp. 65-74.
[26] Lewis C.M., and Grossetti F., 2022. A statistical approach for optimal topic model identification. Journal of Machine Learning Research, 23(58), pp. 1-20.
[27] Pashakhin S.,2016. Topic modeling for frame analysis of news media. Proceedings of the AINL FRUCT, pp. 103-105.
[28] Yarnguy T., and Kanarkard W., 2018. Tuning latent Dirichlet allocation parameters using ant colony optimization. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 10(1-9), pp. 21-24.
[29] Onan A.,2018. Biomedical text categorization based on ensemble pruning and optimized topic modelling. Computational and Mathematical Methods in Medicine, 2018(1), 2497471.
[30] Panichella A.,2019. A systematic comparison of search algorithms for topic modelling—a study on duplicate bug report identification. In Search-Based Software Engineering: 11th International Symposium, SSBSE 2019, Tallinn, Estonia, August 31-September 1, 2019, Proceedings 11, pp. 11-26.
[31] Pathik N., and Shukla P., 2020. Simulated annealing based algorithm for tuning LDA hyper parameters. In Soft Computing: Theories and Applications: Proceedings of Serious and Organized Crime Threat Assessment 2019, pp. 515-521.
[32] Panichella A., Dit B., Oliveto R., Di Penta M., Poshynanyk D., and De Lucia A., 2013. How to effectively use topic models for software engineering tasks? an approach based on genetic algorithms. In 2013 35th International Conference on Software Engineering (ICSE), pp. 522-531.
[33] Agrawal A., Fu W., and Menzies T., 2018. What is wrong with topic modeling? and how to fix it using search-based software engineering. Information and Software Technology, 98, pp. 74-88.
[34] Panichella A.,2021. A systematic comparison of search-based approaches for LDA hyperparameter tuning. Information and Software Technology, 130, 106411.
[35] Tekіn Y.,2020. Optimization of LDA parameters. In 2020 28th Signal Processing and Communications Applications Conference (SIU), pp. 1-4.
[36] Holland J.H.,1992. Genetic algorithms. Scientific American, 267(1), pp. 66-73.
[37] Holland J.,1975. Adaptation in natural and artificial systems. University of Michigan, MIT.
[38] Katoch S., Chauhan S.S., and Kumar V., 2021. A review on genetic algorithm: past, present, and future. Multimedia Tools and Applications, 80, pp. 8091-8126.
[39] McCall J.,2005. Genetic algorithms for modelling and optimization. Journal of Computational and Applied Mathematics, 184(1), pp. 205-222.
[40] Man K.F., Tang K.S., and Kwong S., 1996. Genetic algorithms: concepts and applications [in engineering design]. IEEE Transactions on Industrial Electronics, 43(5), pp. 519-534.
[41] Hassanat A., Almohammadi K., Alkafaween E.A., Abunawas E., Hammouri A., and Prasath V.S., 2019. Choosing mutation and crossover ratios for genetic algorithms—a review with a new dynamic approach. Information, 10(12), 390.
[42] Ronald S.,1997. Robust encodings in genetic algorithms: A survey of encoding issues. In Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC'97), pp. 43-48.
[43] Mirjalili S., Mirjalili S.M., and Lewis A., 2014. Grey wolf optimizer. Advances in Engineering Software, 69, pp. 46-61.
[44] Mirjalili S., and Lewis A., 2016. The whale optimization algorithm. Advances in Engineering Software, 95, pp. 51-67.
[45] Kennedy J., and Eberhart R., 1995. Particle swarm optimization. In Proceedings of ICNN'95-International Conference on Neural Networks, 4, pp. 1942-1948.
[46] Dorigo M., and Birattari M., 2007. Swarm intelligence. Scholarpedia, 2(9), 1462.
[47] Zhang Y., Wang S., and Ji G., 2015. A comprehensive survey on particle swarm optimization algorithm and its applications. Mathematical Problems in Engineering, 2015(1), 931256.
[48] Cheng S., Lu H., Lei X., and Shi Y., 2018. A quarter century of particle swarm optimization. Complex & Intelligent Systems, 4(3), pp. 227-239.
[49] Yang X.S.,2010. Nature-Inspired Metaheuristic Algorithms. Luniver press.
[50] Ma J., Chen H.Y., Su R., Wang Y., Zhang S., and Shan S., 2019. Improved firefly algorithm and its application. In Proceedings of the 4th International Conference on Crowd Science and Engineering, pp. 180-185.
[51] Horng M.H.,2012. Vector quantization using the firefly algorithm for image compression. Expert Systems with Applications, 39(1), pp. 1078-1091.
[52] Horng M.H., Lee Y.X., Lee M.C., and Liou R.J., 2012. Firefly metaheuristic algorithm for training the radial basis function network for data classification and disease diagnosis. Theory and New Applications of Swarm Intelligence, 4(7), pp. 115-132.
[53] Sayadi M.K., Ramezanian R., and Ghaffari-Nasab N., 2010. A discrete firefly meta-heuristic with local search for makespan minimization in permutation flow shop scheduling problems. International Journal of Industrial Engineering Computations, 1(1), pp. 1-10.
[54] Apostolopoulos T., and Vlachos A., 2011. Application of the firefly algorithm for solving the economic emissions load dispatch problem. International Journal of Combinatorics, 2011(1), 523806.
[55] Senthilnath J., Omkar S.N., and Mani V., 2011. Clustering using firefly algorithm: performance study. Swarm and Evolutionary Computation, 1(3), pp. 164-171.
[56] Emary E., Zawbaa H.M., Ghany K.K.A., Hassanien A.E., and Parv B., 2015. Firefly optimization algorithm for feature selection. In Proceedings of the 7th Balkan Conference on Informatics Conference, pp. 1-7.
[57] Sharaqa A., and Dib N., 2014. Circular antenna array synthesis using firefly algorithm. International Journal of RF and Microwave Computer‐Aided Engineering, 24(2), pp. 139-146.
[58] Pan X., Xue L., and Li R., 2019. A new and efficient firefly algorithm for numerical optimization problems. Neural Computing and Applications, 31, pp. 1445-1453.
[59] Yang X.S., and He X., 2013. Firefly algorithm: recent advances and applications. International Journal of Swarm Intelligence, 1(1), pp. 36-50.
[60] Röder M., Both A., and Hinneburg A., 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pp. 399-408.
[61] Mimno D., Wallach H., Talley E., Leenders M., and McCallum A., 2011. Optimizing semantic coherence in topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 262-272.
[62] Aletras N., and Stevenson M., 2013. Evaluating topic coherence using distributional semantics. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)-Long Papers, pp. 13-22.