An Improved Optimal Method for Classification Problem

doi:10.23940/ijpe.19.11.p23.30313041

Abstract

Abstract: In order to better mine and analyze the massive data generated by search engine companies, this paper proposes a search traffic classification and dimension reduction method based on a logistic regression algorithm. Combined with distributed Hadoop technology, a text classification model is designed and implemented by data research, data analysis, and contrast experiments. In the process of feature extraction of word units, the feature combination method is used, and auxiliary information such as URL is introduced as a semaphore and optimized for the problem of low quality of training samples. The experimental results show that the model optimization effectively improves the quality of the training set. The addition of auxiliary information to train the training set can solve the under-fitting to a certain extent and improve the classification effect. The accuracy of the search traffic classification method and other indicators can reach an artificially accepted range.

Key words: search traffic, text categorization, feature reduction, feature extraction, data analysis

Wei Huang, Xiao Dong, Wenqian Shang, Weiguo Lin, and Menghan Yan. An Improved Optimal Method for Classification Problem [J]. Int J Performability Eng, 2019, 15(11): 3031-3041.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Y. W. Chen, H. Z. Wang,H. B. Li, “A Method for Extracting Web Text Semantic Subjects based on Baidu Encyclopedia and Text Classification,” Mini-Micro Systems, Vol. 33, No. 12, pp. 2605-2610, 2012
[2] F. Sebastiani, “Machine Learning in Automated Text Categorization,” Computing Surveys, Vol. 34, No. 1, pp. 1-47, 2002
[3] M. Lalmas, “Combining Document Representations,” International Journal of Cooperative Information Systems, Vol. 9, No. 4, pp. 427-447, 2000
[4] S. Z.Wu and Z. X. Zhang, “Research on Text Representation Model based on Language Network,” inProceedings of the 18th National Medical Information Conference of Chinese Medical Association, 2012
[5] S. R.Safavian and D. Landgrebe, “A Survey of Decision Tree Classifier Methodology,” IEEE Transactions on Systems, Man, and Cybernetics, Vol. 21, No. 3, pp. 660-674, 1991
[6] V. Bijalwan, V. Kumar,P. Kumari, “KNN based Machine Learning Approach for Text and Document Mining,” International Journal of Database Theory and Application, Vol. 7, No. 1, pp. 61-70, 2014
[7] W. T. Zhao, L. J. Meng,H. H. Zhao, “Application of Distributed Naive Bayesian Algorithm in Text Classification,”Measurement and Control Technology, Vol. 6, 2016
[8] W. Y. Peng, “Research on Optimization and Implementation of BP Neural Network Algorithm,” inProceedings of International Conference on Intelligent Computation Technology and Automation, pp. 104-107, 2014
[9] O. Chapelle, V. Sindhwani,S. S. Keerthi, “Optimization Techniques for Semi-Supervised Support Vector Machines,” Journal of Machine Learning Research, Vol. 9, No. 1, pp. 203-233, 2008
[10] F. J.Bu and X. Z. Qian, “KNN Text Classification Algorithm based on Vector Projection,” Computer Engineering and Design, Vol. 30, No. 21, pp. 4939-4941, 2009
[11] L. Zheng and D.H. Xu, “Study on Text Classification based on Improved TFIDF Algorithm,” Computer and Modernization, Vol. 9, pp. 6-9+14, 2014
[12] X. G. Pu, “A Review of Research on Automatic Text Classification Methods,” Information Science, Vol. 26, No. 3, pp. 469-475, 2008
[13] K. F. Deng, “Research on Mutual Information Feature Selection Method in Chinese Text Classification,” Southwest University, 2011
[14] J. F. Zhu, M. Z. Liu,C. Q. Zhao, “Application of Regularization in Logistic Regression and Neural Networks,”Information Technology, Vol. 7, pp. 1-5, 2016
[15] B. Han, Y. J. Liu, W. X. Che, et al., “Research on Incremental Training Method of Chinese Word Segmentation based on Perceptron,” Chinese Journal of Information Science, Vol. 29, No. 5, pp. 49-55, 2015
[16] S. Namburi, “Logistic Regression with Conjugate Gradient Descent for Document Classification,” Kansas State University, 2016