Performance Evaluation of Recommender Systems

doi:10.23940/ijpe.17.08.p7.12461256

Abstract

Abstract: Recommender systems play an important role in e-commerce. This paper discusses three classical methods - offline analytics, user study, and online experiment - to evaluate the performance of recommender systems and also analyzes their application scenarios. Some performance evaluation metrics of recommender systems are reviewed and summarized from four perspectives (machine learning, information retrieval, human-computer interaction and software engineering) combined with the above three evaluation methods. These evaluation methods and evaluation metrics summarized in the paper provide the designers with guidance for the comprehensive evaluation and selection of recommended algorithms.

Submitted on October 29, 2017; Revised on November 17, 2017; Accepted on December 1, 2017
References: 32

Mingang Chen and Pan Liu. Performance Evaluation of Recommender Systems [J]. Int J Performability Eng, 2017, 13(8): 1246-1256.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References 0

	G. Adomavicius and A. Tuzhilin, “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,” IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 6, pp. 734-749, 2015
	T. Bogers and A. V. Bosch, “Recommending scientific articles using citeulike,” in Proceedings of the 2008 ACM conference on Recommender systems, pp. 287-290, Lausanne, Switzerland, October 2008
	R. Burke, B. Mobasher B, C. Williams and R. Bhaumik, “Classification features for attack detection in collaborative recommender systems,” in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 542-547, Philadelphia, PA, USA, August 2006
	ò. Celma and P. Herrera, “A new approach to evaluating novel recommendations,” in Proceedings of the 2008 ACM conference on Recommender systems, pp.179-186, Lausanne, Switzerland, October 2008
	M. Deshpande and G.Karypis, “Item-based top-n recommendation algorithms,” ACM Transactions on Information Systems (TOIS), vol. 22, no. 1, pp. 143-177, 2003
	M. D. Ekstrand, J. T. Riedl and J. A. Konstan, “Collaborative Filtering Recommender Systems,” Foundations and Trends? in Human–Computer Interaction, vol. 4, no. 2, pp. 81-173, 2011
	T. Fawcett, “An introduction to ROC analysis,” Pattern recognition letters, vol. 27, no. 8, pp. 861-874, Amsterdam, The Netherlands, July 2006
	A. Gunawardana and G. Shani, “A survey of accuracy evaluation metrics of recommendation tasks," The Journal of Machine Learning Research, vol.10, pp. 2935-2962, 2009
	C. Hayes and P. Cunningham, “An on-line evaluation framework for recommender systems,” In Workshop on Personalization and Recommendation in E-Commerce, Malaga, Spain, May 2002
	J. Herlocker, J. A. Konstan and J. Riedl, “An empirical analysis of design choices in neighborhood-based collaborative filtering algorithms,” Information retrieval, vol. 5, no. 4, pp. 287-310, 2002
	J. L. Herlocker, J. A. Konstan and L. G. Terveen, “Evaluating collaborative filtering recommender systems,” ACM Transactions on Information Systems (TOIS), vol. 22, no. 1, pp. 5-53, 2004
	K. J?rvelin and J. Kek?l?inen, “Cumulated gain-based evaluation of IR techniques,” ACM Transactions on Information Systems, vol. 20, no. 4, pp. 422-446, 2002
	P. B. Kantor, “Recommender systems handbook,” Springer, Berlin, Germany, 2011
	A. B. Kouki, “Recommender System Performance Evaluation and Prediction: An Information Retrieval Perspective,” Universidad Autónoma de Madrid, 2012
	X. N. Lam, T. Vu, T.D. Le, and A.D. Duong, “Addressing cold-start problem in recommendation systems,” in Proceedings of the 2nd international conference on Ubiquitous information management and communication, pp. 208-211, Suwon, Korea, January 2008
	N. Lathia, S. Hailes and L. Capra, “Evaluating collaborative filtering over time,” in Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, pp. 41-42, Boston, USA, July 2009
	G. Linden, B. Smith, and J. York, “Amazon. com recommendations: Item-to-item collaborative filtering,” IEEE Internet Computing, vol. 7, no. 1, pp. 76-80, 2003
	P. Lops, M. de Gemmis and G Semeraro, “Content-based recommender systems: State of the art and trends,” Recommender Systems Handbook, Springer US, pp. 73-105, 2011
	T. Mahmood and F. Ricci, “Learning and adaptivity in interactive recommender systems,” in Proceedings of the ninth international conference on Electronic commerce, pp. 75-84, Minneapolis, MN, USA, August 2007
	S. M. McNee, J. Riedl and J. A. Konstan, “Being accurate is not enough: how accuracy metrics have hurt recommender systems,” in Proceedings of CHI’‘06 extended abstracts on Human factors in computing systems. ACM, pp. 1097-1101, Montréal, Québec, Canada, April 2006
	B. Mobasher, R. Burke, R. Bhaumik and C. Williams, “Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness,” ACM Transactions on Internet Technology (TOIT), vol. 7, no. 4, pp. 23-38, 2007
	F. Mour?o, L. Rocha and J. A. Konstan, “Exploiting non-content preference attributes through hybrid recommendation method,” in Proceedings of the 7th ACM conference on Recommender systems, pp. 177-184, Hong Kong, China, October 2013
	P. Pu and L. Chen, “Trust building with explanation interfaces,” in Proceedings of the 11th international conference on Intelligent user interfaces, pp. 93-100, Sydney, Australia, January 2006
	J. Reilly, J. Zhang, L. McGinty, P. Pu and B. Smyth, “Evaluating compound critiquing recommenders: a real-user study,” in Proceedings of the 8th ACM conference on Electronic commerce. ACM, pp.114-123, San Diego, California, USA, June 2007
	A. Said, A. Bellog?n, A. D. Vries and B. Kille, “Information Retrieval and User-Centric Recommender System Evaluation,” in Proceedings of the 21st Conference on User Modeling, Adaptation, and Personalization, Rome, Italy, June 2013
	G. Shani and A. Gunawardana, “Evaluating recommendation systems,” Recommender systems handbook. Springer US, pp. 257-297, 2011
	K. Swearingen and R. Sinha, “Beyond algorithms: An HCI perspective on recommender systems,” ACM SIGIR 2001 Workshop on Recommender Systems, New Orleans, USA, November 2001
	H. Wang, N. Wang and D. Y. Yeung, “Collaborative deep learning for recommender systems,” in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1235-1244, Sydney, NSW, Australia, August 2015
	B. Yang, T. Mei, X. S. Hua, L. Yang, S. Yang and M. Li, “Online video recommendation based on multimodal fusion and relevance feedback,” in Proceedings of the 6th ACM international conference on Image and video retrieval, pp. 73-80, 2007
	M. Zhang and N. Hurley, “Avoiding monotony: improving the diversity of recommendation lists,” in Proceedings of the 2008 ACM conference on Recommender systems, pp. 123-130, Lausanne, Switzerland, October 2008
	Y. C. Zhang, D. ó. Séaghdha, D. Quercia and T. Jambor, “Auralist: introducing serendipity into music recommendation,” in Proceedings of the fifth ACM international conference on Web search and data mining, pp.13-22, Seattle, Washington, USA, February 2012
	P. Zigoris and Y. Zhang, “Bayesian adaptive user profiling with explicit & implicit feedback,” in Proceedings of the 15th ACM international conference on Information and knowledge management, pp. 397-404, Arlington, Virginia, USA, November 2006