Multi-Modal Input Mode via Graph Neural Networks for Outfit Compatibility

doi:10.23940/ijpe.21.01.p5.5059

Int J Performability Eng ›› 2021, Vol. 17 ›› Issue (1): 50-59.doi: 10.23940/ijpe.21.01.p5.5059

• Orginal Article • Previous Articles Next Articles

Multi-Modal Input Mode via Graph Neural Networks for Outfit Compatibility

Huaiguang Wu, Yan Li^*, Baohua Jin, Wenjun Shi, and Bin Lu

School of Computer and Communication Engineering, Zhengzhou University of Light Industry, Zhengzhou, 450001, China

Submitted on ; Accepted on
Contact: * Corresponding author. E-mail address: liyanjuly@163.com
About author:
Huaiguang Wu received his Ph.D. in computing from Wuhan University in 2011. He is affiliated with the School of Computer and Communication Engineering at Zhengzhou University of Light Industry. He was a postdoctoral fellow at Peking University and a research visitor at the University of Edinburgh from 2017 to 2018. His research interests include formal methods, software engineering, and algorithms. (Email: hgwu@zzuli.edu.cn)
Yan Li is a postgraduate student studying big data outfit compatibility at Zhengzhou University of Light Industry. Her research interests include artificial intelligence, big data analysis, and machine learning. (Email: liyanjuly@163.com)
Baohua Jin received his master's degree from Huazhong University of Science and Technology in 2003. Currently, he is a professor at Zhengzhou University of Light Industry. His research interests include artificial intelligence, and machine learning. (Email: jinbh@zzuli.deu.cn)
Wenjun Shi is affiliated with the School of Computer and Communication Engineering at Zhengzhou University of Light Industry. Her research interests include big data analysis and algorithms. (Email: swjij@sina.com)
Bin Lu is a postgraduate student studying big data fashion trends at Zhengzhou University of Light Industry. His research interests include artificial intelligence, big data analysis, and machine learning. (Email: binlu.computer@foxmail.com)

Abstract

Abstract:

Technology changes life. Recently, state-of-the-art of deep learning has inspired the development of deep-learning based approaches for outfit compatibility. Constructive and useful advice on clothing matching is provided to people by outfit compatibility technology. Nevertheless, there are still some shortcomings to be overcome. To be specific, outfit compatibility is a subjective method to evaluate the effect of cloth matching because of economic factors such as price, but price factor has not been taken into consideration in previous works. In addition, the relationships among vision, text, and price features are complex. Therefore, a multi-modal input graph neural network (MI-GNN) model is proposed to solve these shortcomings. This model can not only better capture the complex relations between various items in an outfit, but also model outfit compatibility from multiple modalities. In fill-in-the-blank task and outfit compatibility prediction, the performance of compatibility modeling improved the performance by 0.96% and 0.56%, respectively.

Key words: outfit compatibility, graph neural networks, compatibility modeling

Huaiguang Wu, Yan Li, Baohua Jin, Wenjun Shi, and Bin Lu. Multi-Modal Input Mode via Graph Neural Networks for Outfit Compatibility [J]. Int J Performability Eng, 2021, 17(1): 50-59.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References 32.

1.	Z. Al-Halah, R. Stiefelhagen,K. Grauman, “Fashion Forward: Forecasting Visual Style in Fashion,” in Proceedings of the IEEE International Conference on Computer Vision, 2017
2.	X. D. Liang, L. Lin, W. Yang, P. Luo, J. S. Huang,S. C. Yan, “Clothes Co-Parsing via Joint Image Segmentation and Labeling with Application to Clothing Retrieval,” IEEE Transactions on Multimedia, Vol. 18, No. 6, pp. 1175-1186, 2016
3.	K. Yamaguchi, M. H. Kiapour, L. E. Ortiz,T. L. Berg, “Retrieving Similar Styles to Parse Clothing,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, No. 5, pp. 1028-1040, 2014
4.	Z. W. Liu, P. Luo, S. Qiu, X. G. Wang,X. O. Tang, “Deepfashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016
5.	T. Iwata, S. Watanabe,H. Sawada, “Fashion Coordinates Recommender System using Photographs from Fashion Magazines,” in Proceedings of Twenty-Second International Joint Conference on Artificial Intelligence, 2011
6.	A. Veit, B. Kovacs, S. Bell, J. McAuley, K. Bala,S. Belongie, “Learning Visual Clothing Style with Heterogeneous Dyadic Co-Occurrences,” in Proceedings of the IEEE International Conference on Computer Vision, 2015
7.	S. Liu, J. S. Feng, Z. Song, T. Z. Zhang, H. Q. Lu, C. S. Xu, et al., “Hi, Magic Closet, Tell Me What to Wear!” in Proceedings of the 20th ACM International Conference on Multimedia, 2012
8.	X. T. Han, Z. X. Wu, Y. G. Jiang,L. S. Davis, “Learning Fashion Compatibility with Bidirectional LSTMs,” in Proceedings of the 25th ACM International Conference on Multimedia, 2017
9.	Z. Y. Cui, Z. K. Li, S. Wu, X. Y. Zhang,L. Wang, “Dressing as a Whole: Outfit Compatibility Learning based on Node-Wise Graph Neural Networks,” in Proceedings of the World Wide Web Conference, 2019
10.	X. M. Song, F. L. Feng, J. H. Liu, Z. K. Li, L. Q. Nie,J. Ma, “Neurostylist: Neural Compatibility Modeling for Clothing Matching,” in Proceedings of the 25th ACM International Conference on Multimedia, 2017
11.	G. Gao, L. Liu, L. Wang,Y. Zhang, “Fashion Clothes Matching Scheme based on Siamese Network and AutoEncoder,” Multimedia Systems, Vol. 25, No. 6, pp. 593-602, 2019
12.	M. I. Vasileva, B. A. Plummer, K. Dusad, S. Rajpal, R. Kumar,D. Forsyth, “Learning Type-Aware Embeddings for Fashion Compatibility,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018
13.	G. Cucurull, P. Taslakian,D. Vazquez, “Context-Aware Visual Compatibility Prediction,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019
14.	Z. G. Liu, L. Cheng, A. Liu, L. M. Zhang, X. N. He,R. Zimmermann, “Multiview and Multimodal Pervasive Indoor Localization,” in Proceedings of the 25th ACM International Conference on Multimedia, 2017
15.	D. Duvenaud, D. Maclaurin, J. Aguilera-Iparraguirre, R. Gomez-Bmbarelli, T. Hirzel, A. Aspuru-Guzik, et al., “Convolutional Networks on Graphs for Learning Molecular Fingerprints,” inAdvances in Neural Information Processing Systems, 2015
16.	M. Gori, G. Monfardini,F. Scarselli, “A New Model for Learning in Graph Domains,” in Proceedings of 2005 IEEE International Joint Conference on Neural Networks, 2005
17.	F. Scarselli, et al., “The Graph Neural Network Model,” IEEE Transactions on Neural Networks, Vol. 20, No. 1, pp. 61-80, 2008
18.	F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda,M. M. Brnstein, “Geometric Deep Learning on Graphs and Manifolds using Mixture Model CNNs,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017
19.	Y. J. Li, D. Tarlow, M. Brockschmidt,R. Zemel, “Gated Graph Sequence Neural Networks,” arXiv Preprint arXiv:1511.05493, 2015
20.	K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk,Y. Bengio, “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation,” arXiv Preprint arXiv:1406.1078, 2014
21.	X. Qi, et al., “3D Graph Neural Networks for RGBD Semantic Segmentation,” in Proceedings of the IEEE International Conference on Computer Vision, 2017
22.	K. Marino, R. Salakhutdinov,A. Gupta, “The More You Know: Using Knowledge Graphs for Image Classification,” arXiv Preprint arXiv:1612.04844, 2016
23.	R. Y. Li, M. Tapaswi, R. Liao, J. Y. Jia, R. Urtasun,S. Fidler, “Situation Recognition with Graph Neural Networks,” in Proceedings of the IEEE International Conference on Computer Vision, 2017
24.	Z. K. Li, Z. Y. Cui, S. Wu, X. Y. Zhang,L. Wang, “Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction,” in Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2019
25.	C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens,Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” inProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016
26.	K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv Preprint arXiv:1409.1556, 2014
27.	Z. Tan, et al., “Deep Semantic Role Labeling with Self-Attention,” in Proceedings of Thirty-Second AAAI Conference on Artificial Intelligence, 2018
28.	A. Vaswani, et al., “Attention is All You Need,” in Advances in Neural Information Processing Systems, 2017
29.	S. Rendle, et al., “BPR: Bayesian Personalized Ranking from Implicit Feedback,” arXiv Preprint arXiv:1205.2618, 2012
30.	Y. Hu, X. Yi,L. S. Davis, “Collaborative Fashion Recommendation: A Functional Tensor Factorization Approach,” in Proceedings of the 23rd ACM International Conference on Multimedia, 2015
31.	Y. Li, et al., “Mining Fashion Outfit Composition using an End-to-End Deep Learning Approach on Set Data,” IEEE Transactions on Multimedia, Vol. 19, No. 8, pp. 1946-1955, 2017
32.	K. Vaccaro, et al., “The Elements of Fashion Style,” in Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 2016

Multi-Modal Input Mode via Graph Neural Networks for Outfit Compatibility

PDF

Knowledge

Abstract

Cite this article

share this article

References 32.

Related Articles 0

Recommended 0