Deep Learning-Based Face Emotion Recognition: A Comparative Study

doi:10.23940/ijpe.24.01.p1.19

Abstract

Abstract: Face Emotion Recognition (FER) is a form of biometrics used to analyze and deduce the emotional state of any individual. Several challenges arise while implementing such systems due to the variability in expression depending on factors like age and culture, variance in head poses restricting the emotion to be captured, and occlusions degrading the quality of images. With the attention now turning towards Deep Learning (DL), Convolutional Neural Networks (CNN) are the predominant method where the parameters can get tuned, i.e., the number of layers to discover the preferred model for the given problem. This study experiments with a CNN model incorporating 12 layers, operating on two datasets to detect various emotions. A comparison is made between the performance of the CNN model using visual metrics, including confusion matrix and classification reports, along with evaluation metrics such as accuracy, precision, recall, and F1-score. The experimented CNN model obtained an accuracy of 86% with Young AffectNet-HQ and 92% achieved with AffectNet-HQ, thereby showing promising results.

Key words: facial emotion recognition, computer vision, convolutional neural network, deep learning, machine learning

Rohit Chandra Joshi, Aayush Juyal, Abhijeet Mishra, Avni Verma, and Kanika Singla. Deep Learning-Based Face Emotion Recognition: A Comparative Study [J]. Int J Performability Eng, 2024, 20(1): 1-9.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Umer S., Rout R.K., Pero C., andNappi M.Facial Expression Recognition with Trade-Offs between Data Augmentation and Deep Learning Features. Journal of Ambient Intelligence and Humanized Computing, pp. 1-15, 2022.
[2] Sana S.K., Sruthi G., Suresh D., Rajesh G., andReddy G.S.Facial Emotion Recognition Based Music System using Convolutional Neural Networks. Materials Today: Proceedings, vol. 62, pp. 4699-4706, 2022.
[3] Joseph, A. and Geetha, P.Facial Emotion Detection using Modified Eyemap-Mouthmap Algorithm on an Enhanced Image and Classification with Tensorflow. The Visual Computer, vol. 36, no. 3, pp. 529-539, 2020.
[4] Akhand M.A.H., Roy, S., Siddique, N., Kamal, M.A.S., and Shimamura, T. Facial Emotion Recognition using Transfer Learning in the Deep CNN. Electronics, vol. 10, no. 9, pp. 1036, 2021.
[5] Agrawal, A. and Mittal, N. using CNN for Facial Expression Recognition: A Study of the Effects of Kernel Size and Number of Filters on Accuracy. The Visual Computer, vol. 36, no. 2, pp. 405-412, 2020.
[6] Mohan K., Seal A., Krejcar O., andYazidi A.Facial Expression Recognition using Local Gravitational Force Descriptor-Based Deep Convolution Neural Networks. IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-12, 2020.
[7] Khattak A., Asghar M.Z., Ali M., andBatool U.An Efficient Deep Learning Technique for Facial Emotion Recognition. Multimedia Tools and Applications, pp. 1-35, 2022.
[8] Ma F., Sun B., andLi S.Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion. IEEE Transactions on Affective Computing, 2021.
[9] Nonis F., Dagnes N., Marcolin F., andVezzetti E.3D Approaches and Challenges in Facial Expression Recognition Algorithms-A Literature Review. Applied Sciences, vol. 9, no. 18, pp. 3904, 2019.
[10] Febrian R., Halim B.M., Christina M., Ramdhan D., andChowanda A.Facial Expression Recognition using Bidirectional LSTM-CNN. Procedia Computer Science, vol. 216, pp. 39-47, 2023.
[11] Saxena A., Khanna A., andGupta D.Emotion Recognition and Detection Methods: A Comprehensive Survey. Journal of Artificial Intelligence and Systems, vol. 2, no. 1, pp. 53-79, 2020.
[12] Shahzad H.M., Bhatti S.M., Jaffar A., Akram S., Alhajlah M., andMahmood A.Hybrid Facial Emotion Recognition using CNN-Based Features. Applied Sciences, vol. 13, no. 9, pp. 5572, 2023.
[13] Khan A.R.Facial Emotion Recognition using Conventional Machine Learning and Deep Learning Methods: Current Achievements, Analysis and Remaining Challenges. Information, vol. 13, no. 6, pp. 268, 2022.
[14] Kottursamy K.A Review on Finding Efficient Approach to Detect Customer Emotion Analysis using Deep Learning Analysis. Journal of Trends in Computer Science and Smart Technology, vol. 3, no. 2, pp. 95-113, 2021.
[15] Debnath T., Reza M.M., Rahman A., Beheshti A., Band S.S., andAlinejad-Rokny, H. Four-Layer ConvNet to Facial Emotion Recognition with Minimal Epochs and the Significance of Data Diversity. Scientific Reports, vol. 12, no. 1, pp. 6991, 2022.
[16] Hassouneh A., Mutawa A.M., andMurugappan M.Development of a Real-Time Emotion Recognition System using Facial Expressions and EEG Based on Machine Learning and Deep Neural Network Methods. Informatics in Medicine Unlocked, vol. 20, pp. 100372, 2020.
[17] Cadayona A.M., Cerilla N.M.S., Jurilla D.M.M., Balan A.K.D., andde Goma, J.C. Emotional State Classification: An Additional Step in Emotion Classification through Face Detection. In2019 IEEE 6th International Conference on Industrial Engineering and Applications (ICIEA), IEEE, pp. 667-671, 2019.
[18] Patil, M. and Veni, S. Driver Emotion Recognition for Enhancement of Human Machine Interface in Vehicles. In2019 International Conference on Communication and Signal Processing (ICCSP), IEEE, pp. 420-424, 2019.
[19] Li, B. and Lima, D.Facial Expression Recognition via ResNet-50. International Journal of Cognitive Computing in Engineering, vol. 2, pp. 57-64, 2021.
[20] Zhang S., Pan X., Cui Y., Zhao X., andLiu L.Learning Affective Video Features for Facial Expression Recognition via Hybrid Deep Learning. IEEE Access, vol. 7, pp. 32297-32304, 2019.
[21] Mukhopadhyay M., Dey A., Shaw R.N., andGhosh A. Facial Emotion Recognition Based on Textural Pattern and Convolutional Neural Network. In2021 IEEE 4th International Conference on Computing, Power and Communication Technologies (GUCON), IEEE, pp. 1-6, 2021.
[22] Said, Y. and Barr, M.Human Emotion Recognition Based on Facial Expressions via Deep Learning on High-Resolution Images. Multimedia Tools and Applications, vol. 80, no. 16, pp. 25241-25253, 2021.
[23] Makhmudkhujaev F.,Abdullah-Al-Wadud, M., Iqbal, M.T.B., Ryu, B., and Chae, O. Facial Expression Recognition with Local Prominent Directional Pattern. Signal Processing: Image Communication, vol. 74, pp. 1-12, 2019.
[24] Khaireddin, Y. and Chen, Z.Facial Emotion Recognition: State of the Art Performance on FER2013. arXiv preprint arXiv:2105.03588, 2021.
[25] Chowdary M.K., Nguyen T.N., andHemanth D.J.Deep Learning-Based Facial Emotion Recognition for Human-Computer Interaction Applications. Neural Computing and Applications, pp. 1-18, 2021.
[26] Saurav S., Saini R., andSingh S.EmNet: A Deep Integrated Convolutional Neural Network for Facial Emotion Recognition in the Wild. Applied Intelligence, vol. 51, pp. 5543-5570, 2021.
[27] Li K., Jin Y., Akram M.W., Han R., andChen J.Facial Expression Recognition with Convolutional Neural Networks via a New Face Cropping and Rotation Strategy. The visual computer, vol. 36, pp. 391-404, 2020.
[28] Li J., Jin K., Zhou D., Kubota N., andJu Z.Attention Mechanism-Based CNN for Facial Expression Recognition. Neurocomputing, vol. 411, pp. 340-350, 2020.
[29] Doma, V. and Pirouz, M.A Comparative Analysis of Machine Learning Methods for Emotion Recognition using EEG and Peripheral Physiological Signals. Journal of Big Data, vol. 7, no. 1, pp. 1-21, 2020.
[30] Wang, L. and Siddique, A.A.Facial Recognition System using LBPH Face Recognizer for Anti-Theft and Surveillance Application Based on Drone Technology. Measurement and Control, vol.53, no. 7-8, pp. 1070-1077, 2020.
[31] Marechal C., Mikolajewski D., Tyburek K., Prokopowicz P., Bougueroua L., Ancourt C., andWegrzyn-Wolska, K.Survey on AI-Based Multimodal Methods for Emotion Detection. High-performance modelling and simulation for big data applications, vol. 11400, pp. 307-324, 2019.
[32] Mellouk, W. and Handouzi, W.Facial Emotion Recognition using Deep Learning: Review and Insights. Procedia Computer Science, vol. 175, pp. 689-694, 2020.