Human Activity Recognition using Ensemble Convolutional Neural Networks and Long Short-Term Memory

doi:10.23940/ijpe.22.09.p7.660667

Abstract

Abstract: Recent advances in artificial intelligence have transformed the world into a place where things can be recognized, the surroundings can be learned, and future sequences can be predicted. The advent of advanced technologies has resulted in improving the system and reducing the cost of monitoring systems. This study proposes an advanced ensemble approach of convolutional neural networks and long short-term memory (CNN-LSTM) for human activity recognition. The proposed approach evaluates the spatio-temporal features and recognizes the activities with enhanced accuracy. The method determines the activities by utilizing the RGB, skeleton, and depth-based attributes available in the dataset of UTD-MHAD. The experiments are conducted for the hand/arm-based 21 activities for which videos were captured with the help of depth and inertial sensors. The result evaluations are conducted with the measures of accuracy, precision, recall, and f-measure. These evaluations indicate the superior performance of the proposed ensemble approach compared to state-of-art techniques.

Key words: human activity recognition, image and video processing, pattern recognition, convolutional neural networks, long short-term memory

Sonika Jindal, Monika Sachdeva, and Alok Kumar Singh Kushwaha. Human Activity Recognition using Ensemble Convolutional Neural Networks and Long Short-Term Memory [J]. Int J Performability Eng, 2022, 18(9): 660-667.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

1. Srinivasan K., Porkumaran K., andSainarayanan G.Development of 2D Human Body Modeling using Thinning Algorithm. ICTACT Journal on Image and Video Processing, vol. 1, no. 2, pp. 80-86, 2010.
2. Tripathi A., Pathak M., Sharma A., Vijay P., andJain S.A.Automated Human Detection and Tracking for Surveillance Applications. ICTACT Journal on Image and Video Processing, vol. 9, no. 4, pp. 1973-1980, 2019.
3. Saravanakumar S., Vadivel A., andAhmed C. S.Human Object Tracking in Video Sequences. ICTACT Journal on Image and Video Processing, vol. 2, no. 1, pp. 264-273, 2011.
4. Prati, A., Shan, C. and Wang, K.I.K. Sensors, Vision and Networks: From Video Surveillance to Activity Recognition and Health Monitoring. Journal of Ambient Intelligence and Smart Environments, vol. 11, no. 1, pp. 5-22, 2019.
5. Rashidi P., andCook D.J., COM: A Method for Mining and Monitoring Human Activity Patterns in Home-based Health Monitoring Systems. ACM Transactions on Intelligent Systems and Technology (TIST), vol. 4, no. 4, pp. 1-20, 2013.
6. Bianchi V., Bassoli M., Lombardo G., Fornacciari P., Mordonini M., andMunari I. De.IoT Wearable Sensor and Deep Learning: An Integrated Approach for Personalized Human Activity Recognition in a Smart Home Environment. IEEE Internet of Things Journal, vol. 6, no. 5, pp. 8553-8562, 2019.
7. Jalal, A. Kamal, S., and Kim, D. A Depth Video Sensor-based Life-Logging Human Activity Recognition System for Elderly Care in Smart Indoor Environments. Sensors, vol. 14, no. 7, pp. 11735-11759, 2014.
8. Lentzas, A. and Vrakas, D.Non-Intrusive Human activity Recognition and Abnormal Behavior Detection on Elderly People: A Review. Artificial Intelligence Review, vol. 53, no. 3, pp. 1975-2021, 2020.
9. Dhiman, C. and Vishwakarma, D.K.A Review of State-of-the-art Techniques for Abnormal Human Activity Recognition. Engineering Applications of Artificial Intelligence, vol. 77, pp. 21-45, 2019.
10. Samir H.,Abd El Munim, H.E., and Aly, G. Suspicious Human Activity Recognition using Statistical Features.Proceedings of 13th International Conference on Computer Engineering and Systems (ICCES), pp. 589-594, 2018.
11. Chen, Y., Zhong, K., Zhang, J., Sun, Q., and Zhao, X., LSTM Networks for Mobile Human Activity Recognition. Proceedings of 2016 International Conference on Artificial Intelligence: Technologies and Applications, pp. 24-25, 2016.
12. Hammerla N.Y., Halloran S., andPlötz T. Deep,Convol.utional, and Recurrent Models for Human Activity Recognition using Wearables.Proceedings of Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI'16), pp. 1533-1540, 2016.
13. Lee, S.M., Yoon, S.M., and Cho, H. Human Activity Recognition from Accelerometer Data using Convol. utional Neural Network. Proceedings of 2017 IEEE International Conference on Big Data and Smart Computing (Bigcomp), pp. 131-134, 2017
14. Murad, A.,Pyun, J.Y.Deep Recurrent Neural Networks for Human Activity Recognition. Sensors, vol. 17, no. 11, pp. 2556(1-17), 2017.
15. Hur T., Bang J., Lee J., Kim J. I., andLee S.Iss2Image: A Novel Signal-Encoding Technique for CNN-based Human Activity Recognition. Sensors, vol. 18, no. 11, pp. 3910(1-19), 2018.
16. Nweke, H.F. Teh, Y.W.Al-Garadi, M.A., and Alo, U.R. Deep Learning Algorithms for Human Activity Recognition using Mobile and Wearable Sensor Networks: State of the art and Research Challenges. Expert Systems with Applications, vol. 105, pp. 233-261, 2018.
17. Deep, S. and Zheng, X. Hybrid Model Featuring CNN and LSTM Architecture for Human Activity Recognition on Smartphone Sensor Data. Proceedings of 2019 20th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT), pp. 259-264, 2019.
18. Agarwal, P. and Alam, M.A Lightweight Deep Learning Model for Human Activity Recognition on Edge Devices. Procedia Computer Science, vol. 167, pp. 2364-2373, 2020.
19. Siraj, M.S. and Ahad, M.A.R. A Hybrid Deep Learning Framework using CNN and GRU-based RNN for Recognition of Pairwise Similar Activities. Proceedings of 2020 Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 2020 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR), pp. 1-7, 2020.
20. Chen, K. Zhang, D., Yao, L., Guo, B., Yu, Z., and Liu, Y., Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges, and Opportunities. ACM Computing Surveys (CSUR), vol. 54, no. 4, pp. 1-40, 2021.
21. Perez-Gamboa, S., Sun, Q., and Zhang, Y. Improved Sensor Based Human Activity Recognition via Hybrid Convol.utional and Recurrent Neural Networks. Proceedings of 2021 IEEE International Symposium on Inertial Sensors and Systems (INERTIAL), pp. 1-4, 2021.
22. Chen, C., Jafari, R., and Kehtarnavaz, N. UTD-MHAD: A Multimodal Dataset for Human Action Recognition Utilizing a Depth Camera and a Wearable Inertial Sensor. Proceedings of 2015 IEEE International Conference on Image Processing (ICIP), pp. 168-172, 2015.
23. Yang T., Hou Z., Liang J., Gu Y., andChao X., Depth Sequential Information Entropy Maps and Multi-label Subspace Learning for Human Action Recognition. IEEE Access, vol. 8, pp. 135118-135130, 2020.
24. Bulbul M.F., Islam S., andAli H.3D Human Action Analysis and Recognition through GLAC Descriptor on 2D Motion and Static Posture Images. Multimedia Tools and Applications, vol. 78, no. 15, pp. 21085-21111, 2019.
25. Wang, P., Li, W. Li, C., and Hou, Y.Action Recognition based on Joint Trajectory Maps with Convol.utional Neural Networks. Knowledge-Based Systems, vol. 158, pp. 43-53, 2018.
26. Popescu, A.C. Mocanu, I.,Cramariuc, B.Fusion Mechanisms for Human Activity Recognition using Automated Machine Learning. IEEE Access, vol. 8, pp. 143996-144014, 2020.

[1]	C. Rohith Bhat and Madhusundar Nelson. Artificial Intelligence Based Credit Card Fraud Detection for Online Transactions Optimized with Sparrow Search Algorithm [J]. Int J Performability Eng, 2023, 19(9): 624-632.
[2]	Pavneet Singh, Jigyasa Chopra, Amandeep Singh, Nikita Nijhawan, and Kritika. Deep Learning Innovations for Enhanced Drusen Detection in Retinal Images [J]. Int J Performability Eng, 2023, 19(12): 779-787.
[3]	Guilan Dai, Jie Zhang, and Xu Han. A Novel Attention-Based BiLSTM-CNN Model in Valence-Arousal Space [J]. Int J Performability Eng, 2022, 18(12): 833-843.
[4]	Palak Girdhar, Prashant Johri, and Deepali Virmani. An Improved Empirical Hyper-Parameter Tuned Supervised Model for Human Activity Recognition based on Motion Flow and Deep Learning [J]. Int J Performability Eng, 2022, 18(11): 808-816.
[5]	Revati M. Wahul, Archana P. Kale, and Abhishek D. Patange. Health Monitoring of Turning Tool through Vibration Signals Processed using Convolutional Neural Network Architecture [J]. Int J Performability Eng, 2022, 18(10): 730-740.
[6]	Razia Sulthana A., Arokiaraj Jovith, and Jaithunbi A. K.. LSTM and RNN to Predict COVID Cases: Lethality’s and Tests in GCC Nations and India [J]. Int J Performability Eng, 2021, 17(3): 299-306.
[7]	Xiu Kan, Xiafeng Zhang, Le Cao, Dan Yang, and Yixuan Fan. EMG Pattern Recognition based on Particle Swarm Optimization and Recurrent Neural Network [J]. Int J Performability Eng, 2020, 16(9): 1404-1415.
[8]	Jie Lin, Minghua Wei. Network Security Situation Prediction based on Combining 3D-CNNs and Bi-GRUs [J]. Int J Performability Eng, 2020, 16(12): 1875-1887.
[9]	Hui Li, Yuanyuan Zheng, and Pengju Ren. Dual-Channel Attention Model for Text Sentiment Analysis [J]. Int J Performability Eng, 2019, 15(3): 834-841.
[10]	Xiang Duan, Jianyu Long, Chuan Li, Diego Cabrera, and Shaohui Zhang. Intelligent Fault Diagnosis of 3D Printers based on Reservoir Computing [J]. Int J Performability Eng, 2019, 15(12): 3171-3178.
[11]	Jinhua Fu, Mixue Xu, Yongzhong Huang, and Hongwei Tao. A New Network Intrusion Detection System based on Blockchain [J]. Int J Performability Eng, 2019, 15(12): 3187-3195.
[12]	Pinle Qin, Qi Li, Jianchao Zeng, Haiyan Liu, and Yuhao Cui. Fully Convolutional-based Dense Network forLungNodule Image Retrieval Algorithm [J]. Int J Performability Eng, 2019, 15(1): 326-336.
[13]	Wei Jiang, Huiqiang Wang, and Keke Wu. Method for Detecting Javascript Code Obfuscation based on Convolutional Neural Network [J]. Int J Performability Eng, 2018, 14(12): 3167-3173.