Enhanced Emotion Recognition in Videos: A Convolutional Neural Network Strategy for Human Facial Expression Detection and Classification

Arselan Ashraf, Teddy Surya Gunawan, Fatchul Arifin, Mira Kartiwi, Ali Sophian, Mohamed Hadi Habaebi


The human face is essential in conveying emotions, as facial expressions serve as effective, natural, and universal indicators of emotional states. Automated emotion recognition has garnered increasing interest due to its potential applications in various fields, such as human-computer interaction, machine learning, robotic control, and driver emotional state monitoring. With artificial intelligence and computational power advancements, visual emotion recognition has become a prominent research area. Despite extensive research employing machine learning algorithms like convolutional neural networks (CNN), challenges remain concerning input data processing, emotion classification scope, data size, optimal CNN configurations, and performance evaluation. To address these issues, we propose a comprehensive CNN-based model for real-time detection and classification of five primary emotions: anger, happiness, neutrality, sadness, and surprise. We employ the Amsterdam Dynamic Facial Expression Set – Bath Intensity Variations (ADFES-BIV) video dataset, extracting image frames from the video samples. Image processing techniques such as histogram equalization, color conversion, cropping, and resizing are applied to the frames before labeling. The Viola-Jones algorithm is then used for face detection on the processed grayscale images. We develop and train a CNN on the processed image data, implementing dropout, batch normalization, and L2 regularization to reduce overfitting. The ideal hyperparameters are determined through trial and error, and the model's performance is evaluated. The proposed model achieves a recognition accuracy of 99.38%, with the confusion matrix, recall, precision, F1 score, and processing time further quantifying its performance characteristics. The model's generalization performance is assessed using images from the Warsaw Set of Emotional Facial Expression Pictures (WSEFEP) and Extended Cohn-Kanade Database (CK+) datasets. The results demonstrate the efficiency and usability of our proposed approach, contributing valuable insights into real-time visual emotion recognition.


artificial intelligence; convolutional neural networks; emotion recognition; human-computer interaction; machine learning


Olszanowski, M., Pochwatko, G., Kuklinski, K., Scibor-Rylski, M., Lewinski, P., & Ohme, R. K. (2015). Warsaw set of emotional facial expression pictures: a validation study of facial display photographs. Frontiers in Psychology, 5, 1516.

Ashraf, A., Gunawan, T. S., Riza, B. S., Haryanto, E. V., & Janin, Z. (2020). On the review of image and video-based depression detection using machine learning. Indonesian Journal of Electrical Engineering and Computer Science (IJEECS), 19(3), 1677-1684.

Gunawan, T. S., Ashraf, A., Riza, B. S., Haryanto, E. V., Rosnelly, R., Kartiwi, M., & Janin, Z. (2020). Development of video-based emotion recognition using deep learning with Google Colab. TELKOMNIKA, 18(5), 2463-2471.

Nezhad, Z. B., & Deihimi, M. A. (2020). Sarcasm detection in Persian. Journal of Information and Communication Technology, 20(1), 1-20.

Basavaiah, J., & Patil, C. M. (2020). Human activity detection and action recognition in videos using convolutional neural networks. Journal of Information and Communication Technology, 19(2), 157-183.

Najah, G. M. S. (2017). Emotion estimation from facial images, Atilim University, 10.13140/RG.2.2.25113.62565.

Sönmez, E. B. (2018). An automatic multilevel facial expression recognition system. Süleyman Demirel Üniversitesi Fen Bilimleri Enstitüsü Dergisi, 22(1), 160-165.

Abdulsalam, W. H., Alhamdani, R. S., & Abdullah, M. N. (2019). Facial emotion recognition from videos using deep convolutional neural networks. International Journal of Machine Learning and Computing, 9(1), 14-19.

Wingenbach, T. S., Ashwin, C., & Brosnan, M. (2016). Validation of the Amsterdam Dynamic Facial Expression Set–Bath Intensity Variations (ADFES-BIV): A set of videos expressing low, intermediate, and high intensity emotions. PloS one, 11(1), e0147112.

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.

Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., & Anbarjafari, G. (2017). Audio-visual emotion recognition in video clips. IEEE Transactions on Affective Computing, 10(1), 60-75.

Uddin, M. Z., Khaksar, W., & Torresen, J. (2017). Facial expression recognition using salient features and convolutional neural network. IEEE Access, 5, 26146-26161.

Yin, X., & Liu, X. (2017). Multi-task convolutional neural network for pose-invariant face recognition. IEEE Transactions on Image Processing, 27(2), 964-975.

Xie, S., & Hu, H. (2018). Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Transactions on Multimedia, 21(1), 211-220.

Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, 2(3), 1-8.

Pranav, E., Kamal, S., Chandran, C. S., & Supriya, M. H. (2020, March). Facial emotion recognition using deep convolutional neural network. In 2020 6th International conference on advanced computing and communication Systems (ICACCS) (pp. 317-320). IEEE.

Zhou, N., Liang, R., & Shi, W. (2021). A Lightweight Convolutional Neural Network for Real-Time Facial Expression Detection. IEEE Access, 9, 5573-5584. https://doi.org/10.1109/ACCESS.2020.3046715.

Melinte, D. O., & Vladareanu, L. (2020). Facial Expressions Recognition for Human-Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer. Sensors, 20(8), 2393. https://doi.org/10.3390/s20082393.

Fei, Z., Yang, E., Li, D. D.-U., Butler, S., Ijomah, W., Li, X., & Zhou, H. (2020). Deep convolution network based emotion analysis towards mental health care. Neurocomputing, 388, 212-227. https://doi.org/10.1016/j.neucom.2020.01.034.

Akhand MAH, Roy S, Siddique N, Kamal MAS, Shimamura T. Facial Emotion Recognition Using Transfer Learning in the Deep CNN. Electronics. 2021; 10(9):1036. https://doi.org/10.3390/electronics10091036.

Kusuma, G. P., & Lim, A. P. (2020). Emotion recognition on FER-2013 face images using fine-tuned VGG-16. Advances in Science, Technology and Engineering Systems Journal, 5(6), 315-322.

Badrulhisham, N. A. S., & Abu Mangshor, N. N. (2021). Developing a mobile-based application for emotion recognition using facial expression in real-time. Journal of Physics: Conference Series, 1962(1), 012040. https://doi.org/10.1088/1742-6596/1962/1/012040.

Cai, Y., Zheng, W., Zhang, T., Li, Q., Cui, Z., & Ye, J. (2016). Video based emotion recognition using CNN and BRNN. Chinese conference on pattern recognition,

Hum, Y. C., Lai, K. W., & Mohamad Salim, M. I. (2014). Multiobjectives bihistogram equalization for image contrast enhancement. Complexity, 20(2), 22-36.

Naik, S. K., & Murthy, C. (2003). Hue-preserving color image enhancement without gamut problem. IEEE Transactions on image processing, 12(12), 1591-1598.

Al-Sumaidaee, S., Dlay, S., Woo, W., & Chambers, J. (2015). Facial expression recognition using local Gabor gradient code-horizontal diagonal descriptor.

Boubenna, H., & Lee, D. (2018). Image-based emotion recognition using evolutionary algorithms. Biologically inspired cognitive architectures, 24, 70-76.

Zhou, D., Shen, X., & Dong, W. (2012). Image zooming using directional cubic convolution interpolation. IET image processing, 6(6), 627-634.

Valueva, M. V., Nagornov, N., Lyakhov, P. A., Valuev, G. V., & Chervyakov, N. I. (2020). Application of the residue number system to reduce hardware costs of the convolutional neural network implementation. Mathematics and Computers in Simulation, 177, 232-243.

Full Text: PDF


  • There are currently no refbacks.


Indonesian Journal of Electrical Engineering and Informatics (IJEEI)
ISSN 2089-3272

Creative Commons Licence

This work is licensed under a Creative Commons Attribution 4.0 International License.

web analytics
View IJEEI Stats