Advanced Multimodal Emotion Recognition for Javanese Language Using Deep Learning

Abstract

This research utilizes multimodal audio and video datasets to establish a robust Javanese language emotion recognition system, even though there have been few significant developments in emotion recognition systems for Javanese. The investigation investigates three models to optimize the extraction of emotional characteristics: the Spectrogram-Image Model (Model 1), which optimizes the extraction of emotional characteristics by converting audio inputs into spectrogram images and combining them with facial images for emotion labeling; the Convolutional-MFCC Model (Model 2), which improves feature extraction by utilizing convolution techniques for images and Mel-frequency cepstral coefficients for audio; and the Multimodal Feature-Extraction Model (Model 3), which processes video and audio features separately before integrating their recognition results. Comparative testing reveals that the Multimodal Feature-Extraction Model achieves the highest accuracy at 93%, followed by the Convolutional-MFCC Model at 85% and the Spectrogram-Image Model at 71%. According to the results, effective multimodal integration, particularly through separate feature extraction, significantly improves emotion recognition accuracy. This research enhances the effectiveness of communication systems and provides deeper insights into Javanese emotional expressions, with potential applications in human-computer interaction, healthcare, and cultural studies. It also contributes to the development of sophisticated emotion recognition technologies.