An Improved Time-Frequency Based Deep Learning Algorithm for Speech Emotion Recognition

Emotion identification can be very useful in diverse applications including medical diagnosis, social interaction, marketing, etc. Nevertheless, emotions are complex and still pose numerous challenges. In this work, a new framework is developed for improving state-of-art works in identifying emotion types from speech data. We improve the representation of input speech data by adding new features to the time-frequency (TF) spectrogram of audio segments and show that such a modified TF image-like model has a major impact on accurately identifying emotions. For each audio segment, a sequence of spectrogram images was generated, and three additional spectral features were estimated, namely spectral centroid, pitch frequency and spectral roll off. These features were superimposed as layers to the spectrogram images. The new improved spectrogram images, with a proper augmentation step, were then fed to different types of deep networks. The best result in detection accuracy was obtained using the VGG network with a score of 92.05% outperforming state-of-the-art by more than 6% on the average.