Skip navigation
Please use this identifier to cite or link to this item: http://repository.iitr.ac.in/handle/123456789/21861
Title: Towards the explainability of multimodal speech emotion recognition
Authors: Kumar P.
Kaushik V.
Raman, Balasubramanian
Published in: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
Abstract: In this paper, a multimodal speech emotion recognition system has been developed, and a novel technique to explain its predictions has been proposed. The audio and textual features are extracted separately using attention-based Gated Recurrent Unit (GRU) and pre-trained Bidirectional Encoder Representations from Transformers (BERT), respectively. Then they are concatenated and used to predict the final emotion class. The weighted and unweighted emotion recognition accuracy of 71.7% and 75.0% has been achieved on Emotional Dyadic Motion Capture (IEMOCAP) dataset containing speech utterances and corresponding text transcripts. The training and predictions of network layers have been analyzed qualitatively through emotion embedding plots and quantitatively by analyzing the intersection matrices for various emotion classes’ embeddings. Copyright © 2021 ISCA.
Citation: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH (2021), 4: 2927-2931
URI: https://doi.org/10.21437/Interspeech.2021-1718
http://repository.iitr.ac.in/handle/123456789/21861
Issue Date: 2021
Publisher: International Speech Communication Association
Keywords: Deep network explainability
Embedding plot
Intersection matrix
Multimodal emotion recognition
ISBN: 9.78171E+12
ISSN: 2308457X
Author Scopus IDs: 57200340329
57223805924
23135470700
Author Affiliations: Kumar, P., Computer Science and Engg. Dept., Indian Institute of Technology, Roorkee, 247667, India
Kaushik, V., Mechanical Engg. Dept., Indian Institute of Technology, Kanpur, 208016, India
Raman, B., Computer Science and Engg. Dept., Indian Institute of Technology, Roorkee, 247667, India
Corresponding Author: Kumar, P.; Computer Science and Engg. Dept., India; email: pkumar99@cs.iitr.ac.in Kaushik, V.; Mechanical Engg. Dept., India; email: kvishesh@iitk.ac.in
Appears in Collections:Conference Publications [CS]

Files in This Item:
There are no files associated with this item.
Show full item record


Items in Repository are protected by copyright, with all rights reserved, unless otherwise indicated.