第二節介紹了關于多模態對話分析的文獻回顧。在第三節中，詳細介紹了記錄我們自己的數據集及其規格的協議。第四節詳細介紹了多模態方法，以進行級別交互分類。
在駕駛艙乘客互動分析的背景下，這種調查是邊緣化的，因此，仍然是一個科學挑戰。
III.車輛中的多模態對話語料庫
由于受試者不是真正的演員，我們觀察到兩個階段的過渡。第一階段是設置階段：在每個場景的前30秒，受試者不能堅持或斷然拒絕導致 "糟糕的演技"。第二個是在最后：受試者的靈感耗盡，在每個場景的最后20秒內造成呼吸短促。
IV.多模態分析
其中y?是模型對C類的輸出分數。
V.評估和相關分析
另一個導致錯誤分類的問題是一些受試者的糟糕演技。例如，一些受試者在拒絕場景中的表演階段中笑場。或者一個受試者在回頭看后排乘客時表現得很反常。
VI.結論和未來工作
參考文獻：
[1]J. Devlin, M.-W.? Chang,? K. Lee,? and? K. Toutanova,? “BERT: Pre-training of deep bidirectional transformers for language understand- ing,” in Proceedings of the 2019 Conference of the North American Chapter? of? the? Association for? Computational? Linguistics:? Human Language? Technologies,? Volume 1 (Long and Short Papers).? ? Min- neapolis, Minnesota: Association for Computational Linguistics, June?2019, pp. 4171–4186.
[2]L. Martin, B. Muller, P. J. Ortiz Surez, Y. Dupont, L. Romary, E. de la?Clergerie, D. Seddah, and B. Sagot, “CamemBERT: a Tasty French Language Model,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Online: Association for Computational Linguistics, July 2020, pp. 7203–7219.
[3]? B. Santra, P. Anusha, and P. Goyal, “Hierarchical Transformer for Task?Oriented Dialog Systems,” arXiv:2011.08067 [cs], Mar. 2021.
[4]? D.? Chen,? H.? Chen,? Y.? Yang,? A.? Lin,? and? Z.? Yu,? “Action-Based
Conversations Dataset: A Corpus for Building More In-Depth Task- Oriented Dialogue Systems,” arXiv:2104.00783 [cs], Apr. 2021.
[5]? D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning
spatiotemporal features with 3d convolutional networks,” in The IEEE
International Conference on Computer Vision (ICCV), Dec. 2015.
[6]? K. Hara, H. Kataoka, and Y. Satoh, “Learning spatio-temporal features
with 3d residual networks for action recognition,” arXiv:1708.07632 [cs], 2017.
[7]? F.? Eyben,? M. Wllmer,? and? B.? Schuller,? “opensmile –? the? munich
versatile and fast open-source audio feature extractor,” in ACM Mul- timedia, 01 2010, pp. 1459– 1462.
[8]? Y. Yu, X. Si, C. Hu, and J. Zhang, “A Review of Recurrent Neural
Networks: LSTM Cells and Network Architectures,” Neural Compu- tation, vol. 31, no. 7, pp. 1235– 1270, July 2019.
[9] S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L.-P.
Morency,? “Context-dependent? sentiment? analysis? in? user-generated videos,”? ?in? ?Proceedings? ?of? ?the? ?55th? ?Annual? ?Meeting? ?of? ?the Association for Computational Linguistics (Volume 1: Long Papers). Association? ?for? ?Computational? ?Linguistics,? ?2017,? pp.? ?873–883. [Online]. Available: http://aclweb.org/anthology/P17-1081
[10]? M. G. Huddar, S. S. Sannakki, and V. S. Rajpurohit, “An ensemble
approach to utterance level multimodal sentiment analysis,” in 2018 International? Conference? on? Computational? Techniques,? Electronics and Mechanical Systems (CTEMS), 2018, pp. 145– 150.
[11]? A. Agarwal, A. Yadav, and D. K. Vishwakarma, “Multimodal senti-
ment analysis via rnn variants,” in 2019 IEEE International Conference on Big Data,? Cloud? Computing, Data? Science Engineering? (BCD),
2019, pp. 19–23.
[12]? R. Li, C. Lin, M. Collinson, X. Li, and G. Chen, “A dual-attention
hierarchical recurrent neural network for dialogue act classi?cation,” in Proceedings? of? the? 23rd? Conference? on? Computational? Natural Language Learning (CoNLL).? ? Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 383–392.
[13]? Y. Luan, Y. Ji, and M. Ostendorf, “Lstm based conversation models,”2016.
[14]? E.? Cambria,? D.? Hazarika,? S.? Poria,? A.? Hussain,? and? R.? B.? V.?Subramaanyam,? ?“Benchmarking? ?multimodal? ?sentiment? ?analysis,” arXiv:1707.09538 [cs], 2017.
[15]? W.? Li,? W.? Shao,? S.? Ji,? and? E.? Cambria,? “BiERU:? Bidirectional?Emotional? Recurrent? Unit? for? Conversational? Sentiment? Analysis,” arXiv:2006.00492 [cs], Feb. 2021.
[16]? N. Majumder, S. Poria, D. Hazarika, R. Mihalcea, A. Gelbukh, and?E.? Cambria,? “Dialoguernn:? An? attentive rnn? for? emotion? detection in conversations,” Proceedings of the AAAI Conference on Arti?cial Intelligence, vol. 33, pp. 6818–6825, July 2019.
[17]? A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency, “Mosi: Multimodal?corpus? of? sentiment? intensity? and? subjectivity? analysis? in? online opinion videos,” 2016.
[18]? Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchi-cal attention networks for document classi?cation,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.? ?San Diego,? California:? Association? for? Computational Linguistics, June 2016, pp. 1480– 1489.
[19]? Q.? Portes.,? J.? Carvalho.,? J.? Pinquier.,? and? F.? Lerasle.,? “Multimodal
neural? network? for? sentiment? analysis? in? embedded? systems,”? in?Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics? Theory and Applications-Volume 5: VISAPP,, INSTICC.? ? SciTePress, 2021, pp. 387–398.
[20]? B. Bigot, J. Pinquier, I. Ferran′e,and R. Andr-Obrecht, “Looking for?relevant features for speaker role recognition (regular paper),” in IN- TERSPEECH, Makuhari, Japan, 26/09/10-30/09/10.? http://www.isca- speech.org/: International Speech Communication Association (ISCA), 2010, pp. 1057– 1060.
[21]? K.? S. Jones,? “A? statistical? interpretation? of term? speci?city? and? its?application in retrieval,” Journal of Documentation, vol. 28, pp. 11– 21, 1972.
[22]? R. Ranjan, V. M. Patel, and R. Chellappa, “Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation,? and gender recognition,” IEEE? Transactions? on? Pattern Analysis and Machine Intelligence, vol. 41, no. 1, pp. 121– 135, 2019.
