Research ArticleNo Access

From Intricacy to Conciseness: A Progressive Transfer Strategy for EEG-Based Cross-Subject Emotion Recognition

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300401, P. R. China

E-mail Address: [email protected]

Search for more papers by this author

Lingyue Wang

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300401, P. R. China

E-mail Address: [email protected]

Search for more papers by this author

Miaomiao Guo

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300401, P. R. China

E-mail Address: [email protected]

Corresponding author.

Search for more papers by this author

Guizhi Xu

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300401, P. R. China

E-mail Address: [email protected]

Corresponding author.

Search for more papers by this author

Lei Guo

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300401, P. R. China

E-mail Address: [email protected]

Search for more papers by this author

, and

Ying Li

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Hebei University of Technology, Tianjin 300401, P. R. China

E-mail Address: [email protected]

Search for more papers by this author

https://doi.org/10.1142/S0129065722500058Cited by:13 (Source: Crossref)

Abstract

Emotion plays a significant role in human daily activities, and it can be effectively recognized from EEG signals. However, individual variability limits the generalization of emotion classifiers across subjects. Domain adaptation (DA) is a reliable method to solve the issue. Due to the nonstationarity of EEG, the inferior-quality source domain data bring negative transfer in DA procedures. To solve this problem, an auto-augmentation joint distribution adaptation (AA-JDA) method and a burden-lightened and source-preferred JDA (BLSP-JDA) approach are proposed in this paper. The methods are based on a novel transfer idea, learning the specific knowledge of the target domain from the samples that are appropriate for transfer, which reduces the difficulty of transfer between two domains. On multiple emotion databases, our model shows state-of-the-art performance.

Keywords:

References

1. C. Ieracitano, F. C. Morabito, A. Hussain and N. Mammone , A hybrid-domain deep learning-based BCI for discriminating hand motion planning from EEG sources, Int. J. Neural Syst. 31(9) (2021) 2150038. Link, Web of Science, Google Scholar
2. J. Jin, H. Fang, I. Daly, R. Xiao, Y. Miao, X. Wang and A. Cichocki , Optimization of model training based on iterative minimum covariance determinant in motor-imagery BCI, Int. J. Neural Syst. 31(7) (2021) 2150030. Link, Web of Science, Google Scholar
3. J. Shin and C. Im , Performance prediction for a near-infrared spectroscopy-brain-computer interface using resting-state functional connectivity of the prefrontal cortex, Int. J. Neural Syst. 28(10) (2018) 1850023. Link, Web of Science, Google Scholar
4. J. Sorinas, J. C. Fernandez-Troyano, J. M. Ferrandez and E. Fernandez , Cortical asymmetries and connectivity patterns in the valence dimension of the emotional brain, Int. J. Neural Syst. 30(5) (2020) 81–96. Link, Web of Science, Google Scholar
5. A. Burns, H. Adeli and J. A. Buford , Brain-computer interface after nervous system injury, Neuroscientist 20(6) (2014) 639–651. Crossref, Medline, Web of Science, Google Scholar
6. J. R. Wolpaw, N. Birbaumer, D. J. Mcfarland, G. Pfurtscheller and T. M. Vaughan , Brain-computer interfaces for communication and control, Commun. ACM. 54(5) (2002) 60. Google Scholar
7. E. Frant, I. Ispas, V. Dragomir, M. Dascalu, E. Zoltan and I. C. Stoica , Voice-based emotion recognition with convolutional neural networks for companion robots, Rom. J. Inf. Sci. Tech. 20(3) (2017) 222–240. Web of Science, Google Scholar
8. P. Hajek, A. Barushka and M. Munk , Neural networks with emotion associations, topic modeling, and supervised term weighting for sentiment analysis, Int. J. Neural Syst. 31(10) (2021) 2150013. Link, Web of Science, Google Scholar
9. G. Alexandridis, J. Aliprantis, K. Michalakis, K. Korovesis, P. Polydoras, P. Tsantilas and G. Caridakis , A knowledge-based deep learning architecture for aspect-based sentiment analysis, Int. J. Neural Syst. 31(10) (2021) 2150046. Link, Web of Science, Google Scholar
10. L. Zhang and D. Tjondronegoro , Facial expression recognition using facial movement features, IEEE Trans. Affect. Comput. 2(4) (2011) 219–229. Crossref, Web of Science, Google Scholar
11. C. Mhl, B. Allison, A. Nijholt and G. Chanel , A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges, Brain-Comput. Interfaces 1(2) (2014) 66–84. Crossref, Google Scholar
12. Y. Tonoyan, D. Looney, D. P. Mandic and M. M. Van Hulle , Discriminating multiple emotional states from EEG using a data-adaptive, multi-scale information-theoretic approach, Int. J. Neural Syst. 26(2) (2016) 1650005. Link, Web of Science, Google Scholar
13. W. L. Zheng and B. L. Lu , Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks, IEEE Trans. Auton. Mental Develop. 7(3) (2015) 162–175. Crossref, Google Scholar
14. M. Val-Calvo, J. R. Alvarez-Sanchez, J. M. Ferrandez, A. Daz-Morcillo and E. Fernandez-Jover , Real-time multi-modal estimation of dynamically evoked emotions using, EEG, heart rate, and galvanic skin response, Int. J. Neural Syst. 30(4) (2020) 2050013. Link, Web of Science, Google Scholar
15. H. S. Nogay and H. Adeli , Machine learning for the diagnosis of autism spectrum disorder using brain imaging, Rev. Neurosci. 31(8) (2020) 825–841. Crossref, Web of Science, Google Scholar
16. M. H. Rafiei, K. Kelly, A. Borstad, H. Adeli and L. Gauthier , Predicting improved daily use of the more effected arm post-stroke following constraint-induced movement, Phys. Therapy 99(12) (2019) 1667–1678. Crossref, Medline, Web of Science, Google Scholar
17. W. L. Zheng, J. Y. Zhu and B. L. Lu , Identifying stable patterns over time for emotion recognition from EEG, IEEE Trans. Affect. Comput. 10(3) (2017) 417–429. Crossref, Web of Science, Google Scholar
18. X. Q. H, W. L. Zheng and B. L. Lu , Driving fatigue detection with fusion of EEG and forehead EOG, in Proc. Int. Conf. on Neural Networks (IJCNN) (Vancouver, BC, Canada, 2016), pp. 897–904. Google Scholar
19. R. N. Duan, J. Y. Zhu and B. L. Lu , Differential entropy feature for EEG-based emotion classification, in Proc. 6th Int. IEEE EMBS Conf. Neural Encoder (NER) (San Diego, California, USA, 2013), pp. 81–84. Crossref, Google Scholar
20. S. Koelstra, C. Muhl, S. Mohammad, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt and I. Patras , Deap: A database for emotion analysis; using physiological signals, IEEE Trans. Affect. Comput. 3(1) (2012) 18–31. Crossref, Web of Science, Google Scholar
21. S. Katsigiannis and N. Ramzan , DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices, IEEE J. Biomed. Health Inf. 22(1) (2018) 98–107. Crossref, Medline, Web of Science, Google Scholar
22. H. Y. Xu and N. P. Konstantinos , EEG-based affect states classification using deep belief networks, in 2016 Digital Media Industry and Academic Forum (DMIAF) (Santorini, Greece, 2016), pp. 148–153. Crossref, Google Scholar
23. X. Li, Z. Zhao, D. Song, Y. Zhang, C. Niu, J. Zhang, J. Huo and J. Li , Variational autoencoder based latent factor decoding of multichannel EEG for emotion recognition, in Int. Conf. Bioinformatics and Biomedicine (BIBM) (San Diego, CA, USA, 2019), pp. 684–687. Crossref, Google Scholar
24. Y. Li, H. Kambara, Y. Koike and M. Sugiyama , Application of covariate shift adaptation techniques in brain-computer interfaces, IEEE Trans. Bio-Med. Eng. 57(6) (2010) 1318. Crossref, Medline, Web of Science, Google Scholar
25. H. Hashimoto, S. Kameda, H. Maezawa, S. Oshino, N. Tani, H. M. Khoo, T. Yanagisawa, T. Yoshimine, H. Kishima and M. Hirata , A swallowing decoder based on deep transfer learning: AlexNet classification of the intracranial electrocorticogram, Int. J. Neural Syst. 31(11) (2021) 2050056. Link, Web of Science, Google Scholar
26. M. Long, Y. Cao, J. Wang and M. I. Jordan , Learning transferable features with deep adaptation networks, in Int. Conf. Machine Learning (PMLR) (Porto, Portugal, 2015), pp. 85–105. Google Scholar
27. H. S. Nogay and H. Adeli , Detection of epileptic seizure using pre-trained deep convolutional neural network and transfer learning, Euro. Neurol. 83(6) (2020) 602–614. Crossref, Medline, Web of Science, Google Scholar
28. A. R. Sanabria, F. Zambonelli and J. Ye , Unsupervised domain adaptation in activity recognition: A GAN-based approach, IEEE Access. 9 (2021) 19421–19438. Crossref, Web of Science, Google Scholar
29. Y. M. Jin, Y. D. Luo, W. L. Zheng and B. L. Lu , EEG-based emotion recognition using domain adaptation network, in Proc. Int. Conf. Orange Tech (ICOT) (Singapore, 2017), pp. 222–225. Crossref, Google Scholar
30. T. F. Song, W. M. Zheng, P. Song and Z. Cui , EEG emotion recognition using dynamical graph convolutional neural networks, IEEE Trans. Affect. Comput. 11(3) (2018) 532–541. Crossref, Web of Science, Google Scholar
31. H. Li, Y. M. Jin, W. L. Zheng and B. L. Lu , Cross-subject emotion recognition using deep adaptation networks, in Proc. Int. Conf. Neural Information (ICONIP) (Siem Reap, Cambodia, 2018), pp. 403–413. Crossref, Google Scholar
32. Y. Luo, S. Y. Zhang, W. L. Zheng and B. L. Lu , WGAN domain adaptation for EEG-based emotion recognition, in Int. Conf. Neural Information Processing (ICONIP) (Siem Reap, Cambodia, 2018), pp. 275–286. Crossref, Google Scholar
33. B. Q. Ma, H. Li, W. L. Zheng and B. L. Lu , Reducing the subject variability of EEG signals with adversarial domain generalization, in Int. Conf. Neural Information Processing (ICONIP) (Sydney, Australia, 2019), pp. 30–42. Crossref, Google Scholar
34. J. P. Li, S. Qiu, C. D. Du, Y. X. Wang and H. G. He , Domain adaptation for EEG emotion recognition based on latent representation similarity, IEEE Trans. Cogn. Dev. Syst. 12(2) (2019) 344–353. Crossref, Web of Science, Google Scholar
35. F. Wang, W. Zhang, Z. Xu, J. Ping and H. Chu , A deep multi-source adaptation transfer network for cross-subject electroencephalogram emotion recognition, Neural. Comput. App. 33(9) (2021) 1–13. Google Scholar
36. Y. Li, W. M. Zheng, L. Wang, Y. Zong, L. Qi, Z. Cui, T. Zhang and T. Song , A novel bi-hemispheric discrepancy model for eeg emotion recognition, IEEE Tran. Cogn. Dev. Syst. 13(2) (2020) 354–367. Crossref, Web of Science, Google Scholar
37. G. C. Bao, N. Zhuang, B. Yan, J. Shu, L. Y. Wang, Y. Zeng and Z. C. Shen , Two-level domain adaptation neural network for EEG-based emotion recognition, Front. Hum. Neurosci. 14 (2021) 605246. Crossref, Medline, Web of Science, Google Scholar
38. H. Chen, M. Jin, Z. Li, C. H. Fan, J. P. Li and H. He, MS-MDA: Multisource marginal distribution adaptation for cross-subject and cross-session EEG emotion recognition, 2107(07740) (2021). Google Scholar
39. Y. X. Wang, S. Qiu, X. L. Ma and H. G. He , A prototype-based SPD matrix network for domain adaptation EEG emotion recognition, Pattern Recognition 110 (2021) 107626. Crossref, Web of Science, Google Scholar
40. S. Aydin, S. Demirtas, K. Ates and M. A. Tunga , Emotion recognition with eigen features of frequency band activities embedded in induced brain oscillations mediated by affective pictures, Int. J. Neural Syst. 10(3) (2017) 417–429. Google Scholar
41. J. Li, S. Qiu, Y. Y. Shen, C. L. Liu and H. He , Multisource transfer learning for cross-subject EEG emotion recognition, IEEE Trans. Cybernet. 50(7) (2019) 3281–3293. Medline, Web of Science, Google Scholar
42. X. Du, C. Ma, G. Zhang, J. Li and H. Wang , An efficient LSTM network for emotion recognition from multichannel EEG signals, IEEE Trans. Affect. Comput. 99(2020) 1–12. Google Scholar
43. G. Wilson, J. R. Doppa and D. J. Cook , Multi-source deep domain adaptation with weak supervision for time-series sensor data, in Proc. 26th ACM SIGKDD Int. Conf. Knowledge Discover Data Mining (San Diego, California, USA, 2020), pp. 1768–1778. Crossref, Google Scholar
44. J. Mu, W. Qiu, G. D. Hager and A. L. Yuille , Learning from synthetic animals, in 2020 IEEE/CVF Conf. Computer Visual and Pattern Recognition (CVPR) (Seattle, WA, USA, 2020), pp. 12386–12395. Crossref, Google Scholar
45. C. Li and H. L. Gim , From synthetic to real: Unsupervised domain adaptation for animal pose estimation, in Proc. IEEE/CVF Conf. Computer Visual and Pattern Recognition (CVPR) (Vurtual conference, 2021), pp. 1482–1491. Crossref, Google Scholar
46. J. Pan, I. W. Tsang, J. T. Kwok and Q. Yang , Domain adaptation via transfer component analysis, IEEE Trans. Neural Networks 22(2) (2011) 199–210. Crossref, Medline, Web of Science, Google Scholar
47. M. Long, J. Wang, G. Ding, J. Sun and P. S. Yu , Transfer feature learning with joint distribution adaptation, in Proc. Int. Conf. Computer Visual (ICCV) (Sydney, Australia, 2013), pp. 1414–4854. Crossref, Google Scholar
48. P. Hausser, T. Frerix, A. Mordvintsev and D. Cremers , Associative domain adaptation, in Proc. Int. Conf. Computur Visual (ICCV), Vol. 2 (Venice, Italy, 2017), pp. 2784–2792. Crossref, Google Scholar
49. Z. R. Lan, O. Sourina, L. P. Wang, R. Scherer and G. R. Mller-Putz , Domain adaptation techniques for EEG-based emotion recognition: A comparative study on two public datasets, IEEE Trans. Cogn. Dev. Syst. 11(1) (2018) 85–94. Crossref, Web of Science, Google Scholar
50. G. E. Hinton and R. S. Ruslan , Reducing the dimensionality of data with neural networks, Science 313(5786) (2006) 504–507. Crossref, Medline, Web of Science, Google Scholar
51. S. Connor and K. Taghi , A survey on image data augmentation for deep learning, J. Big Data 6(1) (2019) 1–48. Google Scholar
52. L. Perez and J. Wang, The effectiveness of data augmentation in image classification using deep learning, arXiv:1712.04621. Google Scholar
53. E. Lashgari, D. Liang and U. Maoz , Data augmentation for deep-learning-based electroencephalography, J. Neurosci. Meth. 346 (2020) 108885. Crossref, Medline, Web of Science, Google Scholar
54. F. Lotte , Signal processing approaches to minimize or suppress calibration time in oscillatory activity-based brain-computer interfaces, Proc. IEEE 103(6) (2015) 871–890. Crossref, Web of Science, Google Scholar
55. F. Wang, S. H. Zhong, J. F. Peng, J. M. Jiang and Y Liu , Data augmentation for EEG-based emotion recognition with deep convolutional neural networks, in Int. Conf. Multimedia Modeling (MMM) (Bangkok, Thailand, 2018), pp. 82–93. Crossref, Google Scholar
56. Y. Luo, L. Z. Zhu, Z. Y. Wan and B. L. Lu , Data augmentation for enhancing EEG-based emotion recognition with deep generative models, J. Neural Eng. 17(5) (2020) 056021. Crossref, Medline, Web of Science, Google Scholar
57. H. Bcklund, A. Hedblom and N. NeijmanBcklund , A density-based spatial clustering of application with noise, Data Mining TNM033 (2011), pp. 11–30. Google Scholar
58. V. Gupta, M. D. Chopda, R. B. Pachori , Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals, IEEE Sensors J. 19(6) (2019) 2266–2274. Crossref, Web of Science, Google Scholar
59. Y. Wang, S. Qiu, C. Zhao, W. Yang, J. Li, X. Ma and H. G. He , EEG-based emotion recognition with prototype-based data representation, in Int. Conf. IEEE Engineering Medical and Biological Society (EMBC) (Berlin, Germany, 2019), pp. 684–689. Crossref, Google Scholar
60. Y. Wang, S. Qiu, J. Li, X. Ma, Z. Liang, H. Li and H. G. He , EEG-based emotion recognition with similarity learning network, in Int. Conf. IEEE Engineering in Medical and Biological Society (EMBC) (Berlin, Germany, 2019), pp. 1209–1212. Crossref, Google Scholar
61. M. A. Rahman, M. F. Hossain, M. Hossain and R. Ahmmed , Employing PCA and $t$ -statistical approach for feature extraction and classification of emotion from multichannel EEG signal, Egypt. Inform. J. 21(1) (2020) 23–35. Crossref, Web of Science, Google Scholar
62. W. L. Zheng, J. Zhu, Y. Peng and B. L. Lu , EEG-based emotion classification using deep belief networks, in 2014 IEEE Int. Conf. Multimedia and Expo (ICME), (Chengdu, China, 2014), pp. 1–6. Crossref, Google Scholar
63. L. M. Zhao, Y. Xu and B. L. Lu , Plug-and-play domain adaptation for cross-subject EEG-based emotion recognition, in Proc. 35th AAAI Conf. Artificial Intelligence (2021), follows. https://www. aaai.org/AAAI21Papers/AAAI-7688.ZhaoL.pdf Crossref, Google Scholar
64. J. Sorinas, M. D. Grima, J. M. Ferrandez and E. Fernandez , Identifying suitable brain regions and trial size segmentation for positive/negative emotion recognition, Int. J. Neural Syst. 29(2) (2019) 1850044. Link, Web of Science, Google Scholar
65. J. Li, Z. Zhang and H. He , Hierarchical convolutional neural networks for EEG-based emotion recognition, Cogn. Comput. 10(2) (2018) 368–380. Crossref, Web of Science, Google Scholar
66. L. J. P. V. D. Maaten and G. E. Hinton , Visualizing high-dimensional data using $t$ -SNE, J. Mach. Learn. Res. 9 (2008) 2579–2605. Web of Science, Google Scholar
67. G. G. Molina, T. Tsoneva and A. Nijholt , Emotional brain-computer interfaces, in Proc. IEEE 3rd Int. Conf. Affective Computing and Intelligence International and Workshops (ACII) (Amsterdam, Holland, 2009), pp. 1–9. Crossref, Google Scholar
68. W. Samek, F. C. Meinecke and K. R. Muller , Transferring subspaces between subjects in brain-computer interfacing, IEEE Trans. BME. 60(8) (2013) 2289–2298. Crossref, Medline, Web of Science, Google Scholar
69. H. Morioka, A. Kanemura, J. I. Hirayama, M. Shikauchi, T. Ogaw, S. Ikeda, M. Kawanabe and S. Ishii , Learning a common dictionary for subject-transferd ecoding with resting calibration, NeuroImage 111 (2015) 167–178. Crossref, Medline, Web of Science, Google Scholar
70. D. Wu, Y. Xu and B. L. Lu , Transfer learning for EEG-based brain-computer interfaces: A review of progress made since 2016, IEEE Trans. Cogn. Dev. Syst. 99 (2020) 1–16. Google Scholar
71. W. Li, Z. Zhang and A. G. Song , Physiological-signal-based emotion recognition: An odyssey from methodology to philosophy, Measurement 172(4) (2020) 108747. Google Scholar
72. Y. Yao and G. Doretto , Boosting for transfer learning with multiple sources, in Proc. IEEE Conf. Computing Visual Pattern Recognition (CVPR) (San Francisco, CA, USA, 2010), pp. 1855–1862. Crossref, Google Scholar
73. F. Fang, T. Potter, T. Nguyen and Y. C. Zhang , Dynamic reorganization of the cortical functional brain network in affective processing and cognitive reappraisal, Int. J. Neural Syst. 30(10) (2020) 2050051. Link, Web of Science, Google Scholar
74. A. Mehran and H. Adeli , Enhanced probabilistic neural network with local decision circles: A robust classifier, Integrated Computer-Aided Engineering 17(3) (2010) 197–210. Crossref, Web of Science, Google Scholar
75. D. R. Pereira, M. A. Piteri, A. N. Souza, J. Papa and H. Adeli , FEMa: A finite element machine for fast learning, Neural Comput. Appl. 32(10) 2020 6393–6404. Crossref, Web of Science, Google Scholar
76. K. Md. R. Alam, N. Siddique and H. Adeli , A dynamic ensemble learning algorithm for neural networks, Neural Comput. Appl. 32(10) (2020) 8675–8690. Crossref, Web of Science, Google Scholar
77. M. H. Rafiei and H. Adeli , A new neural dynamic classification algorithm, IEEE Trans. Neural Networks Learn. Syst. 28(12) (2017) 3074–3083. Crossref, Medline, Web of Science, Google Scholar