Effect of Action Units, Viewpoint and Immersion on Emotion Recognition Using Dynamic Virtual Faces
Abstract
Facial affect recognition is a critical skill in human interactions that is often impaired in psychiatric disorders. To address this challenge, tests have been developed to measure and train this skill. Recently, virtual human (VH) and virtual reality (VR) technologies have emerged as novel tools for this purpose. This study investigates the unique contributions of different factors in the communication and perception of emotions conveyed by VHs. Specifically, it examines the effects of the use of action units (AUs) in virtual faces, the positioning of the VH (frontal or mid-profile), and the level of immersion in the VR environment (desktop screen versus immersive VR). Thirty-six healthy subjects participated in each condition. Dynamic virtual faces (DVFs), VHs with facial animations, were used to represent the six basic emotions and the neutral expression. The results highlight the important role of the accurate implementation of AUs in virtual faces for emotion recognition. Furthermore, it is observed that frontal views outperform mid-profile views in both test conditions, while immersive VR shows a slight improvement in emotion recognition. This study provides novel insights into the influence of these factors on emotion perception and advances the understanding and application of these technologies for effective facial emotion recognition training.
References
- 1. ,
A framework for recognizing and regulating emotions in the elderly , in Ambient Assisted Living and Daily Activities, eds. L. Pecchia, L. L. Chen, C. Nugent and J. Bravo (Springer International Publishing, Cham, 2014), pp. 320–327. Crossref, Google Scholar - 2. , Lietome: An ensemble approach for deception detection from facial cues, Int. J. Neural Syst. 31(2) (2021) 2050068. Link, Web of Science, Google Scholar
- 3. , Behavioral activity recognition based on gaze ethograms, Int. J. Neural Syst. 30(7) (2020) 2050025. Link, Web of Science, Google Scholar
- 4. , Blocking facial mimicry affects recognition of facial and body expressions, PLoS One 15(2) (2020) e0229364. Crossref, Medline, Web of Science, Google Scholar
- 5. , Impaired face recognition is associated with social inhibition, Psychiatr. Res. 236 (2016) 53–57. Crossref, Medline, Web of Science, Google Scholar
- 6. ,
Facial expression recognition from webcam based on active shape models and support vector machines , in Ambient Assisted Living and Daily Activities, eds. L. Pecchia, L. L. Chen, C. Nugent and J. Bravo (Springer International Publishing, Cham, 2014), pp. 147–154. Crossref, Google Scholar - 7. , Facial expression recognition in ageing adults: From lab to ambient assisted living, J. Ambient Intell. Humaniz. Comput. 8 (2017) 567–578. Crossref, Web of Science, Google Scholar
- 8. , Dynamic reorganization of the cortical functional brain network in affective processing and cognitive reappraisal, Int. J. Neural Syst. 30(10) (2020) 2050051. Link, Web of Science, Google Scholar
- 9. , Discriminative analysis of symptom severity and ultra-high risk of schizophrenia using intrinsic functional connectivity, Int. J. Neural Syst. 30(9) (2020) 2050047. Link, Web of Science, Google Scholar
- 10. , Automated eeg-based screening of depression using deep convolutional neural network, Comput. Methods Programs Biomed. 161 (2018) 103–113. Crossref, Medline, Web of Science, Google Scholar
- 11. , Spatiotemporal analysis of relative convergence of EEGs reveals differences between brain dynamics of depressive women and men, Clin. EEG Neurosci. 44(3) (2013) 175–181. Crossref, Medline, Web of Science, Google Scholar
- 12. , Computer-aided diagnosis of depression using EEG signals, Eur. Neurol. 73(5–6) (2015) 329–336. Crossref, Medline, Web of Science, Google Scholar
- 13. , A novel depression diagnosis index using nonlinear features in EEG signals, Eur. Neurol. 74(1–2) (2015) 79–83. Crossref, Medline, Web of Science, Google Scholar
- 14. , Fractality analysis of frontal brain in major depressive disorder, Int. J. Psychophysiol. 85(2) (2012) 206–211. Crossref, Medline, Web of Science, Google Scholar
- 15. , Brain functional connectivity patterns for emotional state classification in Parkinsons disease patients without dementia, Behav. Brain Res. 298 (2016) 248–260. Crossref, Medline, Web of Science, Google Scholar
- 16. , Impact of machine learning pipeline choices in autism prediction from functional connectivity data, Int. J. Neural Syst. 31(4) (2021) 2150009. Link, Web of Science, Google Scholar
- 17. , An adaptive conjugate gradient learning algorithm for efficient training of neural networks, Appl. Math. Comput. 62(1) (1994) 81–102. Crossref, Web of Science, Google Scholar
- 18. , Machine Learning: Neural Networks, Genetic Algorithms, and Fuzzy Systems (John Wiley & Sons, 1994). Google Scholar
- 19. , Parallel backpropagation learning algorithms on cray y-mp8/864 supercomputer, Neurocomputing 5(6) (1993) 287–302. Crossref, Web of Science, Google Scholar
- 20. , A parallel genetic/neural network learning algorithm for MIMD shared memory machines, IEEE Trans. Neural Netw. 5(6) (1994) 900–909. Crossref, Medline, Web of Science, Google Scholar
- 21. , Emotion classification from EEG with a low-cost BCI versus a high-end equipment, Int. J. Neural Syst. 32(10) (2022) 2250041. Link, Web of Science, Google Scholar
- 22. , Evaluation of brain functional connectivity from electroencephalographic signals under different emotional states, Int. J. Neural Syst. 32(10) (2022) 2250026. Link, Web of Science, Google Scholar
- 23. , Deep learning methods for multi-channel EEG-based emotion recognition, Int. J. Neural Syst. 32(5) (2022) 2250021. Link, Web of Science, Google Scholar
- 24. , From intricacy to conciseness: A progressive transfer strategy for EEG-based cross-subject emotion recognition, Int. J. Neural Syst. 32(3) (2022) 2250005. Link, Web of Science, Google Scholar
- 25. , A hybrid time-distributed deep neural architecture for speech emotion recognition, Int. J. Neural Syst. 32(6) (2022) 2250024. Link, Web of Science, Google Scholar
- 26. , Neural networks with emotion associations, topic modeling and supervised term weighting for sentiment analysis, Int. J. Neural Syst. 31(10) (2021) 2150013. Link, Web of Science, Google Scholar
- 27. , What is meant by calling emotions basic, Emot. Rev. 3(4) (2011) 364–370. Crossref, Google Scholar
- 28. , Facial Action Coding System: A Technique for the Measurement of Facial Movement (Consulting Psychologists Press, Palo Alto, CA, 1978). Google Scholar
- 29. , Holistic gaze strategy to categorize facial expression of varying intensities, PLoS One 7(8) (2012) e42585. Crossref, Medline, Web of Science, Google Scholar
- 30. , Face in profile view reduces perceived facial expression intensity: An eye-tracking study, Acta Psychol. 155 (2015) 19–28. Crossref, Medline, Web of Science, Google Scholar
- 31. , Emotion recognition of facial expressions presented in profile, Psychol. Rep. 125 (2022) 2623–2635. Crossref, Medline, Web of Science, Google Scholar
- 32. , Confusion of fear and surprise: A test of the perceptual-attentional limitation hypothesis with eye movement monitoring, Cogn. Emot. 28 (2014) 1214–1222. Crossref, Medline, Web of Science, Google Scholar
- 33. , Distinction between fear and surprise: An interpretation-independent test of the perceptual-attentional limitation hypothesis, Soc. Neurosci. 12 (2017) 751–768. Medline, Web of Science, Google Scholar
- 34. , Differences in facial expressions of four universal emotions, Psychiatr. Res. 128 (2004) 235–244. Crossref, Medline, Web of Science, Google Scholar
- 35. , The spatial distribution of eye movements predicts the (false) recognition of emotional facial expressions, PLoS One 16 (2021) 1–24. Crossref, Web of Science, Google Scholar
- 36. , Hemiface differences in visual exploration patterns when judging the authenticity of facial expressions, Front. Psychol. 8 (2018) 2332. Crossref, Medline, Web of Science, Google Scholar
- 37. , Acceptance and use of a multi-modal avatar-based tool for remediation of social cognition deficits, J. Ambient Intell. Humaniz. Comput. 1 (2020) 4513–4524. Crossref, Web of Science, Google Scholar
- 38. , Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability, Virtual Reality 18(1) (2014) 61–71. Crossref, Web of Science, Google Scholar
- 39. , Facial emotion recognition in schizophrenia: Intensity effects and error pattern, Am. J. Psychiatry 160(10) (2003) 1768–1774. Crossref, Medline, Web of Science, Google Scholar
- 40. , Virtual Reality Technology (John Wiley & Sons, 2003). Crossref, Google Scholar
- 41. , Constructing a gazebo: Supporting teamwork in a tightly coupled, distributed task in virtual reality, Presence 12(6) (2003) 644–657. Crossref, Google Scholar
- 42. , Evaluating collaboration in distributed virtual environments for a puzzle-solving task, HCI Int. 2005, the 11th Int. Conf. Human Computer Interaction (Las Vegas, NV, 2005), pp. 1–10. Google Scholar
- 43. , Collaborative virtual environments: You cant do it alone, can you? in Int. Conf. Virtual Reality (Springer, Beijing, China, 2007), pp. 224–233. Crossref, Google Scholar
- 44. , Remediating learning from non-immersive to immersive media: Using EEG to investigate the effects of environmental embeddedness on reading in virtual reality, Comput. Educ. 164 (2021) 104122. Crossref, Web of Science, Google Scholar
- 45. , Immersion, presence and performance in virtual environments: An experiment with tri-dimensional chess, in ACM Symp. Virtual Reality Software and Technology (ACM, Hong Kong, 1996), pp. 163–172. Crossref, Google Scholar
- 46. , The relationship between presence and performance in virtual simulation training, Open J. Model. Simul. 3(2) (2015) 41. Crossref, Google Scholar
- 47. , A collaborative workspace architecture for strengthening collaboration among space scientists, in 2015 IEEE Aerospace Conf. (IEEE, Big Sky, MT, 2015), pp. 1–12. Crossref, Google Scholar
- 48. , Recognition profile of emotions in natural and virtual faces, PLoS One 3(11) (2008) e3628. Crossref, Medline, Web of Science, Google Scholar
- 49. , Facsgen 2.0 animation software: Generating three-dimensional facs-valid facial expressions for emotion research, Emotion 12(2) (2012) 351. Crossref, Medline, Web of Science, Google Scholar
- 50. , Virtual faces expressing emotions: An initial concomitant and construct validity study, Front. Hum. Neurosci. 8 (2014) 1–6. Crossref, Medline, Web of Science, Google Scholar
- 51. , Hapfacs 3.0: Facs-based facial expression generator for 3d speaking virtual characters, IEEE Trans. Affect. Comput. 6(4) (2015) 348–360. Crossref, Web of Science, Google Scholar
- 52. , Virtual reality facial emotion recognition in social environments: An eye-tracking study, Internet Interv. 25 (2021) 100432. Crossref, Medline, Google Scholar
- 53. , Investigating the process of emotion recognition in immersive and non-immersive virtual technological setups, in Proc. 22nd ACM Conf. Virtual Reality Software and Technology (Christchurch, New Zealand, 2016), pp. 61–64. Crossref, Google Scholar
- 54. , Design of reliable virtual human facial expressions and validation by healthy people, Integr. Comput.-Aided Eng. 27(3) (2020) 287–299. Crossref, Web of Science, Google Scholar
- 55. , Validation of dynamic virtual faces for facial affect recognition, PLoS One 16(1) (2021) 1–15. Crossref, Web of Science, Google Scholar
- 56. , Facial affect recognition by patients with schizophrenia using human avatars, J. Clin. Med. 10(9) (2021) 1904. Crossref, Medline, Web of Science, Google Scholar
- 57. , Facial emotion recognition in patients with depression compared to healthy controls when using human avatars, Sci. Rep. 13(1) (2023) 6007. Crossref, Medline, Web of Science, Google Scholar
- 58. , Facial affect recognition in depression using human avatars, Appl. Sci. 13(3) (2023) 1609. Crossref, Google Scholar
- 59. , How interpersonal distance between avatar and human influences facial affect recognition in immersive virtual reality, Front. Psychol. 12 (2021) 675515. Crossref, Medline, Web of Science, Google Scholar
- 60. , Influence of the level of immersion in emotion recognition using virtual humans, in Int. Work-Conf. Interplay between Natural and Artificial Computation (Springer, Puerto de la Cruz, Tenerife, 2022), pp. 464–474. Crossref, Google Scholar
- 61. , Emotion recognition in immersive virtual reality: From statistics to affective computing, Sensors 20(18) (2020) 5163. Crossref, Web of Science, Google Scholar
- 62. , Unmasking the Face (Consulting Psychologists Pr, 2003). Google Scholar
- 63. , Escalas PANAS de afecto positivo y negativo: Validacion factorial y convergencia transcultural, Psicothema 11(1) (1999) 37–51. Google Scholar
- 64. N. Hussain, H. Ujir, I. Hipiny and J.-L. Minoi, 3d facial action units recognition for emotional expression arXiv:1712.00195. Google Scholar
- 65. , Facial affect recognition in immersive virtual reality: Where is the participant looking? Int. J. Neural Syst. 32(10) (2022) 2250029. Link, Web of Science, Google Scholar
- 66. , Eye movements during emotion recognition in faces, J. Vis. 14(13) (2014), https://doi.org/10.1167/14.13.14. Crossref, Medline, Web of Science, Google Scholar
- 67. , What makes a smiling face look happy? Visual saliency, distinctiveness, and affect, Psychol. Res. 82 (2018) 296–309. Crossref, Medline, Web of Science, Google Scholar
- 68. , Moving smiles: The role of dynamic components for the perception of the genuineness of smiles, J. Nonverbal Behav. 29 (2005) 3–24. Crossref, Web of Science, Google Scholar
- 69. , Information and viewpoint dependence in face recognition, Cognition 62 (1997) 201–222. Crossref, Medline, Web of Science, Google Scholar
- 70. , The resolution of facial expressions of emotion, J. Vis. 11 (2011) 1–13. Crossref, Web of Science, Google Scholar
- 71. , Temporal dynamics of emotional processing in the brain, Emot. Rev. 7(4) (2015) 323–329. Crossref, Google Scholar
- 72. , Converging evidence for the advantage of dynamic facial expressions, Brain Topogr. 24(2) (2011) 149–163. Crossref, Medline, Web of Science, Google Scholar
- 73. , Emotions in motion: Dynamic compared to static facial expressions of disgust and happiness reveal more widespread emotion-specific activations, Brain Res. 1284 (2009) 100–115. Crossref, Medline, Web of Science, Google Scholar
- 74. , Facial emotion recognition in social anxiety: The influence of dynamic information, Psychol. Neurosci. 9(1) (2016) 1–11. Crossref, Google Scholar
- 75. , Effects of dynamic aspects of facial expressions: A review, Emot. Rev. 5(1) (2013) 41–46. Crossref, Google Scholar
- 76. , Age deficits in facial affect recognition: The influence of dynamic cues, J. Gerontol. B Psychol. Sci. Soc. Sci. 72(4) (2017) 622–632. Medline, Web of Science, Google Scholar
- 77. , Static and dynamic presentation of emotions in different facial areas: Fear and surprise show influences of temporal and spatial properties, Psychology 4(8) (2013) 663–668. Crossref, Google Scholar
- 78. , Recognising subtle emotional expressions: The role of facial movements, Cogn. Emot. 22(8) (2008) 1569–1587. Crossref, Web of Science, Google Scholar
- 79. , Deciphering the enigmatic face the importance of facial dynamics in interpreting subtle facial expressions, Psychol. Sci. 16(5) (2005) 403–410. Crossref, Medline, Web of Science, Google Scholar
- 80. , Tuning functions for automatic detection of brief changes of facial expression in the human brain, NeuroImage 179 (2018) 235–251. Crossref, Medline, Web of Science, Google Scholar
- 81. , Sex differences in perception of emotion intensity in dynamic and static facial expressions, Exp. Brain Res. 171(1) (2006) 1–6. Crossref, Medline, Web of Science, Google Scholar
- 82. , Brief report: Representational momentum for dynamic facial expressions in pervasive developmental disorder, J. Autism Dev. Disord. 40(3) (2010) 371–377. Crossref, Medline, Web of Science, Google Scholar
- 83. , Enhanced neural activity in response to dynamic facial expressions of emotion: An FMRI study, Cogn. Brain Res. 20(1) (2004) 81–91. Crossref, Medline, Google Scholar
Remember to check out the Most Cited Articles! |
---|
Check out our titles in neural networks today! |