LieToMe: An Ensemble Approach for Deception Detection from Facial Cues
Abstract
Deception detection is a relevant ability in high stakes situations such as police interrogatories or court trials, where the outcome is highly influenced by the interviewed person behavior. With the use of specific devices, e.g. polygraph or magnetic resonance, the subject is aware of being monitored and can change his behavior, thus compromising the interrogation result. For this reason, video analysis-based methods for automatic deception detection are receiving ever increasing interest. In this paper, a deception detection approach based on RGB videos, leveraging both facial features and stacked generalization ensemble, is proposed. First, a face, which is well-known to present several meaningful cues for deception detection, is identified, aligned, and masked to build video signatures. These signatures are constructed starting from five different descriptors, which allow the system to capture both static and dynamic facial characteristics. Then, video signatures are given as input to four base-level algorithms, which are subsequently fused applying the stacked generalization technique, resulting in a more robust meta-level classifier used to predict deception. By exploiting relevant cues via specific features, the proposed system achieves improved performances on a public dataset of famous court trials, with respect to other state-of-the-art methods based on facial features, highlighting the effectiveness of the proposed method.
References
- 1. , Real-time smart video surveillance to manage safety: A case study of a transport mega-project, Adv. Eng. Inform. 45 (2020) 1–10. Crossref, Web of Science, Google Scholar
- 2. , Aerial video surveillance system for small-scale uav environment monitoring, in Proc. IEEE Int. Conf. on Advanced Video and Signal Based Surveillance (AVSS) (IEEE, Lecce, Italy, 2017), pp. 1–6. Crossref, Google Scholar
- 3. , Feature-based slam algorithm for small scale uav with nadir view, in Proc. Int. Conf. on Image Analysis and Processing (ICIAP) (Springer, Cham, Trento, Italy, 2019), pp. 457–467. Crossref, Google Scholar
- 4. , Computer vision for assistive technologies, Comput. Vis. Image Understand. 154 (2017) 1–15. Crossref, Web of Science, Google Scholar
- 5. , An interactive and low-cost full body rehabilitation framework based on 3d immersive serious games, J. Biomed. Inform. 89 (2019) 81–100. Crossref, Medline, Web of Science, Google Scholar
- 6. , Deep temporal analysis for non-acted body affect recognition, IEEE Trans. Affect. Comput. (2020) 1. https://doi.org/10.1109/TAFFC.2020.3003816 Crossref, Web of Science, Google Scholar
- 7. , Long activity video understanding using functional object-oriented network, IEEE Trans. Multimedia 21(7) (2019) 1813–1824. Crossref, Web of Science, Google Scholar
- 8. , 2d skeleton-based action recognition via two-branch stacked lstm-rnns, IEEE Trans. Multimedia 22 (2019) 10. Web of Science, Google Scholar
- 9. , Automatic processing and solar cell detection in photovoltaic electroluminescence images, Integr. Comput.-Aid. Eng. 26(2) (2019) 123–137. Crossref, Web of Science, Google Scholar
- 10. , Behavioral cues to deception vs. topic incriminating potential in criminal confessions, Law Human Behav. 29(6) (2006) 683–704. Crossref, Web of Science, Google Scholar
- 11. , The evolution and psychology of self-deception, Behav. Brain Sci. 34(1) (2011) 1–16. Crossref, Medline, Web of Science, Google Scholar
- 12. , Accuracy of deception judgments, Personal. Soc. Psychol. Rev. 10(3) (2006) 214–234. Crossref, Medline, Web of Science, Google Scholar
- 13. , Basis for the implementation of an eeg-based single-trial binary brain computer interface through the disgust produced by remembering unpleasant odors, Neurocomputing 160 (2015) 308–318. Crossref, Web of Science, Google Scholar
- 14. , Tangent space features-based transfer learning classification model for two-class motor imagery brain computer interface, Int. J. Neural Syst. 29(10) (2019) 1950025. Link, Web of Science, Google Scholar
- 15. , Olfactory eeg signal classification using a trapezoid difference-based electrode sequence hashing approach, Int. J. Neural Syst. 30(03) (2020) 2050011. Link, Web of Science, Google Scholar
- 16. , A machine learning approach to reveal the neurophenotypes of autisms, Int. J. Neural Syst. 29(07) (2019) 1850058. Link, Web of Science, Google Scholar
- 17. , A novel method of building functional brain network using deep learning algorithm with application in proficiency detection, Int. J. Neural Syst. 29(1) (2019) 1850015. Link, Web of Science, Google Scholar
- 18. , Empirical functional pca for 3d image feature extraction through fractal sampling, Int. J. Neural Syst. 29(02) (2019) 1850040. Link, Web of Science, Google Scholar
- 19. , Single-trial eeg responses classified using latency features, Int. J. Neural Syst. 30(6) (2020) 2050033. Link, Web of Science, Google Scholar
- 20. , Epileptic seizure detection based on time-frequency images of eeg signals using gaussian mixture model and gray level co-occurrence matrix features, Int. J. Neural Syst. 28(7) (2018) 1850003. Link, Web of Science, Google Scholar
- 21. , Simplified and yet turing universal spiking neural p systems with communication on request, Int. J. Neural Syst. 28(08) (2018) 1850013. Link, Web of Science, Google Scholar
- 22. , Spiking neural p systems with communication on request, Int. J. Neural Syst. 27(08) (2017) 1750042. Link, Web of Science, Google Scholar
- 23. , Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage, rev. edn. (W. W. Norton & Company, Inc., 2009), http://communicationcache.com/uploads/1/0/8/8/10887248/telling_lies-_clues_to_deceit_in_the_market-place_politics_and_marriage.pdf. Google Scholar
- 24. , Lying words: Predicting deception from linguistic styles, Personal. Soc. Psychol. Bull. 29(5) (2003) 665–675. Crossref, Medline, Web of Science, Google Scholar
- 25. V. Pérez-Rosas and R. Mihalcea, Cross-cultural deception detection, in Proc. Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (Baltimore, Maryland, 2014), pp. 440–445. Google Scholar
- 26. , Deception detection using real-life trial data, in Proc. ACM on International Conference on Multimodal Interaction (ICMI) (New York, USA, 2015), pp. 59–66. Crossref, Google Scholar
- 27. , Masking domain-specific information for cross-domain deception detection, Pattern Recogn. Lett. 135 (2020) 122–130. Crossref, Web of Science, Google Scholar
- 28. , Lietome: Preliminary study on hand gestures for deception detection via fisher-lstm, Pattern Recogn. Lett. 138 (2020) 455–461. Crossref, Web of Science, Google Scholar
- 29. , Detecting deception from the body or face, J. Personal. Soc. Psychol. 29(3) (1974) 288–298. Crossref, Web of Science, Google Scholar
- 30. , What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) (Oxford University Press, 1997), https://psycnet.apa.org/record/1998-07552-000. Google Scholar
- 31. , Darwin, deception, and facial expression, Ann. New York Acad. Sci. 1000(1) (2003) 205–221. Crossref, Medline, Web of Science, Google Scholar
- 32. , Verbal and nonverbal communication of deception, Adv. Exper. Soc. Psychol. 14 (1981) 1–59. Crossref, Google Scholar
- 33. , Automatic deception detection in rgb videos using facial action units, in Proc. Int. Conf. Distributed Smart Cameras (ICDSC) (Trento, Italy, 2019), pp. 1–6. Crossref, Google Scholar
- 34. , Cross-dataset learning and person-specific normalisation for automatic action unit detection, in Proc. Int. Conf. and Workshops on Automatic Face and Gesture Recognition (FG) (IEEE, Ljubljana, Slovenia, 2015), pp. 1–6. Google Scholar
- 35. , An optimizer ensemble algorithm and its application to image registration, Integr. Comput.-Aid. Eng. 26(4) (2019) 311–327. Crossref, Web of Science, Google Scholar
- 36. , A dynamic ensemble learning algorithm for neural networks, Neural Comput. Appl. 32(12) (2020) 8675–8690. Crossref, Web of Science, Google Scholar
- 37. , Stacked generalization, Neural Netw. 5(2) (1992) 241–259. Crossref, Web of Science, Google Scholar
- 38. , Bag-of-lies: A multimodal dataset for deception detection, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE, Long Beach, California, 2019), pp. 1–1. Crossref, Google Scholar
- 39. , A bispectrum-based approach for detecting deception using eeg signals, in Proc. IEEE Int. Conf. on e-Health Networking, Applications and Services (Healthcom) (IEEE, Ostrava, Czech, 2018), pp. 1–6. Crossref, Google Scholar
- 40. , Constructing the lie detection system with fuzzy reasoning approach, Granul. Comput. 3 (2018) 169–176. Crossref, Google Scholar
- 41. , Functional mri lie detection: Too good to be true? J. Amer. Acad. Psychiatry Law 36(4) (2008) 491–498. Medline, Web of Science, Google Scholar
- 42. , Detecting deception using functional magnetic resonance imaging, Biol. Psychiatry 58(8) (2005) 605–613. Crossref, Medline, Web of Science, Google Scholar
- 43. , Prospects of functional magnetic resonance imaging as lie detector, Front. Human Neurosci. 7 (2013) 594. Crossref, Medline, Web of Science, Google Scholar
- 44. , Lie detection using fnirs monitoring of inhibition-related brain regions discriminates infrequent but not frequent liars, Front. Human Neurosci. 12 (2018) 1–11. Crossref, Medline, Web of Science, Google Scholar
- 45. , Single-trial lie detection using a combined fnirs-polygraph system, Front. Psychol. 6 (2015) 1–12. Crossref, Medline, Web of Science, Google Scholar
- 46. , Spontaneous lie detection using functional near-infrared spectroscopy in an interactive game, in Proc. Asian Control Conference (ASCC) (IEEE, Kota Kinabalu, Malaysia, 2015), pp. 1–5. Crossref, Google Scholar
- 47. , Deception detecting from speech signal using relevance vector machine and non-linear dynamics features, Neurocomputing 151(3) (2015) 1042–1052. Crossref, Web of Science, Google Scholar
- 48. , Convolutional bidirectional long short-term memory for deception detection with acoustic features, IEEE Access 6 (2018) 76527–76534. Crossref, Web of Science, Google Scholar
- 49. , Lie to me: Deceit detection via online behavioral learning, in Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG) (IEEE, Santa Barbara, CA, USA, 2011), pp. 24–29. Crossref, Google Scholar
- 50. , Deception is in the eye of the communicator: Investigating pupil diameter variations in automated deception detection interviews, in Proc. IEEE Int. Conf. on Intelligence and Security Informatics (ISI) (IEEE, Baltimore, MD, USA, 2015), pp. 97–102. Crossref, Google Scholar
- 51. , Eye blink count and eye blink duration analysis for deception detection, in Proc. Int. Conf. on Advances in Computing, Communications and Informatics (ICACCI) (IEEE, Udupi, India, 2017), pp. 223–229. Crossref, Google Scholar
- 52. , High-stakes deception detection based on facial expressions, in Proc. Int. Conf. on Pattern Recognition (ICPR) (IEEE, Stockholm, Sweden, 2014), pp. 2519–2524. Crossref, Google Scholar
- 53. , The design and development of a lie detection system using facial micro-expressions, in Proc. Int. Conf. on Advances in Computational Tools for Engineering Applications (ACTEA) (IEEE, Beirut, Lebanon, 2012), pp. 33–38. Crossref, Google Scholar
- 54. , Toward end-to-end deception detection in videos, in Proc. IEEE Int. Conf. Big Data (Big Data) (IEEE, Seattle, WA, USA, USA, 2018), pp. 1278–1283. Crossref, Google Scholar
- 55. , Detecting deceptive behavior via integration of discriminative features from multiple modalities, IEEE Trans. Inform. Forensics Secur. 12(5) (2017) 1042–1055. Crossref, Web of Science, Google Scholar
- 56. , Deception detection in videos, in Proc. AAAI Conf. on Artificial Intelligence (AI) (AAAI Press, New Orleans, Louisiana, USA, 2018), pp. 1–8. Crossref, Google Scholar
- 57. , High-level features for multimodal deception detection in videos, in Proc. Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE, Long Beach, CA, USA, 2019), pp. 1565–1573. Google Scholar
- 58. , A deep learning approach for multimodal deception detection, in Proc. Int. Conf. on Computational Linguistics and Intelligent Text Processing (CICLing) (Springer, Cham, Hanoi, Vietnam, 2018), pp. 1–10. Google Scholar
- 59. , Openface 2.0: Facial behavior analysis toolkit, in Proc. IEEE Int. Conf. on Automatic Face Gesture Recognition (FG) (IEEE, Xi’an, China, 2018), pp. 59–66. Crossref, Google Scholar
- 60. , Orientation histograms for hand gesture recognition, in Proc. Int. Workshop on Automatic Face and Gesture Recognition (FGW), (IEEE, Zurich, Switzerland, 1995), pp. 296–301. Google Scholar
- 61. , Performance evaluation of texture measures with classification based on kullback discrimination of distributions, in Proc. Int. Conf. on Pattern Recognition (ICPR) (IEEE, Jerusalem, Israel, 1994), pp. 582–585. Crossref, Google Scholar
- 62. , Action recognition with improved trajectories, in Proc. IEEE Int. Conference on Computer Vision (ICCV) (IEEE, Sydney, NSW, Australia, 2013), pp. 3551–3558. Crossref, Google Scholar
- 63. , Dynamic texture recognition using local binary patterns with an application to facial expressions, IEEE Trans. Pattern Anal. Mach. Intelli. 29(6) (2007) 915–928. Crossref, Medline, Web of Science, Google Scholar
- 64. , Orb: An efficient alternative to sift or surf, in Proc. Int. Conf. on Computer Vision (ICCV) (IEEE, Barcelona, Spain, 2011), pp. 2564–2571. Crossref, Google Scholar
- 65. , Image classification with the fisher vector: Theory and practice, Int. J. Comput. Vis. 105(3) (2013) 222–245. Crossref, Web of Science, Google Scholar
- 66. , Support-vector networks, Mach. Learn. 20(3) (1995) 273–297. Crossref, Web of Science, Google Scholar
- 67. , Dual coordinate descent methods for logistic regression and maximum entropy models, Mach. Learn. 85(1–2) (2011) 41–75. Crossref, Web of Science, Google Scholar
- 68. , Xgboost: A scalable tree boosting system, in Proc. ACM Int. Conf. on Knowledge Discovery and Data Mining (SIGKDD) (New York, USA, 2016), pp. 785–794. Crossref, Google Scholar
- 69. , Multilayer perceptrons for classification and regression, Neurocomputing 2(5) (1991) 183–197. Crossref, Google Scholar
- 70. , Facial expression recognition and histograms of oriented gradients: a comprehensive study, SpringerPlus 4(645) (2015) 1–25. Medline, Google Scholar
- 71. , Face description with local binary patterns: Application to face recognition, IEEE Trans. Pattern Anal. Mach. Intelli. 28(12) (2006) 2037–2041. Crossref, Medline, Web of Science, Google Scholar
- 72. , Moderators of nonverbal indicators of deception: A meta-analytic synthesis, Psychol. Public Policy Law 13(1) (2007) 1–34. Crossref, Web of Science, Google Scholar
- 73. , Facial expression recognition in the wild using improved dense trajectories and fisher vector encoding, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW) (IEEE, Las Vegas, NV, USA, 2016), pp. 1517–1525. Crossref, Google Scholar
- 74. , Action recognition by dense trajectories, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (IEEE, Providence, RI, USA, 2011), pp. 3169–3176. Crossref, Google Scholar
- 75. , Dynamic texture recognition using local binary patterns with an application to facial expressions, IEEE Trans. Pattern Anal. Mach. Intelli. 29(6) (2007) 915–928. Crossref, Medline, Web of Science, Google Scholar
- 76. , 3-d face recognition with local shape descriptors, IEEE Trans. Inform. Forensics Secur. 7(2) (2012) 577–587. Crossref, Web of Science, Google Scholar
- 77. , Surpassing human-level face verification performance on lfw with gaussian face, in Proc. AAAI Conf. on Artificial Intelligence (AI), (AAAI Press, Austin, Texas, USA, 2015), pp. 3811–3819. Google Scholar
- 78. , Face recognition based on improved surf, in Proc. Int. Conf. Intelligent System Design and Engineering Applications (ISDEA) (IEEE, Hong Kong, China, 2013), pp. 755–758. Crossref, Google Scholar
- 79. , Face detection using surf cascade, in Proc. IEEE Int. Conf. Computer Vision Workshops (ICCVW) (IEEE, Barcelona, Spain, 2011), pp. 2183–2190. Google Scholar
- 80. , Fusion of domain-specific and trainable features for gender recognition from face images, IEEE Access 6 (2018) 24171–24183. Crossref, Web of Science, Google Scholar
- 81. , Machine learning for high-speed corner detection, in Proc. European Conf. on Computer Vision (ECCV) (Springer, Berlin, Heidelberg, 2006), pp. 430–443. Crossref, Google Scholar
- 82. , Brief: Binary robust independent elementary features, in Proc. European Conf. Computer Vision (ECCV) (Springer, Berlin, Heidelberg, 2010), pp. 778–792. Crossref, Google Scholar
- 83. , An efficient orb based face recognition framework for human–robot interaction, Procedia Comput. Sci. 133 (2018) 913–923. Crossref, Google Scholar
- 84. , Measuring corner properties, Comput. Vis. Image Understand. 73(2) (1999) 291–307. Crossref, Web of Science, Google Scholar
- 85. , Compressing fisher vector for robust face recognition, IEEE Access 5 (2017) 23157–23165. Crossref, Web of Science, Google Scholar
- 86. , Efficient video face recognition by using fisher vector encoding of binary features, in Proc. Int. Conf. on Pattern Recognition (ICPR) (IEEE, Cancun, Mexico, 2016), pp. 1436–1441. Crossref, Google Scholar
- 87. , Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods, IEEE Trans. Affect. Comput. 9(4) (2018) 563–577. Crossref, Web of Science, Google Scholar
- 88. , Deception detection through automatic, unobtrusive analysis of nonverbal behavior, IEEE Intelli. Syst. 20(5) (2005) 36–43. Crossref, Web of Science, Google Scholar
- 89. , Binary face image recognition using logistic regression and neural network, in Proc. Int. Conf. on Energy, Communication, Data Analytics and Soft Computing (ICECDS) (IEEE, Chennai, India, 2017), pp. 3883–3888. Crossref, Google Scholar
- 90. , Facial expression recognition based on facial action unit, in Proc. Int. Green and Sustainable Computing Conf. (IGSC) (IEEE, Alexandria, VA, USA, 2019), pp. 1–6. Crossref, Google Scholar
- 91. , Emotion recognition using facial expressions, Procedia Comput. Sci. 108 (2017) 1175–1184. Crossref, Google Scholar
- 92. , Detection of deception using facial expressions based on different classification algorithms, in Proc. Scientific Conf. of Electrical Engineering (SCEE) (IEEE, Baghdad, Iraq, 2018), pp. 51–56. Crossref, Google Scholar
- 93. , Microexpressions are not the best way to catch a liar, Front. Psychol. 9 (2018) 1–13. Crossref, Medline, Web of Science, Google Scholar
- 94. , Ensemble-based classifiers, Artifi. Intelli. Rev. 33(1–2) (2010) 1–39. Crossref, Web of Science, Google Scholar