World Scientific
  • Search
Skip main navigation

Cookies Notification

We use cookies on this site to enhance your user experience. By continuing to browse the site, you consent to the use of our cookies. Learn More
Our website is made possible by displaying certain online content using javascript.
In order to view the full content, please disable your ad blocker or whitelist our website

System Upgrade on Tue, Oct 25th, 2022 at 2am (EDT)

Existing users will be able to log into the site and access content. However, E-commerce and registration of new users may not be available for up to 12 hours.
For online purchase, please visit us again. Contact us at [email protected] for any enquiries.

Automated Task Load Detection with Electroencephalography: Towards Passive Brain–Computer Interfacing in Robotic Surgery

    Automatic detection of the current task load of a surgeon in the theatre in real time could provide helpful information, to be used in supportive systems. For example, such information may enable the system to automatically support the surgeon when critical or stressful periods are detected, or to communicate to others when a surgeon is engaged in a complex maneuver and should not be disturbed. Passive brain–computer interfaces (BCI) infer changes in cognitive and affective state by monitoring and interpreting ongoing brain activity recorded via an electroencephalogram. The resulting information can then be used to automatically adapt a technological system to the human user. So far, passive BCI have mostly been investigated in laboratory settings, even though they are intended to be applied in real-world settings. In this study, a passive BCI was used to assess changes in task load of skilled surgeons performing both simple and complex surgical training tasks. Results indicate that the introduced methodology can reliably and continuously detect changes in task load in this realistic environment.

    This paper was recommended for publication in its revised form by Editor Mahdi Tavakoli.

    NOTICE: Prior to using any material contained in this paper, the users are advised to consult with the individual paper author(s) regarding the material contained in this paper, including but not limited to, their specific design(s) and recommendation(s).


    • 1. P. J. Guillou, P. Quirke, H. Thorpe, J. Walker, D. G. Jayne, A. M. H. Smith, R. M. Heath and J. M. Brown, Short-term endpoints of conventional versus laparoscopic-assisted surgery in patients with colorectal cancer (mrc 5clasicc6 trial): Multicentre, randomised controlled trial, The Lancet 365 (9472) (2005) 1718–1726. CrossrefGoogle Scholar
    • 2. R. Berguer, W. D. Smith and Y. H. Chung, Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery, Surg. Endosc. 15 (10) (2001) 1204–1207. CrossrefGoogle Scholar
    • 3. J. R. Korndorffer, J. B. Dunne, R. Sierra, D. Stefanidis, C. L. Touchard and D. J. Scott, Simulator training for laparoscopic suturing using performance goals translates to the operating room, J. Am. Coll. Surg. 201 (1) (2005) 23–29. CrossrefGoogle Scholar
    • 4. A. G. Gallagher, R. M. Satava and G. C. Osullivan, Attentional capacity: An essential aspect of surgeon performance, Ann. Surg. (2013). Google Scholar
    • 5. A. N. Healey, N. Sevdalis and C. A. Vincent, Measuring intra-operative interference from distraction and interruption observed in the operating theatre, Ergonomics 49 (5–6) (2006) 589–604. CrossrefGoogle Scholar
    • 6. D. Stefanidis, M. W. Scerbo, J. R. Korndorffer and D. J. Scott, Redefining simulator proficiency using automaticity theory, Am. J. Surg. 193 (4) (2007) 502–506. CrossrefGoogle Scholar
    • 7. K. E. Hsu, F.-Y. Man, R. A. Gizicki, L. S. Feldman and G. M. Fried, Experienced surgeons can do more than one thing at a time: Effect of distraction on performance of a simple laparoscopic and cognitive task by experienced and novice surgeons, Surgical Endoscopy 22 (1) (2008) 196–201. CrossrefGoogle Scholar
    • 8. B. Zheng, M. A. Cassera, D. V. Martinec, G. O. Spaun and L. L. Swanström, Measuring mental workload during the performance of advanced laparoscopic tasks, Surg. Endosc. 24 (1) (2010) 45–50. CrossrefGoogle Scholar
    • 9. B. Zheng, G. Tien, S. M. Atkins, C. Swindells, H. Tanin, A. Meneghetti, K. A. Qayumi, O. Neely and M. Panton, Surgeon’s vigilance in the operating room, Am. J. Surg. 201 (5) (2011) 673–677. CrossrefGoogle Scholar
    • 10. M. A. Staal, Stress, cognition, and human performance: A literature review and conceptual framework, NASA Technical Memorandum 212824 (2004). Google Scholar
    • 11. M. Fafrowicz and T. Marek, Quo vadis, neuroergonomics?, Ergonomics 50 (11) (2007) 1941–1949. CrossrefGoogle Scholar
    • 12. S. G. Hart and L. E. Staveland, Development of nasa-tlx (task load index): Results of empirical and theoretical research, in Human Mental Workload, eds. P. A. HancockN. Meshkati (North-Holland, Amsterdam, 1988), pp. 139–183. CrossrefGoogle Scholar
    • 13. M. A. Just, P. A. Carpenter and A. Miyake, Neuroindices of cognitive workload: Neuroimaging, pupillometric and event-related potential studies of brain work, Theor. Issues Ergon. Sci. 4 (1-2) (2003) 56–88. CrossrefGoogle Scholar
    • 14. F. F. Zhu, J. M. Poolton, M. R. Wilson, Y. Hu, J. P. Maxwell and R. S. W. Masters, Implicit motor learning promotes neural efficiency during laparoscopy, Surg. Endosc. 25 (9) (2011) 2950–2955. CrossrefGoogle Scholar
    • 15. D. R. Leff, C. E. Elwell, F. Orihuela-Espina, L. Atallah, D. T. Delpy, A. W. Darzi and G.-Z. Yang, Changes in prefrontal cortical behaviour depend upon familiarity on a bimanual co-ordination task: An fnirs study, NeuroImage 39 (2) (2008) 805–813. CrossrefGoogle Scholar
    • 16. K. Ohuchida, H. Kenmotsu, A. Yamamoto, K. Sawada, T. Hayami, K. Morooka, S. Takasugi, K. Konishi, S. Ieiri, K. Tanoue, Y. Iwamoto, M. Tanaka and M. Hashizume, The frontal cortex is activated during learning of endoscopic procedures, Surg. Endosc. 23 (10) (2009) 2296–2301. CrossrefGoogle Scholar
    • 17. D. R. C. James, F. Orihuela-Espina, D. R. Leff, G. P. Mylonas, K.-W. Kwok, A. W. Darzi and G.-Z. Yang, Cognitive burden estimation for visuomotor learning with fnirs, MICCAI 2010, Lecture Notes in Computer Science, 6363 (Springer Berlin Heidelberg, Berlin and Heidelberg, 2010), pp. 319–326. Google Scholar
    • 18. D. R. Leff, F. Orihuela-Espina, T. Athanasiou, V. Karimyan, C. Elwell, J. Wong, G.-Z. Yang and A. W. Darzi, Circadian cortical compensation: a longitudinal study of brain function during technical and cognitive skills in acutely sleep-deprived surgical residents, Ann. Surg. 252 (6) (2010) 1082–1090. CrossrefGoogle Scholar
    • 19. S. Makeig, K. Gramann, T.-P. Jung, T. J. Sejnowski and H. Poizner, Linking brain, mind and behavior, Int. J. Psychophysiol. 73 (2) (2009) 95–100. CrossrefGoogle Scholar
    • 20. K. Gramann, D. P. Ferris, J. Gwin and S. Makeig, Imaging natural cognition in action, Int. J. Psychophysiol. (2013). Google Scholar
    • 21. G.-Z. Yang, G. P. Mylonas, K.-W. Kwok and A. Chung, Perceptual docking for robotic control, in Medical Imaging and Augmented Reality, eds. T. DohiI. SakumaH. Liao, Vol. 5128 (Springer, Berlin, 2008), pp. 21–30. CrossrefGoogle Scholar
    • 22. M. Rötting, T. Zander, S. Trösterer and J. Dzaack, Implicit interaction in multimodal human-machine systems, in Industrial Engineering and Ergonomics, ed. C. M. Schlick (Springer, Berlin, 2009), pp. 523–536. CrossrefGoogle Scholar
    • 23. T. O. Zander and C. Kothe, Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general, J. Neural Eng. 8 (2) (2011) 025005. CrossrefGoogle Scholar
    • 24. J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller and T. M. Vaughan, Brain–computer interfaces for communication and control, Clin. Neurophysiol. 113 (6) (2002) 767–791. CrossrefGoogle Scholar
    • 25. T. O. Zander, C. Kothe, S. Jatzev and M. Gaertner, Enhancing human-computer interaction with input from active and passive brain-computer interfaces, in Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction, eds. D. S. TanA. Nijholt, Human-Computer Interaction Series (Springer, London, 2010), pp. 181–199. CrossrefGoogle Scholar
    • 26. T. O. Zander and S. Jatzev, Context-aware brain–computer interfaces: Exploring the information space of user, technical system and environment, J. Neural Eng. 9 (1) (2012) 016003. CrossrefGoogle Scholar
    • 27. J. Kohlmorgen, G. Dornhege, M. Braun, B. Blankertz, K. R. Muller, G. Curio, K. Hagemann, A. Bruns, M. Schrauf and W. Kincses, Improving human performance in a real operating environment through real-time mental workload detection, in Toward Brain-Computer Interfacing, eds. G. DornhegeJ. R. MillánT. HinterbergerD. J. McFarlandK.-R. Müller (MIT Press, Cambridge, 2007), pp. 409–422. Google Scholar
    • 28. D. Grimes, D. S. Tan, S. E. Hudson, S. Pradeep and R. P. N. Rao, Feasibility and pragmatics of classifying working memory load with an electroencephalograph, in Pro. SIGCHI Conf. Human Factors in Computing Systems, ed. M. Czerwinski (ACM, New York and NY, 2008), pp. 835–844. Google Scholar
    • 29. C. D. Wickens and J. S. McCarley, Applied Attention Theory (CRC Press, Boca Raton, 2008). Google Scholar
    • 30. A. Gevins, Neurophysiological measures of working memory and individual differences in cognitive ability and cognitive style, Cereb. Cortex 10 (9) (2000) 829–839. CrossrefGoogle Scholar
    • 31. A. Gevins and M. E. Smith, Neurophysiological measures of cognitive workload during human-computer interaction, Theor. Issues Ergon. Sci. 4 (1-2) (2003) 113–131. CrossrefGoogle Scholar
    • 32. A. Holm, K. Lukander, J. Korpela, M. Sallinen and K. M. I. Müller, Estimating brain load from the EEG, Scientific World Journal 9 (2009) 639–651. CrossrefGoogle Scholar
    • 33. M. E. Smith, A. Gevins, H. Brown, A. Karnik and R. Du, Monitoring task loading with multivariate eeg measures during complex forms of human-computer interaction, Hum. Factors 43 (3) (2001) 366–380. CrossrefGoogle Scholar
    • 34. C. A. Kothe and S. Makeig, Bcilab: A platform for brain-computer interface development, J. Neural Eng. 10 (5) (2013) 056014. CrossrefGoogle Scholar
    • 35. R. Tomioka, G. Dornhege, G. Nolte, B. Blankertz, K. Aihara and K.-R. Müller, Spectrally Weighted Common Spatial Pattern Algorithm for Single Trial EEG classification, Mathematical Engineering Technical Reports (University of Tokyo, Tokyo, Japan, 2006). Google Scholar
    • 36. R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification, 2nd edn. (Wiley, New York, 2001). Google Scholar
    • 37. B. Blankertz, S. Lemm, M. Treder, S. Haufe and K.-R. Müller, Single-trial analysis and classification of erp components — A tutorial: Multivariate decoding and brain reading, NeuroImage 56 (2) (2011) 814–825. CrossrefGoogle Scholar
    • 38. V. N. Vapnik and A. Y. Chervonenkis, On the uniform convergence of relative frequencies of events to their probabilities, Theor. Probab. Appl. 16 (2) (1971) 264–280. CrossrefGoogle Scholar
    • 39. A. Delorme and S. Makeig, EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis, J. 134 (1) (2004) 9–21. Google Scholar
    • 40. S. Makeig, Auditory event-related dynamics of the eeg spectrum and effects of exposure to tones, Electroencephalogr. Clin. Neurophysiol. 86 (4) (1993) 283–293. CrossrefGoogle Scholar
    • 41. H. Ayaz, P. A. Shewokis, S. Bunce, K. Izzetoglu, B. Willems and B. Onaral, Optical brain monitoring for operator training and mental workload assessment, NeuroImage 59 (1) (2012) 36–47. CrossrefGoogle Scholar
    • 42. K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei and B. Chance, Functional optical brain imaging using near-infrared during cognitive tasks, Int. J. Hum.-Comput. Interact. 17 (2) (2004) 211–227. CrossrefGoogle Scholar
    • 43. A. Sonnleitner, M. S. Treder, M. Simon, S. Willmann, A. Ewald, A. Buchner and M. Schrauf, {EEG} alpha spindles and prolonged brake reaction times during auditory distraction in an on-road driving study, Accid. Anal. Prev. 62 (0) (2014) 110–118. CrossrefGoogle Scholar
    • 44. J. T. Gwin, K. Gramann, S. Makeig and D. P. Ferris, Electrocortical activity is coupled to gait cycle phase during treadmill walking, NeuroImage 54 (2) (2011) 1289–1296. CrossrefGoogle Scholar
    • 45. K. Gramann, J. T. Gwin, N. Bigdely-Shamlo, D. P. Ferris and S. Makeig, Visual evoked responses during standing and walking, Front. Hum. Neurosci. 4 (202) (2010). Google Scholar
    • 46. T. O. Zander, J. Broenstrup, R. Lorenz and L. R. Krol, Towards BCI-based implicit control in human-computer interaction, in Advances in Physiological Computing, eds. S. FaircloughK. Gilleade, Human-Computer Interaction Series (Springer, London). Google Scholar