Related work analysis for determination of fatigue state based on eye movements monitoring

Capa

Citar

Texto integral

Acesso aberto Acesso aberto
Acesso é fechado Acesso está concedido
Acesso é fechado Somente assinantes

Resumo

We have reviewed theoretical background of detecting functional state of fatigue based on the strategy of eye movements. Also, modern methods for assessing eye movements were considered. Based on our literature review, we can conclude that nowadays there are multitude numerical characteristics of eye movements, the dynamics of which can hypothetically make it possible to assess degree of fatigue. However, there are still no proposals for a method for determining the degree of fatigue based on an analysis of the strategy of eye movements. In this regard, according to the concepts of static and dynamic vision, it is proposed to consider the shift in the numerical characteristics of eye movements towards characteristics that reflect the strategy of dynamic vision as evidence of fatigue.

Texto integral

Acesso é fechado

Sobre autores

I. Shoshina

Saint-Petersburg State University

Autor responsável pela correspondência
Email: shoshinaii@mail.ru

Институт когнитивных исследований

Rússia, St. Petersburg

S. Kovalenko

National Research University Higher School of Economics

Email: shoshinaii@mail.ru
Rússia, Moscow

V. Kuznetsov

Federal Research Center “Computer Science and Control”, RAS

Email: shoshinaii@mail.ru
Rússia, Moscow

I. Brak

Novosibirsk State University; Privolzhsky Research Medical University, PRMU

Email: shoshinaii@mail.ru
Rússia, Novosibirsk; Nizhny Novgorod

A. Kashevnik

Saint-Petersburg Federal Research Center, RAS

Email: shoshinaii@mail.ru
Rússia, St. Petersburg

Bibliografia

  1. Isakova M. Topic on VPP № 22 for military personnel serving under contract and conscription // Army Collection. 2021. V. 8. P. 126.
  2. Velichkovsky B.B. Cognitive effects of mental fatigue // Bulletin of Moscow University. Series 14. Psychology. 2019. № 1. P. 108.
  3. Borgianni Y., Rauch E., Maccioni L., Mark B.G. User experience analysis in industry 4.0 – The use of biometric devices in engineering design and manufacturing / IEEE International conference on Industrial Engineering and Engineering Management. (IEEM), Bangkok, Thailand, 16-19 December, 2018. IEEE Computer Society, 2019. P. 192.
  4. Velichkovsky B.M., Ushakov V.L. Cognitive sciences and new medical technologies // Modern Technologies in Medicine. 2019. V. 11. № 1. P. 8.
  5. Robertson C.V., Marino F.E. Cerebral responses to exercise and the influence of heat stress in human fatigue // J. Therm. Biol. 2017. V. 63. P. 10.
  6. Bergasa L.M., Nuevo J., Sotelo M.-A. et al. Real-time system for monitoring driver vigilance // IEEE Trans. Intell. Transp. Syst. 2006. V. 7. № 1. P. 63.
  7. D’Orazio T., Leo M., Guaragnella C., Distante A. A visual approach for driver inattention detection // Patt. Recog. 2007. V. 40. № 8. P. 2341.
  8. Al-Anizy G.J., Nordin M.J., Razooq M.M. Automatic driver drowsiness detection using haar algorithm and support vector machine techniques // Asian J. Appl. Sci. 2015. V. 8. № 2. P. 149.
  9. Golz M., Sommer D., Chen M. et al. Feature fusion for the detection of microsleep events // J. VLSI Sign. Process Syst. Sign. Im. 2007. V. 49. № 2. P. 329.
  10. Liu Z., Peng Y., Hu W. Driver fatigue detection based on deeply-learned facial expression representation // J. Vis. Commun. Image Represent. 2020. V. 71. № 2. 102723.
  11. Mandal B., Li L., Wang G.S., Lin J. Towards detection of bus driver fatigue based on robust visual analysis of eye state // IEEE Trans. Intell. Transp. Syst. 2016. V. 18. № 3. P. 545.
  12. Sigari M.H., Fathy M., Soryani M. A driver face monitoring system for fatigue and distraction detection // Int. J. Vehicul. Technol. 2013. V. 2013. P. 1.
  13. Lyapunov S.I., Shoshina I.I., Lyapunov I.S. Tremor eye movements as an objective marker of driver’s fatigue // Human Physiology. 2022. V. 48. № 1. P. 71.
  14. Golz M., Sommer D., Trutschel U. et al. Evaluation of fatigue monitoring technologies // Somnologie. 2010. V. 14. № 3. P. 187.
  15. Kubarko A.I., Likhachev S.A., Kubarko N.P. [Vision (neurophysiological and neuro-ophthalmological aspects): monograph in 2 volumes. V. 2: Neural mechanisms for controlling the installation and movement of the eyes and their disorders in diseases of the nervous system]. Minsk: BSMU, 2009. 352 p.
  16. Barabanshchikov V.A., Zhegallo A.V. [Eye-tracking: methods of recording eye movements in psychological research and practice]. M.: Kogito-center, 2014. P. 117.
  17. Yarbus A. [The role of eye movements in the process of vision]. M.: Nauka, 1965. 161 p.
  18. Holmqvist K. Eye tracking: A comprehensive guide to methods and measures. O.: OUP Oxford, 2011. 560 p.
  19. Djanian S. Eye movement classification using deep learning. Aalborg University: Department of Electronic Systems, 2019. 76 p.
  20. Mahanama B., Jayawardana Y., Rengarajan S. et al. Eye movement and pupil measures: A review // Front. Comput. Sci. 2022. V. 3. P. 733531.
  21. Li X., Fan Z., Ren Y. et al. Classification of eye movement and its application in driving based on a refined pre-processing and machine learning algorithm // IEEE Access. 2021. V. 9. P. 136164.
  22. Salvucci D.D., Goldberg J.H. Identifying fixations and saccades in eye-tracking protocols / Proceedings of the 2000 symposium on Eye Tracking Research and Applications (ETRA ‘00). Palm Beach Gardens, FL, USA. November 6–8, 2000. Association for Computing Machinery (ACM). New York, NY, USA, 2000. P. 71.
  23. Wang S., Wang Q., Chen H. Research and application of eye movement interaction based on eye movement recognition / MATEC Web Conf. 2018. V. 246. P. 5.
  24. Carpenter R.H.S. Movements of the eyes: Part 1. Movements of the eyes. 2nd ed. Pion, 1988. P. 593.
  25. Shojaeizadeh M., Djamasbi S., Trapp A.C. Density of gaze points within a fixation and information processing behavior / Universal Access in Human-Computer Interaction. Methods, Techniques, and Best Practices. UAHCI 2016. Lect. Notes Comput. // Eds. Antona M., Stephanidis C. Sci. Springer, Cham, 2016. V. 9737. P. 465.
  26. Skaramagkas V., Giannakakis G., Ktistakis E. et al. Review of eye tracking metrics involved in emotional and cognitive processes // IEEE Rev. Biomed. Eng. 2023. V. 16. P. 260.
  27. Foy H.J., Chapman P. Mental workload is reflected in driver behavior, physiology, eye movements and prefrontal cortex activation // Appl. Ergon. 2018. V. 73. P. 90.
  28. Srimal R., Diedrichsen J., Ryklin E.B., Curtis C.E. Obligatory adaptation of saccade gains // J. Neurophysiol. 2008. V. 99. № 3. P. 1554.
  29. Russo M., Thomas M., Thorne D. et al. Oculomotor impairment during chronic partial sleep deprivation // Clin. Neurophysiol. 2003. V. 114. № 4. P. 723.
  30. Warren D.E., Thurtell M.J., Carroll J.N., Wall M. Perimetric evaluation of saccadic latency, saccadic accuracy, and visual threshold for peripheral visual stimuli in young compared with older adults // Invest. Ophthalmol. Vis. Sci. 2013. V. 54. № 8. P. 5778.
  31. Yang Y., McDonald M., Zheng P. Can drivers’ eye movements be used to monitor their performance? A case study // IET Intell. Transp. Syst. 2012. V. 6. № 4. P. 444.
  32. Nakayama M., Takahashi K., Shimizu Y. The act of task difficulty and eye-movement frequency for the “oculo-motor indices” / Proceedings of the 2002 symposium on Eye tracking research & applications (ETRA ‘02). New Orleans Louisiana, March 25-27, 2002. Association for Computing Machinery (ACM), New York, NY, USA, 2002. P. 37.
  33. Van Orden K.F., Limbert W., Makeig S., Jung T.P. Eye activity correlates of workload during a visuospatial memory task // Hum. Factors. 2001. V. 43. № 1. P. 111.
  34. Amor T.A., Reis S.D., Campos D. et al. Persistence in eye movement during visual search // Sci. Rep. 2016. V. 6. P. 20815.
  35. Joseph A.W., Murugesh R. Potential eye tracking metrics and indicators to measure cognitive load in human-computer interaction research // J. Sci. Res. 2020. V. 64. № 01. P. 168.
  36. Chen S., Epps J., Ruiz N., Chen F. Eye activity as a measure of human mental effort in HCI / Proceedings of the 16th international conference on Intelligent user interfaces (IUI’11). Association for Computing Machinery (ACM), February 13–16, 2011. New York, NY, USA, 2011. P. 315.
  37. Zagermann J., Pfeil U., Reiterer H. Measuring cognitive load using eye tracking technology in visual computing / Proceedings of the sixth workshop on Beyond Time and Errors on novel evaluation methods for visualization, BELIV ‘16. Baltimore MD USA, 24 October 2016. New York, NY: ACM Press, 2016. V. 24. P. 78.
  38. Fahimi R., Bruce N.D.B. On metrics for measuring scan path similarity // Behav. Res. Methods. 2021. V. 53. № 2. P. 609.
  39. Holland C., Komogortsev O.V. Biometric identification via eye movement scanpaths in reading / International Joint Conference on Biometrics. 11 October 2011. P. 1. https://api.semanticscholar.org/CorpusID:2528223.
  40. Goldberg J.H., Kotval X.P. Computer interface evaluation using eye movements: Methods and constructs // Int. J. Ind. Ergon. 1999. V. 24. № 6. P. 631.
  41. Yamada, Y., Kobayashi, M. Detecting mental fatigue from eye-tracking data gathered while watching video: Evaluation in younger and older adults // Artif. Intell. Med. 2018. V. 91. P. 39.
  42. Kliegl R., Rolfs M., Laubrock J., Engbert R. Microsaccadic modulation of response times in spatial attention tasks // Psychol. Res. 2009. V. 73. № 2. P. 136.
  43. Krejtz K., Duchowski A.T., Niedzielska A. et al. Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze // PLoS One. 2018. V. 13. № 9. P. e0203629.
  44. Zandi A.S., Quddus A., Prest L., Comeau F.J. Non-Intrusive detection of drowsy driving based on eye tracking data // Transp. Res. Rec. 2019. V. 2673. № 6. P. 247.
  45. Zemblys R., Niehorster D.C., Komogortsev O., Holmqvist K. Using machine learning to detect events in eye-tracking data // Behav. Res. Methods. 2018. V. 50. № 1. P. 160.
  46. Chen J.T., Kuo Y.C., Hsu T.Y., Wang C.A. Fatigue and arousal modulations revealed by saccade and pupil dynamics // Int. J. Environ Res. Public Health. MDPI. 2022. V. 19. № 15. P. 9234.
  47. Brezinova V., Kendell R.E. Smooth pursuit eye movements of schizophrenics and normal people under stress // Br. J. Psychiatry. 1977. V. 130. P. 59.
  48. Rottach K.G., Zivotofsky A.Z., Das V.E. et al. Comparison of horizontal, vertical and diagonal smooth pursuit eye movements in normal human subjects // Vision Res. 1996. V. 36. № 14. P. 2189.
  49. Ranti C., Jones W., Klin A., Shultz S. Blink rate patterns provide a reliable measure of individual engagement with scene content // Sci. Rep. 2020. V. 10. № 1. P. 8267.
  50. Marquart G., Cabrall C., de Winter J. Review of eye-related measures of drivers’ mental workload // Procedia Manuf. 2015. V. 3. P. 2854.
  51. Haq Z.A., Hasan Z. Eye-blink rate detection for fatigue determination / India International Conference on Information Processing (IICIP 2016), Delhi, India, 12-14 August, 2016. Proceedings IEEE Inc., 2017. P. 1.
  52. Horiuchi R., Ogasawara T., Miki N. Fatigue assessment by blink detected with attachable optical sensors of dye-sensitized photovoltaic cells // Micromachines (Basel). 2018. V. 9. № 6. P. 310.
  53. Tolvanen O., Elomaa A. P., Itkonen M. et al. Eye-tracking indicators of workload in surgery: A systematic review // J. Invest. Surg. 2022. V. 35. № 6. P. 1340.
  54. Marshall S.P. The index of cognitive activity: Measuring cognitive workload / Proceedings of the IEEE 7th Conference on Human Factors and Power Plants. 19 September, Scottsdale, AZ, USA, 2002. P. 7. doi: 10.1109/HFPP.2002.1042860.
  55. Duchowski A.T., Krejtz K., Krejtz I. et al. The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation / Proceedings of the Conference on Human Factors in Computing Systems (CHI ‘18). Association for Computing Machinery, New York, NY, USA, 2018. V. 282. P. 1. https://doi.org/10.1145/3173574.3173856
  56. Duchowski A.T., Krejtz K., Gehrer N.A. et al. The low/high index of pupillary activity / Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ‘20). Association for Computing Machinery, New York, NY, USA, 2020. P. 1. https://doi.org/10.1145/3313831.3376394
  57. Alnajar F., Gevers T., Valenti R., Ghebreab S. Calibration-free gaze estimation using human gaze patterns / Proceedings of the IEEE International Conference on Computer Vision. 1–8 Dec. 2013, Sydney, NSW, Australia, 2013. P. 137. doi: 10.1109/ICCV.2013.24.
  58. Rigas I., Economou G., Fotopoulos S. Biometric identification based on the eye movements and graph matching techniques // Patt. Recogn. Lett. 2012. V. 33. № 6. P. 786.
  59. Li J., Li H., Umer W. et al. Identification and classification of construction equipment operators’ mental fatigue using wearable eye-tracking technology // Autom. Constr. 2020. V. 109. P. 103000.
  60. Bitkina O.V., Park J., Kim H.K. The ability of eye-tracking metrics to classify and predict the perceived driving workload // Int. J. Ind. Ergonom. 2021. V. 86. P. 103193.
  61. Shiferaw B., Downey L., Crewther D. A review of gaze entropy as a measure of visual scanning efficiency // Neurosci. Biobehav. Rev. 2019. V. 96. P. 353.
  62. Deravi F., Biosignals S.G. Gaze trajectory as a biometric modality / Proceedings of the International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS-2011). 2011. P. 335. doi: 10.5220/0003275803350341
  63. Peißl S., Wickens C.D., Baruah R. Eye-Tracking measures in aviation: A selective literature review // Int. J. Aerosp. Psychol. 2018. V. 28. № 3–4. P. 98.
  64. Meghanathan R.N., Nikolaev A.R., van Leeuwen C. Refixation patterns reveal memory-encoding strategies in free viewing // Atten. Percept. Psychophys. 2019. V. 81. № 7. P. 2499.
  65. Fukushima K., Fukushima J., Warabi T., Barnes G.R. Cognitive processes involved in smooth pursuit eye movements: behavioral evidence, neural substrate and clinical correlation // Front. Syst. Neurosci. 2013. V. 7. P. 4.
  66. Knox P.C., Davidson J.H., Anderson D. Age-related changes in smooth pursuit initiation // Exp. Brain Res. 2005. V. 165. № 1. P. 1.
  67. Einhäuser W. The pupil as marker of cognitive processes / Computational and Cognitive Neuroscience of Vision. Cognitive Science and Technolog // Ed. Zhao Q. Springer, Singapore, 2017. P. 141.
  68. Richstone L., Schwartz M.J., Seideman C. et al. Eye metrics as an objective assessment of surgical skill // Ann. Surg. 2010. V. 252. № 1. P. 1772.
  69. Zargari Marandi R., Madeleine P., Omland Ø. et al. Eye movement characteristics reflected fatigue development in both young and elderly individuals // Sci. Rep. 2018. V. 8. № 1. P. 13148.
  70. Catalbas M.C., Cegovnik T., Sodnik J., Gulten A. Driver fatigue detection based on saccadic eye movements / 10th International Conference on Electrical and Electronics Engineering (ELECO). IEEE. 30 November – 2 December 2017. Bursa, Turkey, 2017. P. 913.
  71. Hu X., Lodewijks G. Exploration of the effects of task-related fatigue on eye-motion features and its value in improving driver fatigue-related technology // Transp. Res. Part F Traffic Psychol. Behav. 2021. V. 80. P. 150.
  72. Ahlstrom C., Nyström M., Holmqvist K. et al. Fit-for-duty test for estimation of drivers’ sleepiness level: Eye movements improve the sleep/wake predictor // Transp. Res. Part C Emerg. Technol. 2013. V. 26. P. 20.
  73. Abe T., Mishima K., Kitamura S. et al. Tracking intermediate performance of vigilant attention using multiple eye metrics // Sleep. 2020. V. 43. № 3. Р. zsz219.
  74. Di Stasi L.L., McCamy M.B., Macknik S.L. et al. Saccadic eye movement metrics reflect surgical residents’ fatigue // Ann. Surg. 2014. V. 259. № 4. P. 824.
  75. Di Stasi L.L., Renner R., Catena A. et al. Towards a driver fatigue test based on the saccadic main sequence: A partial validation by subjective report data // Transp. Res. Part C Emerg. Technol. 2012. V. 21. № 1. P. 122.
  76. Finke C., Pech L.M., Sömmer C. et al. Dynamics of saccade parameters in multiple sclerosis patients with fatigue // J. Neurol. 2012. V. 259. № 12. P. 2656.
  77. Renata V., Li F., Lee C.H., Chen C.H. Investigation on the correlation between eye movement and reaction time under mental fatigue influence / Proceedings of the 17th International Conference on Cyberworlds (CW 2018), Singapore 3-5 Oct 2018. Institute of Electrical and Electronics Engineers Inc., 2018. P. 207.
  78. Herlambang M.B., Taatgen N.A., Cnossen F. The role of motivation as a factor in mental fatigue // Hum. Factors. 2019. V. 61. № 7. P. 1171.
  79. Stone L.S., Tyson T.L., Cravalho P.F. et al. Distinct pattern of oculomotor impairment associated with acute sleep loss and circadian misalignment // J. Physiol. 2019. V. 597. № 17. P. 4643.
  80. Gergelyfi M., Jacob B., Olivier E., Zénon A. Dissociation between mental fatigue and motivational state during prolonged mental activity // Front. Behav. Neurosci. 2015. V. 9. P. 176.
  81. Schweitzer T., Wyss T., Gilgen-Ammann R. Detecting soldiers’ fatigue using eye-tracking glasses: Practical field applications and research opportunities // Mil. Med. 2022. V. 187. № 11-12. P. e1330.
  82. Borghini G., Astolfi L., Vecchiato G. et al. Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness // Neurosci. Biobehav. Rev. 2014. V. 44. P. 58.
  83. Dziuda Ł., Baran P., Zieliński P. et al. Evaluation of a fatigue detector using eye closure-associated indicators acquired from truck drivers in a simulator study // Sensors. 2021. V. 21. № 19. P. 6449.
  84. Schleicher R., Galley N., Briest S., Galley L. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? // Ergonomics. 2008. V. 51. № 7. P. 982.
  85. Hopstaken J.F., van der Linden D., Bakker A.B. et al. Shifts in attention during mental fatigue: Evidence from subjective, behavioral, physiological, and eye-tracking data // J. Exp. Psychol. Hum. Percept. Perform. 2016. V. 42. № 6. P. 878.
  86. Pomplun M., Sunkara S. Pupil dilation as an indicator of cognitive workload in human-computer interaction / Human-Centered Computing. Cognitive, Social, and Ergonomic Aspects // Eds. Harris D., Duffy V., Smith M., Stephanidis C. Boca Raton: CRC Press, 2019. V. 3. P. 542.
  87. Morad Y., Barkana Y., Zadok D. et al. Ocular parameters as an objective tool for the assessment of truck drivers’ fatigue // Accid. Anal. Prev. 2009. V. 41. № 4. P. 856.
  88. Di Stasi L.L., Marchitto M., Antolí A., Cañas J.J. Saccadic peak velocity as an alternative index of operator attention: A short review // Eur. Rev. Appl. Psychol. 2013. V. 63. № 6. P. 335.
  89. Diaz-Piedra C., Rieiro H., Suárez J. et al. Fatigue in the military: towards a fatigue detection test based on the saccadic velocity // Physiol. Meas. 2016. V. 37. № 9. P. N62.
  90. Ito J., Yamane Y., Suzuki M. et al. Switch from ambient to focal processing mode explains the dynamics of free viewing eye movements // Sci. Rep. 2017. V. 7. № 1. P. 1082.
  91. Pannasch S., Velichkovsky B.M. Distractor effect and saccade amplitudes: Further evidence on different modes of processing in free exploration of visual images // Vis. Cogn. 2009. V. 17. № 6-7. P. 1109.
  92. Velichkovsky B.M., Korosteleva A.N., Pannasch S. et al. [Two visual systems and their eye movements: A fixation-based event-related experiment with ultrafast fMRI reconciles competing views] // Sovrem. Tehnol. Med. 2019. V. 11. № 4. P. 7.
  93. Shoshina I.I., Shelepin Yu.E. Mechanisms of global and local analysis of visual information in schizophrenia. St. Petersburg: VVM Publishing House, 2016. 300 p.
  94. Shoshina I.I., Mukhitova Y.V., Tregubenko I.A. et al. Contrast Sensitivity of the Visual System and Cognitive Functions in Schizophrenia and Depression // Human Physiology. 2021. V. 47. № 5. P. 516.
  95. Milner A.D. How do the two visual streams interact with each other? // Exp. Brain Res. 2017. V. 235. № 5. P. 1297.
  96. Kunasegaran K., Ismail A.M.H., Ramasamy S. et al. Understanding mental fatigue and its detection: a comparative analysis of assessments and tools // Peer J. 2023. V. 11. P. e15744
  97. Tran Y., Craig A., Craig R. et al. The influence of mental fatigue on brain activity: Evidence from a systematic review with meta-analyses // Psychophysiology. 2020. V. 57. № 5. P. e13554.
  98. Hsu T.-Y., Hsu Y.-F., Wang H.-Y., Wang C.-A. Role of the frontal eye field in human pupil and saccade orienting responses // Eur. J. Neurosci. 2021. V. 54. P. 4283.
  99. Bafna T., Hansen J.P. Mental fatigue measurement using eye metrics: A systematic literature review // Psychophysiology. 2021. V. 58. № 6. P. e13828.
  100. Ansari M.F., Kasprowski P., Obetkal M. Gaze tracking using an unmodified web camera and convolutional neural network // Appl. Sci. 2021. V. 11. № 19. P. 9068.
  101. Naeeri S., Kang Z., Mandal S., Kim K. Multimodal analysis of eye movements and fatigue in a simulated glass cockpit environment // Aerospace. 2021. V. 8. № 10. P. 283.
  102. Mengtao L., Fan L., Gangyan X., Su H. Leveraging eye-tracking technologies to promote aviation safety – A review of key aspects, challenges, and future perspectives // Saf. Sci. 2023. V. 168. P. 106295.
  103. Hu X., Lodewijks G. Detecting fatigue in car drivers and aircraft pilots by using non-invasive measures: The value of differentiation of sleepiness and mental fatigue // J. Safety Res. 2020. V. 72. P. 173.
  104. Zhimin L., Ruilin L., Liqiang Y. et al. A benchmarking framework for eye-tracking-based vigilance prediction of vessel traffic controllers // Eng. Appl. Artif. Intell. 2024. V. 129. P. e107660.

Arquivos suplementares

Arquivos suplementares
Ação
1. JATS XML
Baixar
2. Fig. 1. Neural centers of eye movement control. A — subcortical centers of eye movement control, B — cortical centers of eye movement control: Visual Cortex — visual cortex, PEF — parietal visual field, FEF — frontal visual field, DLPFC — dorsolateral prefrontal cortex.

Baixar (245KB)
Metadados
3. Fig. 2. Dorsal (1) and ventral system (2) as the neural basis of spatial dynamic and object static vision.

Baixar (121KB)
Metadados

Declaração de direitos autorais © Russian Academy of Sciences, 2024