Assessment of mental workload states in monitors using functional near-infrared spectroscopy

Abstract

Background

As technology develops rapidly and complex systems become more widespread, the study of mental workload (MWL) for monitors has become increasingly important.

Objective

To explore the signaling features of functional near-infrared spectroscopy (fNIRS) in different MWL states.

Methods

Twenty participants performed the N-back task and fNIRS data were collected to measure changes in blood oxygenation levels in the prefrontal cortex. Additionally, performance metrics and NASA Task Load Index (NASA-TLX) questionnaire results were recorded.

Results

NASA-TLX scores increased progressively from 0-back (M = 8.750) to 1-back (M = 34.792), 2-back (M = 52.292), and 3-back (M = 75.625). Significant differences were observed in oxygenated hemoglobin (HbO) concentrations (F_{3, 57}= 10.557, p < 0.001) across these tasks. HbO increased from 0-back (M = -0.008) to 1-back (M = -0.002) and 2-back (M = -0.001), but decreased to 3-back (M = -0.002). Channels CH16, CH17, and CH18, located in the left dorsolateral prefrontal cortex (DLPFC), were significantly activated in the 1-back, 2-back, and 3-back conditions. CH1 and CH2, which are located in the right DLPFC, showed significant activation in both the 2-back and 3-back tasks.

Conclusions

Our results showed that different levels of the N-back tasks triggered different MWL states. Significant differences in HbO concentrations and activated channels were observed across tasks with varying MWLs. HbO followed an “inverted U” pattern, increasing as the MWL shifted from underload to normal, and subsequently decreasing as it reached overload. Additionally, the left DLPFC was activated earlier than the right DLPFC, with the latter gradually engaging as the MWL increased. This study provides valuable insights into the assessment of MWL states.

Keywords

mental workload monitors N-back functional near-Infrared spectroscopy dorsolateral prefrontal cortex blood oxygenation level

Introduction

With the rapid development and widespread application of computers, automation, and artificial intelligence technologies, the level of automation in human-machine systems has increased significantly. The division of labor between humans and machines has consequently changed substantially compared to the past, with human activities shifting from physical to mental labor.¹ This transformation is particularly evident in complex operational environments, such as nuclear power² and aviation,³ in which the primary role of human operators, often referred to as monitors in these contexts, is to monitor automated systems and make critical decisions when necessary. To achieve these tasks, monitors must process instrument data and information, combine them with the surrounding environment to allow for comprehensive analysis, and quickly make accurate decisions. The human brain is required to allocate cognitive resources during these tasks; as such, the mental workload (MWL) of monitors is receiving increasing attention.

MWL, as opposed to physical workload, reflects the degree to which cognitive resources are allocated during a tasks.⁴ MWL can encounter two types of issues: underload and overload. Underloaded MWL can lead to boredom, drowsiness, and reduced attention, resulting in a lack of monitoring and loss of control of the situation.⁵ However, if the MWL becomes overloaded, it can trigger higher stress levels, increased error rates, and even accidents.⁶ To summarize, improper MWL states can exert serious negative consequences. As such, assessing and managing MWL to maintain it within an appropriate range not only helps preserve situational awareness and task performance, but also enhances safety and user experience.

Generally, there are three methods for assessing MWL: task performance, subjective, and physiological measurement.⁷ Physiological measurements offer several advantages compared to the other two methods: they avoid recall bias, do not interfere with the primary task, and allow for real-time and continuous monitoring of MWL fluctuations during task execution.^8,9 Following advancements in sensor technology, an increasing number of physiological measurement methods of MWL detection have been developed, including electrocardiography, eye tracking, respiration measurement, skin conductance, electroencephalography, and functional near-infrared spectroscopy (fNIRS).⁹ Among these, fNIRS has been proven to offer many advantages, including improved safety, portability, non-invasiveness, minimal interference with tasks, strong spatial resolution, and wireless measurement.¹⁰ The theoretical basis of fNIRS is that the neural activity in the brain regions responsible for task execution intensifies when performing tasks with increased MWL, leading to changes in blood oxygen metabolism. Further, levels of oxygenated hemoglobin (HbO) increase, while deoxygenated hemoglobin (HbR) levels decrease, thereby altering the absorption of near-infrared light by the tissue. Through monitoring these changes in the brain, fNIRS can assess neuronal activity and infer MWL levels.¹¹

In recent years, a substantial body of research has utilized fNIRS to study MWL under various scenarios. For example, Midha et al. (2021) demonstrated the sensitivity of fNIRS in measuring changes in MWL during reading tasks, showing significant differences in left prefrontal cortex (PFC) activation under different reading conditions.¹² Geissler et al. (2021) used fNIRS to examine MWL during simulated driving, finding that activation of the PFC activation increased with MWL, and identifying the rostral part of the right middle frontal gyrus as a potential hub of MWL-related activity.¹³ Further, Hamann and Carstengerdes (2022) conducted a simulated flight experiment, the results of which revealed that higher task difficulty induced greater PFC activity and reduced frontal HbR concentrations.¹⁴ However, existing studies have not specifically defined the different MWL states, such as overload, normal load, and underload, nor did they explore blood oxygenation levels in the PFC under these states. Consequently, it is difficult to use these findings to determine which fNIRS signal features may indicate adverse MWL states requiring intervention. The present study therefore addresses this gap in the literature.

Methodology

Participants

This study was approved by the Institutional Review Board of our University. Twenty college students, comprising ten males and ten females ranging in age from 20 to 26 years (M = 24.3, SD = 2.7), were recruited as participants.

Experimental design

The N-back task, a commonly used experimental paradigm in cognitive neuroscience,¹⁵ was applied to induce different levels of MWL. In the N-back task, participants were required to continuously remember the last n stimuli (e.g., letters, numbers, or images) from a rapidly presented series, and to thereby determine whether the current stimulus matched that from n steps prior (Figure 1). The N-back and monitoring tasks were similar in at least two aspects: First, as in practical scenarios, monitors must maintain prolonged focus, while the N-back task requires participants to continuously track a series of stimuli, demanding sustained attention.¹⁶ Second, monitors must remember a large amount of information, such as different screen regions and event sequences, whereas the N-back task requires participants to remember the prior n stimuli.¹⁷

Figure 1.

Schematic overview of the N-back task.

A within-subjects experimental design was employed, with the independent variable being the number of letters the participants needed to recall, set to four levels: 0-, 1-, 2-, and 3-back. The dependent variables in this study included three aspects: (1) performance, measured as the mean reaction time (RT) and accuracy (ACC); (2) Subjective MWL, assessed using the NASA Task Load Index (NASA-TLX) scale¹⁸; (3) fNIRS signals, primarily focusing on HbO and HbR concentrations, as well as the activated channels.

Apparatus and materials

fNIRS signals were recorded using the NirSmart system, which utilized two wavelengths, 765 nm and 856 nm, to continuously record hemodynamic data from the PFC at a sampling frequency of 11 Hz. Seven light-emitting sensors and seven light receivers were attached to the forehead with distances of 3.5 cm between each sensor and receiver, thereby forming an 18-channel fNIRS signal layout, as outlined in Figure 2.

Figure 2.

Measurement and locations of the 18 channels.

The N-back experimental platform used in this study was programmed in Java (Figure 3), while the RT and ACC data were automatically calculated using the program. Participants were required to press the left arrow key on the keyboard when the current letter matched the one presented n rounds earlier, and to press the right arrow key when they did not match. The experiment was conducted on a ThinkPad X1 laptop, equipped with a 14-inch screen with a resolution of 2240 × 1400.

Figure 3.

The interface of the N-back task.

Experimental procedure

Each participant provided informed consent prior to initiation of the experiment, and were asked to learn the rules of the N-back task and complete a practice trial. Before the formal experiment began, the experimenter assisted the participants in positioning the fNIRS headgear, after which they rested for 3 min to establish a baseline measurement. Each trial was initiated with a 5-s instruction, notifying the participants of the upcoming task (i.e., 0-, 1-, 2-, or 3-back). The letters were subsequently presented every 3.5 s, with each letter displayed for 0.5 s, followed by 3 s on a blank screen (Figure 1). Twenty-two letters were presented in each trial, which had a total duration of 82 s. After each trial, participants filled out the NASA-TLX questionnaire. Upon completion, a cross was displayed on the screen for 60 s to allow the participants to relax before moving on to the next trial. Each participant completed four trials and the experimental order for the different conditions was assessed using a Latin square.

Data analysis

The fNIRS data underwent preprocessing to ensure the accuracy of the analysis. A band-pass filter with a frequency range of 0.01–0.20 Hz was applied to eliminate high-frequency noise, baseline drift, and physiological interference. The modified Beer-Lambert law was also applied to convert the light intensity signals from the two wavelengths in each channel into signals that reflect the concentrations of HbO and HbR. Data were normalized to the baseline data.

Then, statistical analysis was performed using SPSS software. Means and standard deviations were calculated to present the descriptive statistics for RT, ACC, NASA-TLX scores, and HbO and HbR concentrations. Repeated-measures analysis of variance (ANOVAs) were conducted to examine the effect of different N-back task levels on the dependent variables. T-tests were used to analyze whether the HbO concentrations in different channels significantly differed from baseline across task levels. Statistical significance was set at p < 0.05.

Results

Performance

Figure 4 shows the results of the task performance in the N-back task. ANOVA revealed a significant effect of N-back task levels on both RT (F_{3, 57}= 33.545, p < 0.001) and ACC (F_{3, 57}= 4.479, p = 0.009). As the difficulty of the N-back task increased, RT increased, with values for the 0- to 3-back conditions calculated as 0.363, 0.481, 1.077, and 1.213, respectively (Figure 4(a)). At the same time, ACC decreased, with corresponding values of 0.982, 0.948, 0.925, and 0.863, respectively (Figure 4(b)). These data indicate that the different N-back tasks elicited different levels of MWL.

Figure 4.

Average RT (a) and ACC (b) of the participants in the different N-back tasks.

Subjective MWL

Figure 5 shows the subjective MWL measured by the NASA-TLX for different N-back conditions. The ANOVA results indicated a significant effect of the N-back level on subjective MWL (F_{3, 57}= 21.125, p < 0.001), which increased progressively from 0-back (M = 8.750, SD = 2.204) to 1-back (M = 34.792, SD = 7.988), 2-back (M = 52.292, SD = 8.881), and 3-back (M = 75.625, SD = 4.534).

Figure 5.

NASA-TLX scores for the different N-back tasks.

According to a prior study by Vu Thanh Nguyen, a NASA-TLX score below 10 for the 0-back task indicates an underload state, while a mean score above 75 for the 3-back task suggests an overload state.¹⁹ In addition, the NASA-TLX scores for the 1-back and 2-back tasks fall between 25 and 75, indicating normal MWL, with the 1-back task representing lower MWL.¹⁹

fNIRS signals

Table 1 shows the average HbO and HbR concentrations in all channels under different N-back conditions. The ANOVA results indicated a significant effect of N-back levels on HbO concentration (F_{3, 57}= 10.557, p < 0.001). As task difficulty increased, the HbO followed an “inverted U” pattern, initially rising from 0-back to 2-back, but decreasing from 2-back to 3-back. No significant effect of the N-back level on HbR concentration was found (F_{3, 57}= 1.277, p = 0.299).

Table 1.

Average HbO and HbR for the different N-back tasks.

	0-back	1-back	2-back	3-back
	M (SD)	M (SD)	M (SD)	M (SD)
HbO	−0.008 (0.017)	−0.002 (0.009)	−0.001 (0.008)	−0.002 (0.005)
HbR	0.002 (0.022)	−0.001 (0.009)	0.000 (0.004)	0.000 (0.005)

The HbO and HbR concentration curves for each N-back condition and channel were visualized using the NIRspark software. In Figure 6, thicker black lines represent the average HbO and HbR concentrations across all 18 channels, whereas the thinner colored lines represent the HbO and HbR concentrations in each channel.

Figure 6.

Hbo and HbR across the four N-back conditions.

The HbO results indicated no significant activity across channels in the 0-back condition. Notably, in the 1-back and 2-back conditions, most channels showed a substantial increase in HbO concentration compared with the 0-back condition. However, during the 3-back task, the average HbO concentration in most channels was lower than that at the corresponding time points during the 1-back and 2-back tasks. Additionally, in the 3-back task, HbO levels in most channels initially increased, peaked, and then declined subsequently.

HbR concentration showed no significant activity across channels in the 0-back condition. Under the other three conditions, the average HbR in some channels exhibited a downward trend over time.

As there was minimal activity across channels in the 0-back condition, a t-test was conducted on HbO concentrations from the 18-channels under the 1-back to 3-back conditions to explore the activated brain regions. The results (Table 2) showed that under the 1-back to 3-back condition, CH16, CH17, and CH18, all located in the left hemisphere, were significantly activated. Additionally, in the 2- and 3-back conditions, CH1 and CH2, located in the right hemisphere, were significantly activated.

Table 2.

Activated channels in different N-back conditions.

Channel	1-back		2-back		3-back
Channel	t	p	t	p	t	p
CH 1	-	-	1.247	0.039	2.252	0.028
CH 2	-	-	1.946	0.046	2.751	0.024
CH 16	3.830	0.014	6.216	< 0.001	3.735	0.018
CH 17	3.320	0.032	6.382	< 0.001	4.342	0.009
CH 18	2.382	0.048	5.199	0.002	4.324	0.011

Discussion

Overall, this study utilized the NASA-TLX scale to determine the MWL states corresponding to different N-back tasks: the 0-back task was associated with an underloaded MWL state; the 1-back and 2-back tasks were associated with a normal MWL state; and the 3-back task was associated with an overloaded MWL state. These findings are further supported by the performance data.

The fNIRS results indicated that the activity of the dorsolateral prefrontal cortex (DLPFC) activity followed an inverted U-shaped pattern. As the MWL increased from an underloaded level (i.e., 0-back) to a normal level (i.e., 1-back and 2-back), the HbO concentration increased, reflecting increased DLPFC activity. However, under the overload condition (i.e., 3-back task), the HbO concentration began to decrease, indicating reduced DLPFC activity. The DLPFC is a key anatomical region associated with the central executive system and plays a critical role in task monitoring and cognitive control.²⁰ During the early stages of increased MWL, HbO levels in the DLPFC increased, reflecting heightened brain activity to meet the cognitive demands of the task, and leading to increased oxygen consumption and blood flow. However, when the MWL exceeded the brain's compensatory capacity, HbO levels in the DLPFC declined, indicating that the brain's functional limits have been reached or exceeded.

Additionally, channels in the left DLPFC (CH16, CH17, and CH18) were activated earlier than those in the right DLPFC. As task difficulty increased, channels in the right DLPFC (CH1 and CH2) became significantly activated, possibly because the left hemisphere is the dominant hemisphere, as people tend to engage the left DLPFC first when processing tasks. However, when MWL reaches higher levels, the non-dominant hemisphere (i.e., the right side) can be recruited as a compensatory resource.²¹

This study further revealed differences in oxygenation response patterns and channel activation levels in the DLPFC under varying MWL conditions, particularly in the overloaded state, thus providing valuable insights for future efforts to assess MWL using fNIRS, which could help to enable timely interventions and strategies to enhance work efficiency and ensure operational safety. Further, these results contribute to a deeper understanding of the activation characteristics of different brain regions under various MWL conditions.

However, this study has several limitations. First, it lacked further exploration and explanation of the temporal characteristics of DLPFC activation, which could provide valuable insights into the dynamic processes of brain activity. Future research should focus on analyzing the time-based relationship of DLPFC activation to better understand these dynamics. Additionally, while the N-back task is a widely used cognitive paradigm and shares some similarities with monitoring tasks, it may not fully represent the broader range of real-world tasks that involve complex attention management and decision-making. Therefore, the conclusions of this study should be generalized with caution. Future research should explore the cognitive processes and neural mechanisms in real-world monitoring contexts (e.g., nuclear power or aviation) to identify fNIRS features that are more applicable in practical settings. Finally, the participants in this study were students aged 20–26, a relatively narrow demographic whose cognitive abilities may not be representative of the general population. To enhance the generalizability of the findings, future studies should include participants from a broader range of ages and occupations.

Conclusion

The findings of the present study revealed that different N-back task levels elicited varying MWL states. Significant differences were observed in HbO concentrations and activated channels across tasks with varying MWL states. HbO further exhibited an “inverted U” pattern, increasing as MWL shifted from underload to normal, and decreasing as MWL reached overload. Moreover, the left DLPFC was activated earlier than the right DLPFC, reflecting hemispheric dominance. Finally, as the MWL increased, the right DLPFC gradually became activated. Overall, this study confirmed the utility of fNIRS to assess the MWL state in monitors.

Footnotes

Acknowledgements

We thank the associate editor and the reviewers for their useful feedback that improved this paper.

Ethical approval

This study was approved by the Institutional Review Board of Tsinghua University and complied with all relevant ethical regulations.

Informed consent

Informed consent was obtained from all participants involved in this study.

Funding

This work was supported by the Foundation of National Key Laboratory of Human Factors Engineering (Grant No. HFNKL2023J09) and the National Natural Science Foundation of China (Grant No. 72171134).

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Kakkos

Dimitrakopoulos

Sun

, et al. EEG Fingerprints of task-independent mental workload discrimination. IEEE J Biomed Health Inform 2021 Oct; 25: 3824–3833.

Liu

Jia

, et al. Using artificial neural networks for predicting mental workload in nuclear power plants based on eye tracking. Nucl Technol 2020 Jan 2; 206: 94–106.

Xie

, et al. Evaluating mental workload during multitasking in simulated flight. Brain Behav 2022; 12: e2489.

Longo

Wickens

Hancock

, et al. Human Mental Workload: A Survey and a Novel Inclusive Definition.Front Psychol [Internet]. 2022 Jun 2 [cited 2024 Sep 2]; 13: 883321. Available from: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2022.883321/full

Cañas

Ferreira

de Frutos

, et al. Mental workload in the explanation of automation effects on ATC performance. In: Longo

Leva

(eds) Human mental workload: models and applications. Cham: Springer International Publishing, 2019, pp.202–221.

Longo

. Experienced mental workload, perception of usability, their interaction and impact on task performance. PLOS ONE 2018 Aug 1; 13: e0199661.

Tao

Tan

Wang

, et al. A systematic review of physiological measures of mental workload. Int J Environ Res Public Health 2019 Jan; 16: 2716.

. Subjective and objective assessments of mental workload for UAV operations. Work 2022 Jan 1; 72: 291–301.

Charles

Nixon

. Measuring mental workload using physiological measures: a systematic review. Appl Ergon 2019 Jan 1; 74: 221–232.

10.

Herff

Heger

Fortmann

, et al. Mental workload during n-back task—quantified in the prefrontal cortex using fNIRS. Front Hum Neurosci [Internet]. 2014 Jan 16 [cited 2024 Sep 2]; 7: 935. Available from: https://www.frontiersin.org/journals/human-neuroscience/articles/10.3389/fnhum.2013.00935/full

11.

Peck

Afergan

Yuksel

, et al. Using fNIRS to measure mental workload in the real world. In: Fairclough

Gilleade

(eds) Advances in physiological computing [internet]. London: Springer, 2014 [cited 2024 Sep 2], pp.117–139. Available from: https://doi.org/10.1007/978-1-4471-6392-3_6

12.

Midha

Maior

Wilson

, et al. Measuring mental workload variations in office work tasks using fNIRS. Int J Hum-Comput Stud 2021 Mar 1; 147: 102580.

13.

Geissler

Schneider

Frings

. Shedding light on the prefrontal correlates of mental workload in simulated driving: a functional near-infrared spectroscopy study. Sci Rep 2021 Jan 12; 11: 705.

14.

Hamann

Carstengerdes

. Investigating mental workload-induced changes in cortical oxygenation and frontal theta activity during simulated flights. Sci Rep 2022 Apr 19; 12: 6449.

15.

Wang

Gwizdka

Chaovalitwongse

. Using wireless EEG signals to assess memory workload in the n-back task. IEEE Trans Hum-Mach Syst 2016 Jun; 46: 424–435.

16.

Frost

Moussaoui

Kaur

, et al. Is the n-back task a measure of unstructured working memory capacity? Towards understanding its connection to other working memory tasks. Acta Psychol (Amst) 2021 Sep 1; 219: 103398.

17.

Scharinger

Prislan

Bernecker

, et al. Gamification of an n-back working memory task – is it worth the effort? An EEG and eye-tracking study. Biol Psychol 2023 Apr 1; 179: 108545.

18.

Hart

. Nasa-Task load Index (NASA-TLX); 20 years later. Proc Hum Factors Ergon Soc Annu Meet 2006 Oct 1; 50: 904–908.

19.

Vu Thanh Nguyen

. An Investigation of Three Subjective Rating Scales of Mental Workload in Third Level Education. Dissertations [Internet] 2019 Jan 1. Available from: https://arrow.tudublin.ie/scschcomdis/178

20.

Klaus

Schutter

DJLG

. The role of left dorsolateral prefrontal Cortex in language processing. Neuroscience 2018 May 1; 377: 197–205.

21.

Lundstrom

Ingvar

Petersson

. The role of precuneus and left inferior frontal cortex during source memory episodic retrieval. NeuroImage 2005 Oct 1; 27: 824–834.