Detection of primary RGB colors projected on a screen using fNIRS

In this study, functional near-infrared spectroscopy (fNIRS) is utilized to measure the hemodynamic responses (HRs) in the visual cortex of 14 subjects (aged 22–34 years) viewing the primary red, green, and blue (RGB) colors displayed on a white screen by a beam projector. The spatiotemporal characteristics of their oxygenated and deoxygenated hemoglobins (HbO and HbR) in the visual cortex are measured using a 15-source and 15-detector optode configuration. To see whether the activation maps upon RGB-color stimuli can be distinguished or not, the t-values of individual channels are averaged over 14 subjects. To find the best combination of two features for classification, the HRs of activated channels are averaged over nine trials. The HbO mean, peak, slope, skewness and kurtosis values during 2–7s window for a given 10s stimulation period are analyzed. Finally, the linear discriminant analysis (LDA) for classifying three classes is applied. Individually, the best classification accuracy obtained with slo...


Introduction
Color is a fundamental aspect of human perception and also a®ects the human spirit. It should come as no surprise, then, that color stimulation and detection in the visual cortex have been the subjects of long-standing study and debate. [1][2][3][4][5][6][7][8][9][10][11][12][13] The objective of the present study is to characterize appropriate features of the hemodynamic response (HR) signals obtained from the visual cortex upon color stimuli. Five di®erent features (i.e., mean, slope, peak, skewness, and kurtosis) of the obtained HR signals are examined in order to classify the three di®erent color stimuli (i.e., red, green, and blue (RGB)).
Scientists have investigated visual responses via color stimuli using various modalities such as functional magnetic resonance imaging (fMRI), electroencephalography (EEG), and functional nearinfrared spectroscopy (fNIRS), among others. The works on fMRI-and EEG-based color detection are extensive and various, though the literature on fNIRS is sparse.
The work of Xiao et al. 1 on cortical activation in response to chromatic stimuli revealed a topographic representation of chromaticity. They found that upon color stimulation, a cortical response is evoked in the visual cortex, and that this usually occurs as a result of the°ickering of various colors with gray, speci¯cally red-gray, green-gray, bluegray and yellow-gray combinations. Their results showed that based on color blobs in the monkey visual cortex, the red-green and red-gray°ickers produced the strongest responses. Lu and Roe 2 used optical imaging to map color-selective responses in V1-V2 and Tanigawa et al. 3 in V4. Fine alignment of optical maps and cytochrome-oxidase (CO)stained tissue revealed that color domains in V1 strongly associate with CO blobs. They also found that color domains in V1 align along the centers of ocular dominance columns. Another fMRI study done by Goddard et al. 4 showed greater blood-oxygenation-level-dependent (BOLD) activation for certain cone-opponent channels, whose¯nding suggests integration of those channels as early as V1. However, Brouwer and Heeger 5 found that the transition from cone-opponent color coding to perceptual color coding began at V1 and that the activations were strongest at V4 and VO1. Similarly, Parkes et al., 6 having measured the fMRI BOLD responses to°ickering radial patterns composed of perceptual colors (red, green, yellow, blue), found that the BOLD responses evoked were similar, even though the individual di®erences with respect to each color pattern were su±ciently reliable to predict the color viewed. Mullen et al., 7 having measured the fMRI BOLD response in the visual cortex to radial chromatic red/green and yellow/ blue grating patterns di®ering either in cone activation or in multiples of detection threshold, determined that the amplitude of the BOLD response was not linearly related to either measure. In their later investigation of the response of early and extra-striate visual areas to color, speci¯cally highcontrast red-green (RG), they found that the most evident in the early visual areas (V1 and V2), but selective responses, revealed as greater adaptation between the same stimuli than cross-adaptation between di®erent stimuli, emerged in the ventral cortex, in V4 and VO in particular. 8 Laeng et al., 9 working with a cohort of synaesthesia patients viewing both chromatic letters and illusory colors, measured the color di®erences in RGB and CIExyY color spaces, and identi¯ed a correlation between the BOLD activation and the color di®erence calculated using either color space. About human visual cortex research, Kuriki et al. 10 applied a deriving histogram of hue-selective voxels measured using fMRI with a novel stimulation paradigm. They demonstrated that there exists a wide spectrum of hue selectivity in early visual cortex.
As for EEG research, Tang and Zhang, 11 after conducting a brain-activity-based color-determination study, reported that the dispositive brain activity is a function of color energy and frequency. Rasheed and Marini, 12 applying an EEG modality to the classi¯cation of RGB-color stimuli, classi¯ed three visual conditions to 84%, 89% and 98% accuracies with linear, polynomial, and radial basis function kernels, respectively. Alharbi et al. 13 similarly evaluated a single-trial classi¯cation model for RGB-color-stimulus-evoked EEG signals, and determined that the empirical mode decomposition residual provides the most accurate, fastest, and most reliable classi¯cation (average accuracy: 88.5%, execution time: 14 s).
fNIRS is a noninvasive, optical imaging technique for continuous monitoring of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR). 14-33 It can provide both topographic 15,16,[34][35][36] and tomographic 14,37 brain images. Speci¯cally, fNIRS monitors regional cerebral blood°ow (rCBF) variations by measuring, at wavelengths between 700 nm and 1000 nm, the near-infrared-lightabsorption changes caused by concentration variations of HbO and HbR, the two primary absorbing chromophores in brain capillary blood. 24,38 Likewise, fNIRS has been used to measure¯ngeropposition activation of the cerebral cortex. 39 The same¯nger-opposition task also has been shown to activate the primary motor cortex. A relevant previous work indicated that normal subjects, during the performance of a verbal°uency task, showed activation of the left superior parietal cortex. Right frontal activation during testing of continuous performance also has been reported. 40 These studies' determined HbO and HbR timecourse changes in the relevant activated cerebral cortices are consistent with the present results; that is, during cortical activation, HbO increases and HbR decreases.
Classi¯cation as a process for distinguishing data classes 41 entails two steps: feature selection and classi¯er execution. 42 As features for distinguishing di®erent stimuli, the mean, peak, slope, kurtosis, and skewness values of HbO signals for individual trials, as averaged over all channels, have been adopted. As a classi¯cation algorithm, the linear discriminant analysis (LDA) technique that separates data into two or more classes have been widely utilized in various fNIRS applications involving the detection of drowsiness 43 and mental workload, 44 as well as in a hybrid brain-computer interface (BCI) applications using both fNIRS and EEG. 45 The usability of these algorithms has been well established in the literature; see, for example, relevant review papers 46,47 and multi-class problems. 48,49 In the present study, fNIRS is applied to detect the visual cortex's HR to primary RGB-color stimuli, and the correlation between those responses and the various stimuli are explored. Noise-removal preprocessing and statistical analysis are used to enhance the classi¯cation accuracy. Then, the HbO signal is used via the three-class classi¯er LDA employing the mean, peak, slope, skewness, and kurtosis values as features to distinguish the RGB colors. The best two-feature combination is turned out to be peak and skewness. This indicates signi¯cantly that the colors detected by fNIRS can be di®erentiated using the above-mentioned features. To the best of our knowledge, this is the¯rst fNIRS work on color detection in the human visual cortex.

Materials and Methods 2.1. Subjects
A total of 14 healthy volunteers (10 males and 4 females, aged 24-31 years, two left-handed) participated in the experiment. None of them had any history of neurological disorder. All of them were informed about the experiment and provided a written consent according to the latest Declaration of Helsinki. During the experiment, the subjects were asked to sit comfortably and remain motionless as much as possible. A projector screen was  Detection of primary RGB colors placed in front of them at a distance of approximately 1 m. The subjects were asked to keep their eyes during both the test and rest periods. Also, they were instructed to keep their blinking to the minimal level.

fNIRS data
The data were acquired with a continuouswave NIRS imaging system (Dynamic Near-infrared Optical Tomography (DYNOT); NIRx Medical Technologies, USA) at a sampling rate of 1.81 Hz. The system emits laser light at di®erent wavelengths (760 nm and 830 nm) from each source. The optodes con¯guration illustrated in Fig. 1 indicates the channel distribution and measurement locations. The distance between adjacent optodes is 27 mm. The source-emitter pairs are positioned above the visual cortex (the Oz position, according to the international 10-20 system). During the experiment, all lights were eliminated in order to minimize signal contamination. The measured intensity data of the two wavelengths were converted to relative HbO and HbR concentration changes using the modi¯ed Beer-Lambert law given by [50][51][52] Ác HbO ðtÞ where ÁA is the optical density variation of the light emitted at wavelength j , HbX ð j ) is the extinction coe±cient of HbX in M À1 mm À1 , d is the unit-less di®erential pathlength factor (DPF), and l is the emitter-detector distance (in mm  Figure 2 provides the experimental paradigm used in this work. As indicated, one task block consists of each display of RGB colors, and the rest block

Mean-Peak
Mean-Slope (0.033 Hz) and cardiac noises (1 Hz) contained in HRs were removed by a low-pass¯lter working at a cut-o® frequency 0.15 Hz. Second, the trend of the signal (e.g., low-frequency drift) was removed from the time-series data using the de-trending technique [55][56][57] (the detrend function in MATLAB TM ). Third, the MATLAB TM function mat2 gray was used to normalize the signal data within the 0-1 range.

Activation map
Plotting of the cortical activation map is the most important step in fNIRS data analysis. This map is an intuitive display of brain cortex activity. The t-value-based activation map represents the ratio of the weighting coe±cient to the modeled HR. A high t-value indicates that the measured signal is highly correlated with the modeled HR. In the present study, the t-values were calculated for the averaged trials of individual color stimuli using the robust¯t function available in MATLAB TM . 56-58 Figure 3 shows activation maps based on RGBcolor stimuli average over 14 subjects.

Feature selection and classi¯cation
For feature selection and classi¯cation, we used the HbO signals averaged over the channels whose t-value was higher than a critical value, which was set to 1.6736. 57 To improve the classi¯cation accuracy and to reduce the classi¯cation time, the scheme whereby only a 2-7 s window within the overall 10-s task period was adopted. 22 Then, the mean, peak, slope, skewness, and kurtosis values were computed. The mean was obtained by averaging of 5 s window data points; the slope was calculated using the poly¯t function in Matlab TM , and the skewness and kurtosis were calculated using Matlab TM 's skewness and kurtosis functions. In the classi¯cation, we used the LDA classi¯er, which is to say the classify function available in Matlab TM , to verify the individual HbO signals. 22,54,[58][59][60][61][62] 3. Results Figure 3 is the activation map averaged over the 14 subjects. The obtained t-values are displayed in a color scale for more e®ective illustration of regional activation. The color bar indicates the signal intensity and the color in a pixel represents the pixel's t-value, respectively. Figure 4 plots the HbOs averaged over 14 subjects with the standard deviations for the RGB-color stimuli. It clearly di®erentiates the peak values for individual color stimuli: the maximum peak value, 0.1816, is the green-color stimulus; 0.1642 is the red-color stimulus; the smallest peak value, 0.1405, is the blue-color stimulus. Figure 5 shows the 2D feature spaces representing all of the two-feature combinations among the mean, peak, slope, skewness and kurtosis, respectively, for Subject 1. Figure 6 and Table 1 plot and list, respectively, the subjects' two feature combination classi¯cation accuracies as averaged over all activation channels and trials. All, notably, were above the chance level (33.33%). The best averaged accuracy, 55.29%, was achieved with the peak-skewness feature combination for 14 data (Fig. 7).

Discussion
The fNIRS technique was employed, in this study, owing to its high potential as a neuro-imaging tool. Speci¯cally, it was utilized to examine the brain activities' spatial-and temporal-distribution information. Figure 3 depicts, in the form of a t-map, the spatial distribution together with the expected HR induced by a given stimulus (i.e., RGB-color stimulus). [63][64][65] However, as the signal strength in the target brain area cannot be revealed on a t-map, and the temporal magnitude of the HR must also be taken into account. Also, in Fig. 3, evident are the brain activation di®erences according to the three RGB-color stimuli. Figure 3(a) depicts that the red-color stimulus induces more activity in the upper-right visual cortex; Fig. 3(b) indicates that upon green color stimulus there is brain activity on both sides of the visual cortex, but more on the right side than on the left; Fig. 3(c) shows that whereas there is no activity on the right side, there are, in fact, a number of active channels in the lower-left visual cortex. The peak HR values plotted in Fig. 4, meanwhile, distinguish the RGB-color stimuli as well: 0.1816 (green), 0.1642 (red), and 0.1405 (blue).
The intuitive results plotted in Fig. 7 show the peak-skewness combination to be the best feature selection for RGB-color classi¯cation (highest accuracy: 55.29%). Although the HR and, thus, the classi¯cation accuracies varied by subjects (due to individual di®erences), none of the subjects showed an average classi¯cation accuracy below the chance level. Variations in classi¯cation accuracy potentially can be reduced using the optimum activation period under better environmental conditions to increase the number of trials.
There were some limitations observed during the experimental study. First, potential anomalies were evoked by sensory activity and attention/anticipation. Eye blinking, for example, incurs°uctuation in neuronal activity and peripheral HR. Second, subjects' hair generates noise in the data. In order to compensate for this problem, we recruited subjects with short hair. Taking these issues into account and avoiding them will further improve experimental results.

Conclusion
This study investigated the feasibility of detecting the visual cortex responses to RGB-color stimuli using fNIRS. RGB colors were detected by monitoring the visual cortex activity by means of t-map and¯ve features (the mean, peak, slope, skewness, and kurtosis values). The t-map results showed the right visual cortex to be more active during the redand green-color stimuli and the left visual cortex to be more active during the blue-color stimulus. According to the three-class classi¯cation results, all of the average classi¯cation accuracies were above the chance level (33.33%), and the highest accuracy, for Subject 1, was 74.07%. The best two feature combination was the peak-skewness combination, which yielded the highest accuracy, 55.29 AE 9.58%, as averaged over 14 subjects. The overall results suggest that fNIRS-based detection of RGB-color stimuli in the human visual cortex is feasible.