1.

Introduction

Functional near-infrared spectroscopy (fNIRS) is an emerging functional imaging technique with fewer physical restrictions, greater practicality, and better portability than other imaging techniques.¹ Using fNIRS, brain functional activity can be assessed by recording the concentration of both oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) with relatively high temporal resolution (e.g., 10 Hz). In recent decades, task activation studies based on fNIRS have been extensively carried out in the field of cognitive and clinical neuroscience,^{2, 3, 4, 5, 6} and the results have been demonstrated to be reliable.^{7, 8, 9}

Recently, fNIRS has been adopted to investigate resting-state (i.e., task-free) brain function by White ¹⁰ and our group.^{11, 12, 13} During rest, the human brain intrinsically fluctuates with a low-frequency (<0.1 Hz) character.^{14, 15} These low-frequency fluctuations (LFFs) are considered to be information rich,^{16, 17, 18} and the synchronization of LFFs within widely distributed neuroanatomical systems [i.e., resting-state functional connectivity (RSFC)] is believed to reflect a pivotal functional architecture of the brain.^{19, 20, 21} On the basis of resting-state fNIRS (rs-fNIRS), RSFC was successfully detected within the sensorimotor,^{10, 11, 12} visual,^{10, 12} auditory,¹¹ and language systems,¹³ as well as within whole-brain networks.^{22, 23} Because rs-fNIRS experiments can be carried out with few requirements to the subjects, they are particularly suitable for studying special populations (e.g., neonates, infants, or hospital patients), therefore contributing to better understanding of human development and rehabilitation procedures. Furthermore, due to its low cost, rs-fNIRS is quite suitable for repetitive or continuous recording in brain plasticity studies.

However, an essential question remains to be elucidated: is the RSFC detected based on rs-fNIRS measurements and then proposed as a biomarker in previous studies reliable? In investigation of a scientific problem, reliability indicates to what extent we can trust our result. If it is not reliable, then the interpretation, inference, comparison, and integration of this result must be conducted with caution. Unfortunately, several factors may have a negative effect on the reliability of the fNIRS-based RSFC. First, the resting-state paradigm, in which subjects are instructed to keep still and relax their mind, is only a descriptive experimental paradigm without a specific task engagement or operative instruction, which makes it inherently unconstrained.²⁴ Therefore, experimental variations related to time and participants exist (e.g., different subjects have different moods during scanning, and the thoughts of a subject may vary at times), which impacts reliability. Second, the performance of fNIRS machines and the scanning environment (e.g., the illumination level) may also introduce variations between scanning sessions, which may also reduce reliability. Third, the location of the fNIRS optode may vary, though perhaps not significantly, across participants and scanning sessions³ and will thus reduce reliability.²⁵ Finally, other influencing factors, such as noise level and calculating parameters, may also reduce the reliability of RSFC result. Considering these factors, it is unclear whether RSFC findings based on rs-fNIRS are reliable enough to be adopted as biomarkers in cognitive neuroscience and clinical study. Thus, addressing such a question is of preeminent importance in the field of rs-fNIRS.

In this study, RSFC in the sensorimotor areas was calculated by using the seed-correlation approach utilized by the previous rs-fNIRS studies,^{10, 11, 13, 22} and its reliability was comprehensively assessed via a test-retest experiment. On the basis of the findings, practical guidance was provided for future fNIRS-based RSFC study.

2.

Materials and Methods

2.2.

Protocols and fNIRS Measurements

In both sessions, each subject underwent an 11-min resting-state scan and a subsequent 6-min bilateral finger-tapping task scan. During the resting-state experiment, subjects were required to be seated in a chair and to keep still with eyes closed. In the finger-tapping task, subjects tap their bilateral index fingers in different sequences as indicated by visual instruction in a block-design experiment (see detailed descriptions in Lu ¹¹). The fNIRS scanning was conducted with a 52-channel ETG-4000 Optical Topography System (Hitachi Medical Company, Tokyo) with 17 emitters and 16 detector optodes (interoptode distance = 30 mm) in a holder cap located above the sensorimotor area, based on the international 10–20 system for electroencephalogram electrode placement (the optode between channels 47/48 was in Cz, and channels 32 and 42 were placed in T3 and T4, respectively). The channel locations were illustrated in Fig. 1a. The absorption of near-infrared light at two wavelengths (695 and 830 nm) was measured with a 10-Hz sampling rate. The HbO and HbR hemoglobin signals were computed with the modified Beer–Lambert law,²⁷ and their sum produced the total hemoglobin (HbT) concentration signal.

Fig. 1

Seed-region definition: (a) labeling map of the sensorimotor areas for all channels and (b) the averaged group-level task-activation map with normalized t-values ranging from –1 to 1 across two fNIRS recording sessions and three different hemoglobin concentration signals. The channels in white circles represent the seed channels.

3.

Results

Figure 2 shows the group-level RSFC (t-maps) derived from step 5 in the RSFC calculation for the HbO, HbR, and HbT signals (first three rows, respectively) and for two sessions (left and right panels). Both were symmetrically distributed within the bilateral sensorimotor areas. The spatial patterns of the group-level RSFC from two sessions were highly similar, and both resembled the predefined sensorimotor region-of-interest [(preROI), last row of Fig. 2]. The t-statistical values, measuring the group-level RSFC strength, were generally higher for the HbO and HbT than those for the HbR signals.

Fig. 2

Group-level RSFC maps for the HbO, HbR, and HbT signals (first three rows) derived from session 1 (left panel) and session 2 (right panel), together with the predefined sensorimotor region-of-interest (last row). Note that the value of each channel in those group-level RSFC maps equals the original t-value generated by the group analyses without normalized to –1 to 1.

3.1.

Mapwise Reliability Assessment

To quantitatively evaluate the similarity between the group-level RSFC maps for two sessions, we plotted the group-level RSFC strength (t value) derived from session 1 against the corresponding RSFC strength from session 2 for each channel, producing the scatter plots depicted in Fig. 3. The scatter plots for the HbO, HbR, and HbT signals are depicted from left to right, respectively. Each data point represents the strength of the group-level RSFC at a single channel. Good-to-excellent mapwise reliability is indicated by the data points closely distributed near the fitted line with the spatial correlation coefficients r = 0.78, 0.70, and 0.88 for the HbO, HbR, and HbT signals, respectively. It is noteworthy that all the significantly functional connected channels (at the top right of the two gray threshold lines with p < 0.01, uncorrected) for the HbO and HbT signals, and most for the HbR signals, were within the preROI (depicted by red dots). Such a result indicates our justifiable preROI definition, as well as the good quality of our rs-fNIRS data.

Mapwise reliability was also evaluated at the individual level. Table 1 summarizes the spatial correlation coefficients (r) between individual RSFC maps derived from two sessions for all 16 subjects and for the three hemodynamic parameters. For HbO, of the total 16 subjects, four showed excellent, nine with good, two with fair, and one with poor reliability. In this case, the reliability can be considered to be good to excellent. For HbR, there was one with excellent, three with good, eight with fair, and four with poor reliability. In this case, the reliability can be considered to be fair. For HbT, reliability data demonstrates three with excellent, four with good, five with fair, and four with poor. From this, it is difficult to classify the reliability.

Fig. 3

Scatter plots of the group-level RSFC for the (left) HbO, (middle) HbR, and (right) HbT signals for the mapwise reliability assessment. Each data point represents the strength of the group-level RSFC at a single channel derived from sessions 1 and 2. The red data points indicate the channels in the predefined sensorimotor region. The solid black line is the fitted line, and the gray lines indicate the threshold of p < 0.01, uncorrected.

Table 1

Mapwise reliability at the individual level.

Subject	HbO	HbR	HbT
1	0.64	0.54	0.70
2	0.78	0.52	0.79
3	0.70	0.57	0.43
4	0.62	0.65	0.20
5	0.75	0.73	0.80
6	0.70	0.11	0.50
7	0.62	0.54	0.53
8	0.60	0.32	0.55
9	0.63	0.58	0.56
10	0.68	0.54	0.62
11	0.41	0.58	0.31
12	0.76	0.64	0.77
13	0.10	0.22	0.34
14	0.77	0.76	0.65
15	0.46	0.09	0.00
16	0.64	0.41	0.71
Mean (Std)	0.62 (0.17)	0.49 (0.20)	0.53 (0.23)

4.

Discussion

In this study, the test-retest reliability of the fNIRS-based, seed-correlation–derived RSFC was assessed at different spatial scales: map-, cluster-, and channelwise reliability. In all cases, we found acceptable map- and clusterwise reliability, but generally lower channelwise reliability (see Fig. 3 and Tables 1, 2, 3, 4).

The different reliabilities identified among the map-/cluster- and channelwise assessments suggests that reliability should be interpreted with different levels of confidence for different spatial scales of the RSFC. Taking HbO data for example, both the overall pattern of the RSFC map and the quantitative/distributive characters of the RSFC clusters demonstrate good-to-excellent reliability. These results can thus be confidently trusted. In contrast, the individual RSFC for a single channel should likely be interpreted with caution due to a lower channelwise reliability. Practically, Instead of being interested in a RSFC result at a single channel, we suggest researchers turn to the averaged RSFC value within a round cluster centered at the interested channel [as frequently performed in region-of-interest (ROI) analysis often adopted in the community of functional magnetic resonance imaging (fMRI)] to increase the reliability of the final result. In other words, it is favorable to construct an ROI and place the more reliable cluster-level result back to the specific channel to replace the less reliable channel-level result. Such an approach is actually a trade-off, balancing between the spatial resolution and reliability.

Previously, investigations have been carried out on test-retest reliability of the task activation based on fNIRS during a similar motor,⁸ as well as for other,^{7, 9} task protocols. Those studies, together with ours, are both important and significant for the fNIRS community: together they contribute to the knowledge on fNIRS-based results’ reliability or reproducibility. By now, it has been extended to include the measurement of the intrinsic functional architecture within resting-state brain.

Compared to the reliability of the task activations,^{7, 8, 9} the reliability of RSFC demonstrates comparable (range from fair to excellent) map- and clusterwise, but lower channelwise, reliability. The difference in channelwise reliability between task activations and RSFC may lie in the different signal generation biomechanics evaluated (i.e., task-evoked signals or spontaneous fluctuations). As speculated in Sec. 1, in contrast to the more consistent hemodynamic response during engagement of a specific task, the unconstrained nature of the resting state²⁴ makes it more difficult to replicate the channel-level result between sessions and across subjects. Such an effect may be further magnified due to other factors that include the potential placement-variability of the NIRS optodes between sessions and across subjects, absence of an optode coregistration algorithm based on craniocerebral correlation, and the low spatial resolution of the fNIRS. When all these factors are considered together, the exact channel-to-channel match for both RSFC strength and location is impossible, leading to the lower channelwise reliability observed.

Regarding the different hemodynamic parameters measured by fNIRS, HbO and HbT provided better RSFC reliability than HbR. Specifically, the map-, cluster- and channelwise reliability of the individual-level RSFC derived from HbO signals was higher than that derived from HbT and HbR signals (where result showed the weakest reliability) (see Tables 1, 3, 4). For the group-level RSFC (Fig. 3 and Table 2), the result was a little different, with reliability for HbT slightly being higher than those for HbO and HbR. However, in all cases, the HbR signal-derived RSFC demonstrated the lowest reliability compared to the other two hemodynamic variables, especially at the individual level. Such a result is in line with the weaker RSFC strength derived from the HbR signals (see Fig. 2; also reported in previous rs-fNIRS studies),^{3, 11} which has been speculated to be related to the lower signal-to-noise ratio of the HbR signal than that of the HbO and HbT signals.³ This speculation also came from previous reliability studies based on task fNIRS,^{8, 9} in which similar result to ours were found. These results suggest that in subsequent studies, it would be better to use HbO signals to conduct a seed-correlation–based RSFC analysis than HbR or HbT data.

Concerning the generally poor (except for a small number of channels with acceptable) channelwise reliability (averaged ICC <0.4 but maximum ICC >0.4 in Table 4), we had hoped to demonstrate that the channels with high reliability were those with high RSFC strength. However, further examination revealed that the channels with high channelwise reliability did not specifically lie within the predefined sensorimotor areas or the post hoc-defined RSFC clusters. Additionally, there was nonsignificant correlation (p > 0.05) between ICC values and group-level RSFC strength across channels. This result is in agreement with previous findings,^{37, 38} which identified highly functional connected channels with low reliability and subthreshold channels having high reliability (see comparison between maximum ICC across channels in the entire probe, preROI, and postROI in Table 4).

Because of the fact that the optical pathway of the near-infrared light also includes superficial nonbrain tissues (e.g., skull, scalp, cerebrospinal fluid, etc.), fNIRS measurements also contain signals related to systemic activity and/or superficial signals.¹⁰ Therefore, it would be natural for concerns to rise if our findings were affected by systemic activity and/or superficial signals. However, we argue that such an effect could not be dominant based on the direct and indirect evidences from the literature as noted below. A new study, which also used a continuous-wave single distance fNIRS device similar to us, demonstrated that the RSFC between homologous regions was significantly larger than the RSFC between “control” regions, where the RSFC was not expected.³⁹ The result plus the previous findings from our papers^{11, 13} suggested that the RSFC detected was nondominantly affected by superficial signals and/or systemic noise. Direct evidence was from the study of Katura,⁴⁰ where systemic cardiovascular dynamics (as one of the sources of systemic noise) have a nondominant contribution to the hemodynamic changes in the low-frequency range (with contributions of 35% for HbO and 7% for HbR). Moreover, in a previous study,⁴¹ the effect of the skin blood-flow fluctuation, as one of the sources of the superficial signals, on the low-frequency signal has been demonstrated to be nondominant. Attributed to this finding, this type of the superficial signal may thus have a small effect on the RSFC derived from the low-frequency signal in our study. In addition, the high-frequency components of the systemic noises, contributed by cardiac pulsations (0.6–1.2 Hz) and respiration fluctuation (0.1–0.5 Hz),¹⁴ have been reduced by the bandpass filter in our study with an upper-limit frequency of 0.08 Hz.

Despite the interesting findings from this test-retest study, we must note its limitations as discussed here. First, this study is mainly designed to test the reliability of a new technical development rather than the validity of it. Specifically in this study, reliability assesses the consistency of the fNIRS-based, seed-correlation–derived RSFC between sessions. However, reliability does not necessarily present implications regarding the validity, which assesses the accuracy of the RSFC. As an important characteristic, validity deserves full investigation in future. As such, our ongoing study is underway with a goal of determining the validity of the fNIRS-based, seed-correlation–derived RSFC by using simultaneous recording of fNIRS and fMRI.

Second, in this study we focused on consistency between repeat measurements (i.e., test-retest reliability) rather than the reproducibility between laboratories, other NIRS systems, other RSFC calculation methods or, even other neuroanatomical systems. These problems are equally important and need further study. As such, when interpreting our results one should always consider the specific data recording and analyzing methods we used. In changing these parameters, the result itself may be also altered. We suggest that the reliability or reproducibility be taken as a “golden standard” to compare difference in data measuring, analyzing methods, and parameters (e.g., different frequency filtering bands) to figure out the most optimized RSFC-detection pipeline.

Third, the poor channelwise reliability we identified may be somewhat connected to absence of optode coregistration during data preprocessing. In future studies, several existing algorithms and procedures (such as probabilistic registration⁴² and virtual registration)⁴³ should be borrowed and utilized in fNIRS-based RSFC studies. In summary, reliability characterization is the foundation of a novel scientific researching tool under development. It is important, necessary, and cannot be overlooked.

5.

Conclusions

The test-retest assessment of the seed-correlation–derived RSFC based on rs-fNIRS demonstrates acceptable map- and clusterwise reliability for both individual- and group-level RSFC (and even better for HbO signal-derived RSFC). However, the assessment does not demonstrate adequate channelwise reliability. Such a result suggests that the fNIRS-based, seed-correlation–derived RSFC can be treated as a reliable biomarker if interpreted at a larger scale, but the channelwise interpretation of individual RSFC should be conducted with caution.