Neural correlates of cross-modal correspondence: fNIRS evidence from color-vibrotactile perception.

Abstract

The neural mechanism of cross-modal correspondence in multimodal perception has not been fully studied. This study investigated the neural basis of color-vibrotactile cross-modal correspondence using functional near-infrared spectroscopy (fNIRS) and combined the neural evidence in multimodal perception with subjective evaluation.
Approach. Twenty adults completed a 2×2 factorial experiment manipulating visual color chroma (dark or light red) and vibrotactile amplitude (strong or weak vibration). Within different multimodal conditions, congruent and incongruent stimulus combinations were presented in a block design. Change in oxygenated hemoglobin concentration (ΔHbO) was analyzed across five cortical regions of interest (ROIs), including occipital cortex (OC), posterior parietal cortex (PPC), medial prefrontal cortex (mPFC), primary somatosensory cortex (S1), and premotor cortex (PMC). Multisensory integration was assessed using additive and max-criterion models. Effective connectivity was examined using Multivariate Granger Causality (MVGC). Subjective correspondence ratings were collected to assess brain-perception relationships.
Main results. Incongruent conditions elicited significantly greater activation in the PPC compared to congruent conditions. For unimodal difference, the OC showed a significant main effect of color, while the mPFC showed a significant main effect of vibration intensity. Most cortical responses exhibited sub-additive integration and failed to exceed the strongest unimodal response, supporting divisive normalization principles. A general, stable relationship among different cortices was observed, modulated by stimulus intensity and cross-modal correspondence. Congruent cross-modal stimuli engaged balanced cortex interactions, while incongruent stimuli recruited the PPC-mPFC axis as a resolution hub. Color-vibrotactile correspondence modulated cortical activation and network dynamics in accordance with predictive coding and normalization accounts. Overall, cross-modal correspondence might promote neural efficiency by optimizing cortical activation and balancing network dynamics.
Significance. These findings extend multisensory integration theory to the visual-tactile domain, provide the neural evidence of color-vibrotactile cross-modal correspondence, and demonstrate the utility of fNIRS for investigating network-level mechanisms of cross-modal processing.