Models are needed that account for interaction among multimodal features and temporal dependence when predicting neural response to continuous AV stimuli. One of the major challenge is that, stimulus related information can affect multiple neural response targets at varying time delays, making  fixed time alignment insufficient. This study is motivated by the need to develop delay-aware multimodal neural encoding framework that combines cross-modal attention, temporal state-space processing and learnable parcel-specific delay alignment. Time varying neural responses are estimated simultaneously by visual, auditory, and linguistic representations. The framework is tested using a controlled synthetic benchmark of 200 brain parcels, also using the same experimental protocol for six encoding approaches. The highest mean test Pearson correlation of 0.0776 was achieved by the proposed hybrid model. More importantly, temporal ablation experiments demonstrated that the state-space module boosted the correlation from 0.0601 to 0.0769, which is 28.0% higher than the configuration without temporal processing. Similarly, the learned parcel-level delays were also correlated with the reference delays, with a correlation of 0.636. While the absolute predictive correlations stayed low, the results show that there is measurable temporal-learning behavior and encourage further research into adaptive temporal alignment in multimodal neural encoding.

Neural encoding,multimodal learning,temporal state space models,cross-modal attention,adaptive delay alignment,brain response prediction