Interactive multi-feature fusion improves semantic reconstruction from non-invasive brain recordings

A new framework addresses the representational mismatch in continuous semantic reconstruction by fusing static lexical representations (W2V) with dynamic contextualized representations (GPT). The study benchmarks two integration approaches: linear Naive Concatenation and non-linear Multi-Head Cross-Attention.

• The approach uses an interactive gating mechanism to facilitate cooperative processing during language comprehension.
• Evaluation reveals a performance hierarchy of Cross-Att > Concat > GPT > W2V.
• The non-linear cross-attention fusion method achieves state-of-the-art performance.

The authors consider this significant because it demonstrates that neural language decoding benefits from simulating collaborative modulation between contextual information and core lexical attributes, offering a viable non-invasive brain-to-text decoding method.