Dream: Visual Decoding from Reversing Human Visual System (github.com)

Researchers have introduced DREAM, an innovative fMRI-to-image decoding framework that reconstructs visual stimuli by mimicking the human visual system’s reverse processing pathways. Unlike prior approaches that treat brain-to-image decoding as direct mapping, DREAM explicitly replicates the hierarchical and parallel structures of the visual cortex to separately decode semantics, color, and depth cues from fMRI data. This is achieved via two key modules: R-VAC, which extracts semantic information as CLIP embeddings by reversing the visual association cortex pathways; and R-PKM, which concurrently predicts color spatial palettes and depth maps, informed by surrogate ground-truth depth from MiDaS. The decoded components are then integrated using Guided Image Reconstruction, leveraging Stable Diffusion with specialized Color and Depth Adapters to generate final images. DREAM’s significance lies in its biologically inspired architecture that bridges the gap between neuroscience and generative AI, offering a more interpretable and structured approach to visual decoding from brain activity. Importantly, it tackles the complexities of disentangling semantics from low-level visual cues and combines multimodal information to produce images reflecting natural scene structure and appearance. While manual tuning is sometimes needed for optimal output stability, DREAM sets a new benchmark by embedding domain knowledge of the human visual pathway into the ML pipeline, promising advancements in brain-computer interfaces, neuroscience research, and neuroprosthetics. The project is open-source with pre-trained models and evaluation metrics for depth and color consistency, facilitating further exploration and development in neural decoding.