SpikingBrain Technical Spiking Brain-Inspired Large Models (arxiv.org)

Researchers released SpikingBrain, a family of brain-inspired large models and system stack designed to tackle Transformers’ efficiency limits for very long contexts and to enable large-model development on non‑NVIDIA hardware. The project combines architectural changes (linear and hybrid-linear attention with adaptive spiking neurons), algorithmic tools (a conversion-based training pipeline and a spike-coding framework), and system engineering (custom operators, parallelism strategies, and a MetaX-tailored training framework) to build and run large LLMs on MetaX C550 GPU clusters. They present two instantiations—SpikingBrain-7B (linear) and SpikingBrain-76B (hybrid-linear Mixture-of-Experts)—trained with only ~150B tokens and showing competitive performance versus open-source Transformer baselines while delivering dramatic long-sequence gains: partially constant-memory, event-driven inference and over 100× faster Time-to-First-Token on 4M-token inputs. System metrics include stable multi‑week training on hundreds of MetaX GPUs, 23.4% Model FLOPs Utilization for the 7B model, and a spiking sparsity of ~69.15%, suggesting lower power needs. The work demonstrates that spiking mechanisms plus tailored software/hardware co-design can enable scalable, energy‑efficient, long‑context LLMs and make large-model research more viable on alternative GPU platforms.