SpikingBrain：Spiking Brain-Inspired Large Models (github.com)

SpikingBrain is a novel large-scale AI model architecture inspired by the brain’s spiking neuron mechanisms, integrating hybrid efficient attention, Mixture-of-Experts (MoE) modules, and spike encoding to achieve high efficiency and performance. Leveraging a universal conversion pipeline compatible with open-source models, SpikingBrain enables continual pre-training using less than 2% of typical data volumes while delivering competitive results matching mainstream open-source large models. This approach also incorporates advanced system-level optimizations—such as tailored parallel strategies and communication primitives—that support stable training and inference on non-NVIDIA MetaX clusters, broadening hardware flexibility and scalability. Technically, SpikingBrain achieves remarkable computational efficiency, exemplified by a 100× speedup in token train-from-token (TTFT) processing for 4-million-token sequences and over 69% micro-level sparsity through spiking activity. Coupled with macro-level sparsity from MoE modules, these advances offer promising blueprints for next-generation neuromorphic chip design. The project provides a comprehensive open-source repository of SpikingBrain-7B, including HuggingFace and vLLM inference versions, plus a quantized variant (W8ASpike) optimized for low-precision inference using pseudo-spiking approximations—facilitating flexible deployment and experimentation across hardware platforms. Furthermore, the vLLM-hymeta plugin modularizes backend support for NVIDIA GPUs, enhancing maintainability and fast integration of new hardware. By bridging biologically inspired neural computation with practical engineering, SpikingBrain marks a pivotal step toward more energy-efficient, scalable large models. Its pioneering pseudo-spiking technique and mixed-sparsity architecture present valuable directions for both AI model design and neuromorphic hardware research, making it a key resource for the AI/ML community pursuing heightened efficiency without sacrificing performance.