Mamba-3: Improved Sequence Modeling Using State Space Principles (openreview.net)

Mamba-3 is an “inference-first” state-space model (SSM) for sequence modeling that reshapes the sub-quadratic alternative to Transformers by focusing on practical decoding efficiency and quality. Built from SSM principles, the model introduces improved discretization for more accurate temporal behavior, richer (complex) dynamics to capture nuanced state trajectories, and a multi-input multi-output (MIMO) update scheme that parallelizes and accelerates autoregressive decoding on hardware. The result: better retrieval, state-tracking, and downstream language modeling performance while operating with linear-time, constant-memory inference and beating strong baselines under fixed inference budgets. Technically, Mamba-3 combines a more expressive recurrent core and a complex state update rule that enables finer-grained state maintenance than many existing linear-style models, which often trade capability for asymptotic efficiency. The MIMO formulation batches multiple inputs/outputs per step to exploit hardware parallelism during decoding, reducing real-world latency beyond theoretical linear-time gains. Together with architectural refinements, these changes move the Pareto frontier for inference cost vs. quality, offering a practical path for deploying capable large-sequence models where test-time compute and latency are the primary constraints.