ReplaySSM: Cache SSM Inputs, Not State (Tri Dao) (tridao.me)

Tri Dao's new method, ReplaySSM, reimagines the state space model (SSM) approach to decoding in AI by caching recent inputs instead of writing the state back to memory at each step. This innovative technique directly addresses the inefficiencies associated with traditional SSM decoding, where the recurrent state must be continuously updated and stored. By retaining a small buffer of recent inputs, ReplaySSM allows the model to reconstruct the required state on-demand, significantly reducing memory traffic and leading to improved speed—achieving up to 1.96 times faster decoding in speculative scenarios, and 1.48 times in standard decoding. This advancement is particularly important for the AI/ML community as it addresses the long-standing bottleneck of memory traffic and latency in large-scale models, especially those operating in resource-constrained environments. Current transformers, though effective, struggle with manageability as sequence lengths increase due to their extensive memory requirements. ReplaySSM's approach not only optimizes performance but also simplifies rollback processes in speculative decoding, paving the way for faster and more efficient AI inference and training methodologies, while keeping the output consistent with traditional methods.