A 0.76% adapter that adds memory attention structurally cannot express (huggingface.co)

A new production adapter called M-Series has been announced, built on the frozen TinyLlama-1.1B-Chat model. This adapter integrates a novel memory attention mechanism, incorporating cross-window memory through a liquid state with multi-timescale selective-decay recurrence. Remarkably, it does so with only an additional 0.76% of parameters, enhancing the model's ability to retain and recall information across different contexts without growing in size as the context expands. This development marks a significant advancement in the AI/ML community, particularly in the realm of long-context language models. The adapter has demonstrated enhanced performance in memory tasks, outperforming traditional static decay methods. With features like selective decay and fast-weight matrices, the M-Series adapter offers efficient mechanisms for improving memory recall, which can't be achieved with standard attention models. This innovation opens the door for more sophisticated applications in NLP and beyond, especially for tasks requiring nuanced understanding across extended text or conversation sequences.