Show HN: A linear model is an embedding table: using VW as transformer memory (www.liminalplanes.com)

DeepSeek recently unveiled a novel approach to enhance transformer models with their "Engram" module, which utilizes a large lookup table to serve as a conditional memory. This innovation allows the model to reference past tokens efficiently, hashing n-grams to fetch corresponding entries from the table. The key advantage here is that this embedding table can be precomputed using Vowpal Wabbit (VW), an online learning library, significantly reducing computation costs and training time without requiring complex adjustments to existing transformer architectures. Experimentation revealed that a transformer model enhanced with Engram achieves 29% lower perplexity compared to a traditional model, with minimal additional computational costs. The VW embedding acts as a static lookup table while allowing continuous updates through online learning, enabling the model to incorporate new information without retraining. This modular approach could pave the way for more adaptable and scalable AI systems, allowing the integration of domain-specific knowledge and real-time updates, highlighting a significant leap in AI capabilities and efficiency.