Retrieval Centric Deep Learning: Replacing Weight Matrices with Vector Databases (twitter.com)

In a groundbreaking development, researchers have introduced "Retrieval-Centric Deep Learning" (RCDL), a paradigm that replaces traditional weight matrices in neural networks with vector databases. This approach, motivated by the potential for increased computational power and memory, challenges the conventional reliance on gradient descent for model optimization. Instead of training weights, RCDL emphasizes retrieving and recombining past gradients and inputs, leveraging attention mechanisms to enhance the learning process. This shift could revolutionize how AI models are built and optimized, especially in sequence modeling. The team, led by insightful members from the Paradigms of Intelligence Team, demonstrated that by using advanced kernels like softmax self-attention, they could create a more efficient learning algorithm. They found that their novel "Growing Weighted Softmax Attention layer" could effectively optimize models without relying on fixed weight matrices, opening doors to potentially more powerful model architectures. Although initial results showed mixed performance compared to established methods like Adam and Muon, the ability to construct and optimize AI frameworks without the traditional weight matrix constraints marks a significant step forward. This research is set to be presented at NeurIPS, further igniting interest in alternative approaches to AI model training.