Show HN: O(1) memory attention – 512K tokens in 3.85 GB (eval binary) (github.com)

A recent demo showcased a groundbreaking implementation of O(1) memory attention, enabling effective processing of extremely long sequences—up to 512K tokens—using just 3.85 GB of memory. This significant advancement was demonstrated on multiple NVIDIA GPUs, including the H100 and A100, effectively handling sequences that traditional attention mechanisms would find impossible, demanding over 1,000 GB of memory. This development is crucial for the AI/ML community, as it opens doors for training and inference on massive datasets that were previously unmanageable due to memory constraints. The ability to maintain efficiency with such large token sizes helps reduce the computational burden, making it feasible to deploy sophisticated models in various applications, from natural language processing to complex data analysis. As this technology matures, it could fundamentally alter how large-scale AI models are designed, leading to more accessible and resource-efficient machine learning solutions.