Lighthouse Attention (nousresearch.com)

Researchers have unveiled Lighthouse Attention, a revolutionary selection-based hierarchical attention mechanism that significantly enhances training speed and efficiency for models managing long-context input. Operating approximately 17 times faster than traditional attention methods at 512k context and achieving up to 1.7 times increased end-to-end pretraining speed at 98k context, Lighthouse Attention streamlines the pooling of queries, keys, and values across a multi-resolution pyramid. The innovation relies on a parameter-free scoring system using $\ell_2$ norms, enabling a dense sub-sequence to be processed efficiently without the need for specialized sparse attention kernels. This advancement is crucial for the AI/ML community, as it addresses the long-standing bottleneck created by the quadratic compute cost of attention in transformer models. Importantly, Lighthouse Attention not only accelerates the pretraining process but also maintains the model's capability to employ full-dense attention post-training, dispelling concerns that sparse training might hinder performance. The method has been validated using the 530M Llama-3 model over 50 billion tokens, showcasing competitive results against dense training methodologies. The implementation is accessible through GitHub, paving the way for broader adoption and exploration within the AI research community.