Show HN: WaveletLM – wavelet-based, attention-free model with O(n log n) scaling (github.com)

WaveletLM, a newly proposed wavelet-based, attention-free language model, introduces a groundbreaking approach to natural language processing with scaling that achieves O(n log n) performance for sequence lengths. This model utilizes techniques like learned lifting wavelet decomposition and the Fast Walsh-Hadamard Transform to facilitate token mixing without traditional attention mechanisms. With enhancements such as per-scale gated spectral mixing and expanded MLPs, WaveletLM replicates top performance benchmarks while significantly reducing computational complexity. The significance of WaveletLM lies in its potential to challenge existing attention-centric architectures by offering a more efficient alternative for large-scale text generation and other NLP tasks. Designed for compatibility with Python, PyTorch, and CUDA, it incorporates a variety of configurable parameters to allow researchers to tweak performance effectively. Initial training results show it achieves strong performance metrics with significantly less VRAM usage compared to conventional models, positioning it as a promising candidate for future AI model developments and research.