Show HN: Spectra – A drop-in optimizer that cuts LLM training VRAM by 50% (github.com)

Spectra Labs has introduced a new optimizer, SpectraAdamW, designed to significantly reduce the VRAM requirements for training Large Language Models (LLMs) on consumer hardware by 50%. Traditional optimizers like AdamW demand that two substantial state tensors—momentum and variance—be stored, effectively doubling the VRAM needed compared to the model parameters. SpectraAdamW addresses this bottleneck by utilizing a hybrid approach that applies factorized variance approximations and frequency-domain momentum compression, thus maintaining full-precision convergence while halving the state overhead. This breakthrough is particularly significant for researchers and developers working with LLMs in resource-constrained environments, as it simplifies the optimization process without compromising model performance. In trials with simulated 8192-dimension Transformer layers, the state VRAM required was cut from 256 MB to 128 MB with SpectraAdamW, demonstrating considerable efficiency gains. The optimizer's design ensures that gradient integrity is preserved even with aggressive compression techniques, making it a promising tool for fine-tuning large models more effectively. Currently available in a closed beta, the optimizer acts as a straightforward drop-in replacement for existing PyTorch implementations and aims to foster wider accessibility to cutting-edge AI development tools.