Show HN: LoRA over GGUF – Train Qwen3.8-Flash-Next in 40G VRAM (github.com)

A recent demonstration on HN showcases significant advancements in training large language models (LLMs) using LoRA (Low-Rank Adaptation) with a new model format called GGUF, which is set to replace bitsandbytes. The user successfully trained Qwen3.8-Flash-Next within 40 GiB of VRAM, showcasing GGUF's ability to efficiently handle low-VRAM configurations while optimizing the training framework for local hardware. This development is pivotal for democratizing AI model training, allowing more users to engage in fine-tuning and experimenting with substantial models without needing extensive resources. Key technical innovations include a collaboration of various advanced methods such as enhanced GEMM kernels, quantization techniques, and a tailored training loop with transformers and PEFT (Parameter-Efficient Fine-Tuning). Notably, the integration of fast LoRA backward formulas, optimized for both linear and MoE (Mixture of Experts) layers, highlights the growing efficiency in training practices. Furthermore, the report emphasizes the importance of kernel tuning for non-quantized MoE LoRA and strategies for managing VRAM during model inference, indicating a robust framework that pushes the boundaries of what can be accomplished with limited hardware.