Show HN: Best setup local LLM found for a 5090 (llama.cpp fork + turboquant) (local-llm.utop.workers.dev)

A new report highlights a groundbreaking setup that enables the Qwen 3.6 35B Mixture of Experts (MoE) model to run at a massive context window of 450,000 tokens on a single 32GB NVIDIA RTX 5090 GPU. This configuration utilizes llama.cpp alongside TurboQuant for memory optimization, effectively compressing storage needs and allowing complex computations to be performed without the overhead often seen in dual-boot systems. Key technical decisions include using a Q6_K quantization to balance memory efficiency with logical accuracy, which is crucial for tasks such as code generation. This achievement is significant for the AI/ML community as it demonstrates the potential for running large models on consumer-grade hardware, a feat previously thought to be limited to specialized infrastructure. By leveraging advanced techniques like KV cache quantization and RoPE scaling, researchers can conduct broader analyses and more elaborate tasks without needing cloud resources. However, the report warns of the trade-offs associated with expanding the context beyond the model's native capabilities, indicating that while this setup excels for exploratory and summarization tasks, critical operations might benefit more from keeping to the original limits to maintain reasoning accuracy.