Ternary-Bonsai-8B-Gguf (huggingface.co)

The Ternary-Bonsai-8B-Gguf model, created by PrismML, has been released as a cutting-edge 1.58-bit language model utilizing the GGUF Q2_0 format catered for llama.cpp. Designed with a base model of Qwen3-8B, this model boasts an impressive architecture featuring 8.19 billion parameters and a significant context length of 65,536 tokens. It achieves remarkable compression by using ternary weights (-1, 0, +1) with a packed Q2_0 format, resulting in an astonishing reduction in file size to just 2.03 GiB compared to the original FP16 size of 16.38 GB. Furthermore, its innovative encoding technique and weight scaling notably improve efficiency, making it a leading choice for resource-constrained environments. This release is significant for the AI/ML community as it showcases advancements in model compression techniques without sacrificing performance, ranking second among evaluated models despite its compact size. The Ternary-Bonsai model operates effectively on various platforms, including CPU and Metal, and is compatible with the upcoming Q2_0 updates in llama.cpp, emphasizing its role in ongoing developments in efficient AI/ML model design. Researchers and developers are encouraged to access the demo repository for serving and benchmarking examples, further promoting community collaboration and innovation in language model applications.