AI Proposes BitNet Distillation (BitDistill): A Lightweight Pipeline (thedailypac.blogspot.com)

Microsoft Research unveiled BitNet Distillation, a practical 3‑stage pipeline to convert pretrained FP16 LLMs into 1.58‑bit “BitNet” students that use ternary weights and INT8 activations. The pipeline preserves near‑FP16 task accuracy while delivering up to ~10× memory reduction and ≈2.65× CPU inference speedup across Qwen3 backbones (0.6B, 1.7B, 4B) on benchmarks including MNLI, QNLI, SST‑2 and CNN/DailyMail. That matters because prior work showed 1.58‑bit models can match full‑precision quality when trained from scratch, but directly quantizing pretrained FP16 models typically loses accuracy—especially as model size grows—blocking practical deployment for on‑premise and edge scenarios. BitNet Distillation addresses this with three targeted steps: (1) architectural refinement by inserting SubLN normalization before MHSA and FFN output projections to stabilize activation variance feeding quantized projections; (2) short continued pretraining (10B FALCON tokens) to reshape weight distributions so small gradients can flip weights among [-1,0,1]; and (3) distillation that combines temperature‑softened logits KL loss and attention‑relation distillation (MiniLM/MiniLMv2 style) to transfer Q/K/V relations without matching head counts. Training uses straight‑through estimators through the quantizer; activations are INT8. The method is compatible with post‑training quantizers (GPTQ/AWQ) and ships optimized CPU/GPU kernels (bitnet.cpp), making it a pragmatic path to extreme quantization without full retraining.