Speculations about DeepSeek-OCR and Quantization (dgsq.net)

DeepSeek recently released DeepSeek-OCR, a model and paper that treat rendered text as images to “compress” text into far fewer visual tokens. They report strong OCR performance: roughly 97% decoding precision when the number of text tokens is within 10× that of vision tokens (compression ratio <10×), and about 60% accuracy even at ~20× compression. The approach is notable because it sidesteps traditional tokenizers and leverages visual pattern recognition—an intuitively human-like reading strategy that can encode multi-character patterns as single vision tokens, potentially reducing autoregressive decode length and latency. The write-up raises an important caveat: DeepSeek evaluated with BF16 weights, but modern low-precision inference (e.g., NVFP4-style formats) can cut model size to ~4 bits per weight/activation with modest accuracy loss. If true, the claimed 10–20× token-compression advantage might shrink to ≈2.5–5× relative to a 4-bit-quantized text model, implying text tokens could be effectively representable in ~0.8–1.6 bits (near 1-bit) or that image tokens simply use available representational “space” more efficiently. That shifts the core tradeoff: many ultra-low-bit text tokens vs. fewer higher-precision visual tokens, with implications for latency (autoregressive decoding), hardware targeting (current GPUs favor 16-bit math), and real-world quantized accuracy. Empirical evaluation of DeepSeek-OCR under aggressive quantization is needed to settle how much of the compression wins survive low-bit inference.