The Inference Stack Behind OCRia (www.llamaindex.ai)

OCRia-1 has achieved a significant enhancement in document parsing efficiency, displaying a remarkable nearly 10x improvement in inference speed for token generation, increasing from approximately 285 tokens per second to around 2800 tokens per second on a single B200 GPU. This advancement is crucial for the AI/ML community as it addresses a long-standing challenge of latency in document parsing workflows, ultimately allowing for faster iteration on benchmarks and more responsive user experiences. The optimizations were achieved through a combination of techniques including quantization, speculative decoding, and numerous micro-optimizations that reduced data movement and CPU overhead. The underlying technical details reveal that the model's inference is dominated by the decode phase, where operations had previously been memory-bound. By employing workload-aware mixed precision quantization and implementing speculative decoding with a DFlash2 model to draft multiple future tokens, the team was able to minimize the need for expensive data transfers during the decoding process. Additionally, optimizations like kernel fusion and improved CPU-side preparation significantly reduced latency, enabling greater throughput. Collectively, these enhancements resulted in a solution that not only retains model accuracy but dramatically accelerates performance, showcasing a substantial leap in handling dense document inputs efficiently.