Two different tricks for fast LLM inference (www.seangoedecke.com)

Anthropic and OpenAI recently unveiled distinct approaches for enhancing the speed of large language model (LLM) inference. Anthropic's method boosts token processing speed to 2.5 times that of their Opus 4.6 model, while OpenAI's fast mode offers an impressive 1,000 tokens per second, making it up to 15 times faster than their previous GPT-5.3-Codex. Despite this speed, OpenAI's implementation utilizes a less capable interim model called GPT-5.3-Codex-Spark, contrasting with Anthropic's use of their full model capabilities. The significance of these advancements lies in their contrasting computational strategies. Anthropic is believed to leverage low-batch-size inference to minimize wait times, akin to a bus service that departs immediately when passengers board. In contrast, OpenAI employs Cerebras chips, which support ultra-low latency through larger memory capacities, drastically boosting inference speeds. However, the trade-off for OpenAI is the introduction of a less capable model, raising questions about the balance between speed and accuracy. This competition between two leading AI labs underscores the evolving landscape of AI inference, where speed becomes a critical factor even as model quality and capabilities are debated.