How Taalas "prints" LLM onto a chip? (www.anuragk.com)

Startup Taalas has introduced a groundbreaking ASIC chip designed to run the Llama 3.1 8B language model at an impressive inference rate of 17,000 tokens per second, which translates to processing roughly 30 A4-sized pages in just one second. This chip is touted to be ten times cheaper and more energy-efficient compared to traditional GPU-based systems, while also delivering performance that is ten times faster than current state-of-the-art inference solutions. Taalas achieves this by physically "hardwiring" the model’s weights directly into the silicon of the chip, eliminating the memory bandwidth bottlenecks typically encountered in GPU architectures. Rather than relying on external memory like DRAM, Taalas’s chip utilizes on-chip SRAM for temporary storage, enabling continuous data flow without latency. Their innovative approach includes a 'magic multiplier' that allows for efficient 4-bit data processing with single transistors, resulting in a streamlined performance that keeps data within the chip rather than cycling back and forth to memory. Although custom chip fabrication is typically slow and costly, Taalas has designed a versatile base chip that can be adapted for different models more rapidly, illustrating the potential for faster deployment of specialized hardware in the AI landscape.