High-Throughput Chips for LLMs (matx.com)

A new chip designed for large language models (LLMs), the MatX One, has been announced, boasting unparalleled throughput and competitive latencies, which is crucial for frontier labs pushing the boundaries of AI capabilities. The chip’s architecture supports high FLOPS for training and prefill tasks while ensuring low latencies for decoding and reinforcement learning. Notably, it can produce over 2000 output tokens per second for extensive 100-layer Mixture of Experts (MoE) models, a significant achievement for processing large datasets efficiently. This innovation is particularly significant for the AI/ML community as it enables the scaling of models without size limitations, fostering advancements in areas like natural language understanding and generation. The MatX One integrates advanced SRAM for immediate access to weights and high-bandwidth memory (HBM) for effective long context support, enhancing its performance across diverse applications. With strong scale-up and scale-out interconnect capabilities, it enables clusters comprising hundreds of thousands of chips, providing developers with direct programming control over the hardware to optimize their models effectively. The interest from notable investment firms further underscores the potential impact of this technology in shaping the future landscape of AI research and application.