Analog In-Memory Computing Attention Mechanism for Fast LLMs (arxiv.org)

Researchers have developed a novel analog in-memory computing architecture tailored for the self-attention mechanism in large language models (LLMs), addressing major latency and energy challenges in generative Transformers. By leveraging emerging charge-based gain cells, this architecture stores token projections directly in memory and performs parallel analog dot-product computations efficiently during sequence generation. This circumvents the costly data movement bottleneck where GPU-stored projections must be repeatedly loaded into SRAM, drastically reducing both delay and power consumption. To tackle the analog circuit non-idealities that prevent straightforward use of pre-trained models, the team introduced a specialized initialization algorithm enabling comparable text processing performance to GPT-2 without retraining from scratch. Their approach achieves up to two orders of magnitude faster attention latency and five orders of magnitude lower energy use compared to conventional GPU implementations. This breakthrough marks a significant leap toward ultra-fast, energy-efficient Transformer inference hardware, offering a promising pathway to scaling and deploying large language models in more resource-constrained environments and edge devices. The work’s synthesis of analog memory technology with neural architecture innovations highlights a key direction in ML hardware co-design, emphasizing in-memory analog computation to unlock new levels of efficiency and speed. For the AI/ML community, this presents exciting opportunities to rethink Transformer deployment beyond digital accelerators, potentially accelerating future advances in real-time natural language generation and on-device intelligence.