Post-transformer inference: 224× compression of Llama-70B with improved accuracy (zenodo.org)

A groundbreaking study from Anima Core Inc and Shamim Institute of Soul Systems has introduced a novel approach to AI inference by eliminating transformers while achieving significant compression and accuracy improvement. This method replaces the traditional 70-billion parameter Llama-3.3-70B model with a compact 256-dimensional meaning field derived from seven internal activation layers, achieving an impressive 224× reduction in size. Notably, the technique enhances performance, yielding an average gain of +1.81 percentage points across various classification tasks, including a +3.25 pp improvement on low-resource tasks. The key innovation lies in the concept of Field Processing Units (FPUs), which shift the paradigm from deep matrix operations to shallow field operations, thus enabling inference without hefty transformer architectures. A subsequent 30M-parameter student model learns to recreate these fields directly from raw text, resulting in a throughput increase of 60 times with minimal accuracy loss. This research not only paves the way for more efficient AI systems but also opens doors for independent verification and further exploration in post-transformer inference, marking a significant step forward in the AI/ML community.