Do Transformers Need Three Projections? Systematic Study of QKV Variants (arxiv.org)

A recent study systematically explores the necessity of the three projections—query, key, and value (QKV)—in transformer architectures, which are critical to a wide array of AI tasks. The researchers evaluated various projection-sharing configurations: shared key-value (Q-K=V), shared query-key (Q=K-V), and a single projection (Q=K=V). Remarkably, these simplified variants performed comparably or even surpassed standard QKV transformers across tasks in vision and language modeling. Notably, the Q-K=V configuration achieved a significant 50% reduction in key-value cache usage with minimal impact on performance, highlighting its potential for enhancing efficiency in on-device AI applications. This research holds substantial implications for the AI/ML community, particularly regarding inference memory efficiency and practical deployment in edge devices. The findings suggest that projection sharing can be synergistically combined with head sharing strategies (GQA/MQA) to further reduce memory requirements—up to 96.9%—while maintaining model robustness. The study characterizes projection sharing as an underexamined area in attention mechanisms, emphasizing its capability to facilitate low-latency applications without compromising accuracy, thus paving the way for more efficient transformer models in real-world scenarios. The code is publicly available for community use, encouraging further exploration in this domain.