Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention (arxiv.org)

A recent study introduces Grouped Query Experts (GQE), a novel mixture-of-experts approach to optimize self-attention mechanisms in Transformer models. Traditional dense attention techniques are computationally expensive, particularly for long sequences, as they require uniform activation of attention heads across all tokens, which can waste resources. GQE leverages grouped-query attention (GQA) by utilizing a router to select a subset of query-head experts tailored to the complexity of each token while maintaining the standard key-value (KV) interactions. This innovative method allows for a significant reduction in computational demands. The significance of GQE lies in its ability to enhance the efficiency of Transformers without compromising accuracy, particularly in handling large-scale datasets. By activating only half the query heads per token while achieving comparable performance on a downstream task with a fixed budget of 30 billion tokens, GQE promises to make training and inference more accessible. This breakthrough could pave the way for more efficient AI models, enabling developers to handle longer contexts with reduced computational costs, ultimately driving advancements in various applications of machine learning.