Complex KDA:Understanding and Enhancing the Expressivity of Kimi Delta Attention (arxiv.org)

Researchers have introduced Complex Kimi Delta Attention (CKDA), an advanced framework designed to enhance the expressivity of linear recurrent neural networks (RNNs) by facilitating more complex sequence modeling. CKDA builds upon the foundational Kimi Delta Attention (KDA) by integrating a single delta-rule transformation with a channel-wise gate, thereby achieving 2D rotations without sacrificing stability or efficiency. This innovative approach allows CKDA to track every finite group isomorphic to a subgroup of SO(3) and requires fewer layers for certain state-tracking tasks, presenting a notable improvement over previous models. The significance of this development lies in CKDA's potential to outperform traditional architectures in language modeling scenarios, such as Transformers and other linear RNNs. By extending the parameter ranges to include gate values between -1 and 1, along with a delta-rule coefficient of 0 to 2, CKDA achieves superior performance, particularly in length extrapolation tasks. The open-source release of their code and models enhances accessibility for researchers, encouraging further exploration and development in the AI/ML community. This progress marks a critical step towards more expressive and efficient neural network architectures capable of addressing a wider range of computational challenges.