Motif-3-Beta (huggingface.co)

Motif Technologies has announced the release of an intermediate beta version of their large-scale Mixture-of-Experts (MoE) language model, Motif-3. With approximately 314 billion parameters and a unique architecture built from scratch, rather than re-engineering existing frameworks, Motif-3 boasts an impressive context length of 256,000 tokens, enabling it to handle extensive text inputs natively. Its architecture features 384 experts, with 8 activated per token, enhancing its ability to process language in a sparse manner, making it both efficient and powerful. This announcement is significant for the AI/ML community as it advances the capabilities of MoE models, particularly with innovations like Grouped Differential Latent Attention and Grouped PolyNorm activation for individual experts. The model’s open availability allows researchers and developers to experiment with advanced features freely, although commercial use is restricted. With custom modeling components and a comprehensive serving guide on the way, Motif-3 positions itself as a versatile tool for multilingual and general-purpose applications, further pushing the boundaries of what is achievable with large language models.