Baidu released ERNIE 5.5 Flash Base (ernie.baidu.com)

Baidu has announced the release of ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts (MoE) language model boasting 280 billion parameters, with 14 billion activated per token. This model has been trained from scratch on 33.5 trillion tokens and designed for exceptional efficiency in both training and inference. Significantly, it allows developers to start with a well-prepared base that can be fine-tuned for various applications without incurring the high costs of extensive pre-training. The release emphasizes efficiency through innovations like Hybrid Compressed Attention and Multi-Router Latent MoE, which enhance the model's computational resources and scalability. This release is crucial for the AI/ML community as it provides a clean and flexible starting point for a range of tasks, including domain adaptation, instruction tuning, and reinforcement learning. It supports up to a 1 million-token context for fine-tuning, allowing for deeper and more complex interactions. The model architecture includes distinct improvements such as a new BPE tokenizer for optimized numerical processing, a multi-temperature softmax output head, and a unique ECHO method for future token prediction, improving both reasoning and coding capabilities. Collectively, these advancements position ERNIE-5.5-Flash-Base as a major contribution to the ongoing evolution of language models in AI.