Aleph Alpha releases open-weight Kolibri with 1M context (www.testingcatalog.com)

Aleph Alpha has announced the release of Kolibri, an open-weight bilingual model specifically designed for mission-critical applications in government and regulated industries. This English-German Mixture-of-Experts Transformer boasts a staggering 78.1 billion parameters, activating 3.46 billion per token, and supports a remarkable context of up to one million tokens. By allowing users to download the model with full weights from Hugging Face, Kolibri enables on-premises operation, preserving sensitive internal data and enhancing compliance with European regulations. Significantly, Kolibri builds on the capabilities of its predecessor, Kolibri Origin, and employs advanced training techniques, including a 21-day training on 20 trillion tokens using 768 B200 GPUs. The architecture integrates 384 experts but activates six per token, balancing full attention with efficient inference costs. The model has been fine-tuned for various domains such as math, coding, and long-context tasks, featuring a specialized grounding approach that improved its accuracy compared to earlier models. With competitive performance benchmarks, Kolibri positions itself as a strong contender against larger models, emphasizing both quality and cost-efficiency for users requiring high-level reasoning in both English and German.