Kolibri Has Landed: A Sovereign Open-Weight Model (aleph-alpha.com)

Kolibri, a new English-German Mixture-of-Experts Transformer model, has been launched with 78 billion total parameters and 3 billion actively used, designed for mission-critical applications in regulated sectors like public administration and aerospace. This significant advancement features a remarkable context length of up to 1 million tokens and showcases a marked performance optimization from its predecessor, Kolibri Origin, which had a context limit of only 65,000 tokens. Kolibri allows users complete control over deployment and data sovereignty, ensuring compliance with EU regulations, including the General Data Protection Regulation (GDPR). The development process was refined through a highly automated pipeline that enabled rapid iteration and robust pre-training without manual intervention during hardware failures. This approach facilitated around 200 trillion tokens of raw data processing, resulting in tailored performance and specialized capabilities aimed at enhancing return on investment for customers. By focusing on domain-specific applications and incorporating both German and English data organically, Kolibri not only aims for high quality in performance but also prioritizes transparency and traceability, further establishing itself as a strong contender in the burgeoning AI landscape.