We built DeepL's next-generation LLMs with FP8 for training and inference (2025) (www.deepl.com)

DeepL has announced the development of its next-generation large language models (LLMs) utilizing 8-bit floating point (FP8) precision for both training and inference, leveraging NVIDIA's H100 Tensor Core GPUs. This shift to FP8 enables significantly improved throughput, allowing for the training of larger models with more parameters while maintaining low latency—resulting in translations that outperform previous models by 1.4x for European languages and 1.7x for complex pairs like English and Japanese. The transition from 16-bit to FP8 precision not only enhances computational efficiency but also effectively doubles the throughput capacity in inference scenarios, allowing DeepL to handle more requests simultaneously without compromising user experience. The implications of using FP8 over the previous BFloat16 (BF16) format are noteworthy for the AI/ML community. FP8's reduced precision and smaller range can be advantageous, echoing the increasingly accepted notion that certain applications require less absolute precision to maintain quality outcomes, especially in LLM training. DeepL’s implementation involved sophisticated techniques, including mixed-precision training and optimizations provided by NVIDIA's Transformer Engine and TensorRT-LLM, showcasing a path towards achieving high-quality models faster and with lower memory demand. This development could set a precedent for other organizations working with LLMs, indicating a new frontier in enhancing AI model training and deployment efficiency.