Eliminating Nondeterminism in LLM Inference with Integer-Only Arithmetic (nathan.rs)

In a groundbreaking project dubbed DetLLM, researchers have proposed a method to eliminate nondeterminism in large language model (LLM) inference by replacing floating-point arithmetic with integer-only computations. Traditional greedy decoding is prone to variations in output due to the non-associative nature of floating-point addition, which can yield different results based on how calculations are ordered during execution. This nondeterminism becomes pronounced across various hardware configurations, where incremental rounding errors can lead to divergent outputs even with the same input. DetLLM addresses this challenge by using integers for all computation steps—encompassing matrix multiplications, softmax functions, and normalizations—ensuring consistent outputs irrespective of the underlying hardware or batch size. The project demonstrated that integer operations provide deterministic behavior across different devices, yielding identical results in 9 configurations, while maintaining performance levels close to traditional floating-point methods. By achieving high decoding speeds and maintaining perplexity comparable to floating-point models, DetLLM opens fresh avenues for reproducibility in AI/ML applications, particularly significant for deployment in diverse environments.