Understanding, Mitigating Numerical Sources of Nondeterminism in LLM Inference (arxiv.org)

Recent research highlights a significant challenge in the reproducibility of Large Language Model (LLM) performance, revealing that minor changes in system configurations can lead to substantial variations in model outputs. The study, which systematically investigates numerical sources of nondeterminism in LLM inference, demonstrates that aspects such as evaluation batch size and GPU configurations can impact reasoning models, with observed variances in accuracy reaching up to 9% and response length differences soaring to 9,000 tokens. This unpredictability stems from the non-associative nature of floating-point arithmetic, particularly under constrained numerical precision. To address this issue, the researchers introduced a lightweight inference pipeline called LayerCast, which optimizes model performance by storing weights in 16-bit precision while executing computations in FP32. This approach not only enhances numerical stability but also strikes a balance between memory efficiency and accurate outputs. The findings underscore the vital role of floating-point precision in evaluating LLMs, urging the AI/ML community to incorporate these considerations into standard evaluation practices for more reliable and consistent model assessments. The study and its accompanying code are expected to propel further advancements in reproducible AI model development.