LLM inference is nearly deterministic. We use this to audit providers (adamkarvonen.github.io)

A new method called Token-DiFR (Divergence From Reference) has been introduced to verify the inference quality of Large Language Models (LLMs), addressing significant concerns about the reliability of inference providers. The challenge lies in the inherent non-determinism of LLM token generation, compounded by potential issues like quantization and implementation bugs. Recent findings show that LLM inference can be nearly deterministic when the sampling seed is fixed, allowing researchers to regenerate outputs that match over 98% of the time. Token-DiFR exploits this property by comparing the tokens generated by an inference provider against a reference implementation, making it possible to detect anomalies such as quantization errors with minimal output tokens. This approach is significant for the AI/ML community as it provides a practical solution for auditing inference providers, ensuring adherence to claimed model performance without incurring overhead. By allowing real-time verification of outputs, Token-DiFR could establish a new standard for quality assurance in AI services, fostering greater trust and reliability in LLM deployments. The technique can be utilized under varied conditions, providing flexibility for auditing even in cases where sampling processes may not be fully synchronized. This advancement is crucial at a time when the inference landscape is often described as chaotic and unregulated.