One token is enough: fingerprinting LLMs from one token output distributions (arxiv.org)

A recent study introduces a groundbreaking method for fingerprinting large language models (LLMs) using the output distribution of just a single token. This method addresses a crucial challenge in the AI/ML community: verifying that the model being accessed through opaque API services is indeed the correct one. Traditional identification techniques often demand lengthy text samples or model owner cooperation, whereas this new approach utilizes trivial one-word prompts, allowing researchers to gather empirical distributions of responses quickly and at minimal cost. The researchers assessed 165 models via a commercial aggregator and found that the resulting token output distributions were highly consistent and distinguishable between models, achieving a 59.5% accuracy in associating models with their families. Moreover, the proposed biometric-style verification maintained a low error rate, showcasing the viability of this technique for auditing AI models. By revealing significant discrepancies in model outputs, such as a proprietary model mimicking an open-weight one, this method enhances model transparency and integrity in an increasingly opaque AI ecosystem. The open availability of the protocol and data sets ensures the potential for broader application and validation within the AI/ML community.