The Verifiability Gap in Quantized LLM Supply Chains (arxiv.org)

A recent study highlights a critical issue known as the verifiability gap in the supply chains of quantized large language models (LLMs). While model signing authenticates the full-precision weights, users ultimately operate on quantized artifacts that lack clear reproducibility due to unspecified transformation processes. The research introduces the Independent Verifiability Rate (IVR), which assesses the reproducibility of output files based on disclosed input details, revealing a concerning IVR of 0/198 among the most-downloaded Hugging Face repositories. This absence of documentation for the build environment directly impacts the verifiability of LLMs, meaning that many artifacts cannot be accurately reproduced. The significance of this research lies in its implications for AI/ML practitioners and developers who rely on reproducible results for model reliability and security. The study found that by including critical build parameters, such as floating-point-contraction settings, reproducibility can be dramatically improved, allowing for accurate retesting across different platforms. Moreover, this work underscores the need for better documentation practices within the AI community, as current gaps pose risks not only to model efficiency but also to security, by potentially allowing undetected manipulations. The researchers are sharing their audit data and a framework for others to explore these issues further, paving the way for enhanced trust in LLM deployments.