What GPT-OSS Leaks About OpenAI's Training Data (fi-le.net)

OpenAI’s open-weights release GPT-oss enabled an analysis showing that model parameters leak nontrivial signals about training data and the training stack. By inspecting the o200k tokenizer’s embedding matrix and plotting L2 norms of each token embedding, researchers found an anomalous tail of high-norm tokens. English high-norm tokens clustered around reasoning and code phrases (suggesting late-stage coding/RL fine-tuning), while many non‑ASCII high-norm tokens corresponded to spammy, gambling and adult-site phrases in Chinese and other languages. Querying GPT-oss and GPT-5 for translations of these “glitch” tokens—a form of membership inference—confirmed that many explicit strings were seen during training. A Spearman correlation (ρ ≈ 0.448) between token recognizability and GitHub hit counts further implicates public code/spam repositories as likely sources for some tokens. This work matters because it shows concrete, automated ways an open-weight model can reveal dataset composition, training steps (e.g., effective gradient updates/weight decay signals), and exposure to sensitive content. Technical takeaways: embedding-norm analysis can surface candidate membership tokens; model completions can validate membership; and open model releases combined with large token vocabularies risk leaking private or undesirable provenance. For ML practitioners and auditors, the findings argue for tighter dataset provenance, careful tokenizer construction, and privacy-aware release practices to mitigate unintended disclosure and moderation risks.