Probe-Detected Grokking in Multi-Probe DPO (openinterp.org)

A recent study introduced a significant revelation in the realm of AI/ML involving the 27-billion parameter reasoning model Qwen3.6-27B. Researchers observed a phase-transition-like behavior in the model during training with Multi-Probe Direct Preference Optimization (DPO). Despite using two activation probes—FabricationGuard for factuality and ReasoningGuard for reasoning quality—no observable changes were detected by these original probes. However, a new linear probe trained on the activations during each checkpoint revealed a consistent increase in area under the receiver operating characteristic (AUROC), indicating that the model was indeed learning in a manner that the original probes could not measure. This discrepancy aligns with the concept of "grokking," where the model undergoes a profound reconstruction of its representational capabilities. The findings have substantial implications for the AI/ML community, particularly in how probe-derived training signals are utilized. The study highlights a potential failure mode where models might learn effectively in directions that are structurally invisible to the original evaluation signals. This discovery emphasizes the necessity for fresh probes in assessing model safety and performance accurately. Overall, these insights not only deepen our understanding of training dynamics in large models but also urge a reconsideration of how we evaluate their capabilities, as traditional metrics may obscure critical learning progressions.