Why don't machine learning research agents overfit? (www.amazon.science)

Recent research has revealed that machine learning (ML) agents avoid overfitting by learning compressible models of data, which limits their capacity for memorization. Contrary to traditional beliefs that iterative evaluations against the same validation sets should lead to overfitting, experiments show that ML research agents can utilize highly compressible strategies—sufficiently concise that essential information can be distilled into as few as 16 tokens. This encapsulation process ensures that the strategy reflects genuine data structure rather than rote memorization, thereby maintaining the model's generalization capabilities. The significance of these findings extends to understanding the behavior of large language models (LLMs), which are effective as compression decoders due to their extensive knowledge of the ML landscape. The ability to sharply compress information highlights that successful ML strategies can be communicated in succinct forms, making it easier to replicate them without losing performance. This new perspective not only addresses a long-standing puzzle in ML—how research continuously yields improvements without succumbing to overfitting—but also reinforces the concept that the essence of machine learning is fundamentally about generalization rather than simple memorization.