The Einstein test: what happens when AI tries to rediscover relativity? (www.nature.com)

Recent discussions in the AI community have centered around the "Einstein test," proposed by Google DeepMind co-founder Demis Hassabis, which examines whether large language models (LLMs) can reproduce groundbreaking theories like general relativity using only historical data up to 1911. This test aims to evaluate the potential for artificial general intelligence (AGI) by assessing if AI can achieve creative breakthroughs similar to human scientists. The initial results from various teams experimenting with "vintage" models have highlighted current limitations in AI, particularly its reliance on inductive reasoning rather than the abductive reasoning required for paradigm-shifting discoveries. Several researchers, including Michael Hla and Nick Levine, have attempted to train LLMs on pre-1900 and pre-1930 data respectively, finding that while these models occasionally exhibit flashes of insight, they often lack true understanding, failing to make coherent connections like those of human theorists. The challenges include filtering historical data accurately and the models' propensity to generate erroneous theories that complicate the distinction between viable ideas and misinterpretations. Despite these hurdles, there is optimism that future models could potentially aid in predicting scientific discoveries, underscoring the ongoing exploration into the role of AI in scientific innovation and reasoning.