We're Learning Backwards: LLMs build intelligence in reverse (pleasedontcite.me)

Professor Alyosha Efros has sparked debate in the AI/ML community with a recent lecture revealing that modern large language models (LLMs) like GPT have effectively "learned backwards." By contrasting early scene completion techniques from 2007, which relied solely on massive datasets without sophisticated models, Efros illustrates a critical point: the performance of LLMs largely stems from scaled computation and data rather than from advanced algorithms or a true understanding of the underlying world. As models grow more capable, they still demonstrate significant blind spots, leading to "spiky intelligence," where they excel in some areas yet falter in basic reasoning tasks. The implications are profound for the pursuit of artificial general intelligence (AGI). Current LLMs possess significant "crystallized intelligence" due to their vast knowledge bases, but they struggle with "fluid intelligence," which is essential for solving novel problems without prior experience. Recent benchmarks like ARC-AGI-3 exemplify this gap; models rooted in exploration and hypothesis testing outperformed LLMs, which rely on prior knowledge. To achieve true AGI, the article suggests that new architectures that facilitate dynamic learning through interaction may be necessary, as simply scaling LLMs may not lead to the desired flexibility and adaptability seen in human intelligence.