Text Is Unnatural, and LLMs Are Built on It (www.innerkore.com)

Recent insights into the limitations of large language models (LLMs) reveal that they are fundamentally built on text, which is described as a lossy, symbolic compression of reality. This design means LLMs acquire knowledge through the descriptions of experiences, lacking direct sensory perception. Consequently, while they excel in articulated knowledge, they struggle with tacit, embodied understanding—reflecting Moravec's paradox, where tasks easy for humans are challenging for machines. The article argues that a shift towards multimodal learning—which incorporates video, audio, and other sensory data—is essential for advancing general intelligence in AI. Multimodal approaches provide a richer, denser source of information, grounding language in real-world context. Although text remains a crucial interface for interacting with AI, future models must prioritize learning from raw sensory experiences, enhancing their ability to connect words with tangible realities. This could lead to more sophisticated AI systems capable of understanding and interacting with the world in a human-like manner.