Building AGI Using Language Models (2020) (bmk.sh)

This 2020 thought experiment argues that while GPT-3 isn’t AGI, large autoregressive language models are uniquely positioned to become “world models” because natural language densely encodes human knowledge and behavior. The training objective—predicting the next token—first drives surface skills (grammar, coherence) and then, as loss approaches the Shannon entropy of text, pushes models toward deeper semantic, causal, and behavioral knowledge. The author’s key claim: by scaling models (1T, 10T, 100T parameters) we may empirically drive down loss enough to embed robust world models without bespoke architectures. The paper lays out a concrete blueprint for turning such a world model into an agent: use the LM to enumerate possible actions, estimate rewards for action sequences, and run Monte Carlo Tree Search over rollouts generated by the LM to compute state-action values; implement hierarchical decomposition of high-level actions into executable steps; represent and prune agent state as compressed natural-language summaries; and convert multimodal inputs/outputs into text (e.g., iGPT for images, HTML for webpages) and into executable code/commands. Major caveats include the central, unresolved assumption that scale alone will yield sufficient world-modeling, current empirical shortfalls in GPT-3, and safety/oracle concerns, but the proposal supplies a testable, scalable path to a proto-AGI that the AI community can evaluate as larger models become available.