The Bitter Lesson (www.incompleteideas.net)

In "The Bitter Lesson," Rich Sutton argues that 70 years of AI progress shows a clear pattern: general methods that leverage massive computation—principally search and learning—consistently outperform systems built by encoding human knowledge, especially as compute becomes cheaper (Moore’s Law). He marshals historical examples—computer chess (deep search), Go (search + self-play learning), speech recognition (HMMs then deep learning), and vision (from hand-crafted features to convolutional nets)—to show that domain-specific engineering yields short-term gains but plateaus, while scalable, computation-driven approaches continue to improve and win. The technical takeaway for AI/ML is prescriptive: prioritize meta-methods that scale with computation (search, representation learning, self-play, deep nets) over brittle, hand-crafted priors. Minimal, general inductive biases (e.g., convolution, invariances) that enable scaling are useful, but embedding detailed human concepts of how the world works is likely to inhibit long-term progress. Practically, this implies investing in scalable architectures, learning algorithms, large datasets, and compute-efficient implementations—designing systems that discover complexity rather than attempting to hard-code it. The “bitter” part is that this success often comes at the expense of favored human-centric approaches, but Sutton insists embracing computation-first methods is the cleaner path to continued breakthroughs.