Generalist AI Doesn't Scale (www.daemonology.net)

Recent discussions within the tech industry have spotlighted a critical challenge in AI: the rising costs associated with training generalist models. As pointed out by industry insiders, the computational demands of size-N models, characterized by O(N^2) training complexity due to their reliance on a vast number of parameters, threaten to disrupt not only AI development but also the semiconductor industry that underpins it. The inherent scalability limitations of generalist AI highlight the unsustainable nature of this approach, which struggles to generate sufficient training data and ultimately risks becoming prohibitively expensive. A proposed solution advocates for a shift towards specialized AI models, dividing the workload among multiple expert systems rather than relying on a singular generalist model. By training these expert AIs on smaller, focused datasets (N/k), the computational costs can be drastically reduced to O(N^2 / k). While this model may forfeit some creative cross-domain insights, it is more efficient and practical. The concept of a "dispatcher" AI could further enhance this approach by directing queries to the appropriate specialist, making it clear that specialization could lead to a smarter and more scalable AI landscape, mirroring successful strategies seen in other fields.