Outrageously Small NNs: Emergent Reasoning at 6,616 Tok/s on One Intel AMX Core (gregdiamos.com)

A new study has unveiled the potential of remarkably small neural networks (NNs) running at a performance rate of 6,616 tokens per second (tok/s) on a single Intel Xeon AMX core, challenging the conventional reliance on large GPU clusters for model training and inference. The objective was to process a massive dataset efficiently and develop a model capable of handling extraction and classification tasks economically. Utilizing an architecture designed around the advanced capabilities of AMX (Advanced Matrix Extensions), the results demonstrated that operating on a minimally sized model can yield impressive performance, underscoring the potential of using CPUs to run straightforward models at high speeds. This development is significant for the AI/ML community as it redefines the landscape for model training and execution, suggesting that smaller models could be leveraged effectively in environments constrained by hardware budgets. The study also indicates that when models are optimized for specific tasks and data curation becomes more sophisticated, even smaller neural networks can achieve emergent reasoning capabilities that were previously thought to be possible only with larger architectures. Key findings include the necessity of innovative routing methods in the model design to avoid bottlenecks in processing and the observation that with the right configuration, the training phase can continuously improve without the risk of saturating performance, all at a remarkably low parameter count.