Use fewer threads for CPU inference (www.nobodywho.ai)

A recent update from NobodyWho highlights the importance of optimizing thread usage for CPU inference, particularly regarding the differences between physical and logical cores in modern CPUs. The conventional approach of spawning threads equivalent to the number of logical cores can lead to inefficiencies, especially when hyperthreading is involved, as multiple threads often compete for shared resources on a single physical core. By contrasting performance benchmarks on different architectures, including Ryzen and MacBook CPUs, the findings reveal that using only as many threads as there are physical cores significantly enhances inference speed. This insight is crucial for the AI/ML community, as it can lead to more efficient execution of deep learning models and applications, especially in environments where different core types exist. The NobodyWho framework automatically identifies the count of fast physical cores, ensuring optimal thread allocation even on heterogeneous processors. This optimization reduces contention and increases performance, enabling better utilization of computational resources. With this refined approach, users can attain faster processing for AI inference tasks, especially when competing workloads are present, showcasing the continuous evolution in optimizing AI model inference on diverse hardware setups.