GPU probably isn't helping your retrieval system (strake.dev)

Recent benchmarking of a small embedding model has revealed that using GPUs may not be beneficial for certain retrieval systems, particularly when working with small models and single-query retrievals. The study compared a 33 million parameter model across various hardware setups, including a Mac M2 Pro with Metal, an Intel 13700K CPU, and GPUs like the RTX 2080 Ti and RX 6600 XT. Surprisingly, the results showed that the best performance came from Metal on the M2 Pro due to its unified memory, which eliminates host-to-device transfer overhead. In contrast, while CUDA on the RTX 2080 Ti provided slight advantages in latency (about 20% faster for p50), it still performed modestly at best, leading to a situation where the GPU acceleration didn't significantly outpace the CPU for small workloads. This finding highlights a critical understanding within the AI/ML community: small models running single queries do not benefit from GPU acceleration as typically suggested. The overhead of dispatching to the GPU overshadows its computational prowess, making CPU alternatives more efficient for certain tasks. As a result, researchers and developers are advised to evaluate their specific workload requirements before assuming that GPUs will enhance performance, especially in cases involving limited model sizes and low batch counts. This nuanced perspective challenges conventional wisdom and emphasizes the importance of benchmarking in aligning hardware capabilities with application needs.