Scaling "E-Waste": Benchmarking LLMs on Multiple Cheap GPUs (esologic.com)

🤖 AI Summary
A recent benchmarking study has demonstrated that utilizing multiple older GPUs, such as Tesla K80s, P100s, and V100s, can effectively scale the performance of large language models (LLMs), making high-capacity models accessible for local use. The research highlights that despite the cheapness of these decommissioned GPUs—often seen as undesirable due to their high operating temperatures—they can still deliver significant outputs. With an updated benchmarking tool featuring over 130 tests, throughput for various models like Qwen and Gemma have been tested under consistent hardware conditions, revealing that configurations with multiple GPUs, particularly three, allow users to experiment with LLMs as large as 72 billion parameters. This development is significant for the AI/ML community as it offers a cost-effective pathway for individuals and smaller teams to utilize advanced LLMs without relying on expensive, high-end hardware. The findings underscore that while adding more GPUs doesn't necessarily increase throughput, it can allow for the handling of larger models by optimizing GPU memory usage, with efficient power tracking added to the analysis. With models like Qwen 3.6 achieving impressive generation speeds of over 70 tokens per second on these setups, the approach broadens accessibility for deploying complex AI models and promotes a DIY ethos in AI research and applications.
Loading comments...
loading comments...