DumpsterCluster: From Dumpster Diving to Serving Llama-70B on $60 GPUs (arxiv.org)

A recent study titled "DumpsterCluster" explores the potential of repurposing retired GPUs from AI datacenters to create a cost-effective, sustainable computing cluster capable of serving large language models (LLMs) like LLaMA-70B. The researchers constructed a 128-GPU "DumpsterCluster" using second-hand components for approximately $22,000, a stark contrast to the $600,000 price tag of modern systems. By implementing pipeline-parallel optimizations, the DumpsterCluster matched competitive throughput levels, indicating a viable approach to extend AI capabilities affordably. However, the deployment unveiled significant environmental considerations. Older GPUs, while cheaper, have higher energy consumption, making their operation economically viable only in regions with low electricity costs. The study highlights that under typical carbon intensity conditions, second-hand systems can result in significantly higher carbon emissions per token compared to contemporary hardware—up to 40 times more for larger models—underscoring the importance of coupling GPU reuse with sustainable energy sources. These findings emphasize that while repurposing GPUs can enhance AI accessibility, it must be carefully managed to align with environmental goals.