Running a 35B LLM at 128K Context, Full Speed, on €870 of Used Hardware (medium.com)

A recent benchmark highlights the capability of running a 35 billion parameter (35B) large language model (LLM) at full speed with 128K context on a used GPU setup costing only €870, demonstrating significant advancements in cost-effective AI hardware configurations. The author tested two builds: one using a combination of an RTX 4070 and an older RTX 2070 Super, and another incorporating a newer RTX 5060 Ti. The results indicate that while increasing VRAM has its advantages, it doesn't always translate to speed improvements for smaller models; however, for models exceeding 20 GB, the additional VRAM significantly enhances performance by reducing reliance on CPU memory. This experiment is pivotal for the AI/ML community as it reveals how hardware changes can optimize model performance without necessitating expensive cloud solutions. It underscores the importance of model architecture in determining throughput and highlights that careful placement of model weights across GPUs can yield drastic performance gains. The findings suggest that understanding VRAM utilization and memory bandwidth become crucial for practitioners looking to optimize their deployments, especially with the rise of models requiring extensive context lengths in practical applications.