Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't (vettedconsumer.com)

A recent analysis highlights the surprising efficiency of mini PCs equipped with unified memory compared to powerful GPUs when running large language models (LLMs). For instance, while a high-end NVIDIA RTX 5090 with 32GB of VRAM cannot run a 70-billion-parameter model due to memory limitations, an AMD Ryzen AI Max+ 395 "Strix Halo" mini PC can easily manage it using 128GB of shared LPDDR5X memory. This difference stems from the concept of unified memory, which eliminates the separate VRAM barrier, allowing almost all available memory to be utilized for model processing. This shift has significant implications for the AI/ML community, particularly in terms of capacity versus speed. The analysis outlines that memory capacity determines whether a model can load, while memory bandwidth influences generation speed. Consequently, while the mini PC can effectively manage larger models at a lower cost, it sacrifices speed for capacity, delivering approximately 4 to 6 tokens per second for dense models. This insight suggests a new strategy for deployment: mini PCs are ideal for users who require large models without real-time processing demands, while dedicated GPUs remain the preferred choice for applications requiring both speed and efficiency.