How to Deploy LLM Locally (blog.lyc8503.net)

This piece is a practical, translated guide for deploying LLMs locally, aimed at bridging the gap between small pre-LLM models that ran everywhere and today’s massive transformer giants (e.g., GPT-3 at 175B parameters, ~350 GB of weights). It explains why running models locally matters—transparency, lower cost and vendor lock-in, censorship/workflow control, and the ability to fine-tune or hack models—and walks through the basic flow: choose hardware, download weights, install an inference runtime, and run. The author highlights real-world local success (an ONNX Wasm CAPTCHA solver running in-browser with >95% accuracy in <0.2s) to show what’s possible at smaller scales. Technically, the guide prioritizes hardware factors (VRAM is decisive; e.g., H100 80GB, A100 40/80GB, RTX 4090 24GB), memory bandwidth (H100 ~2 TB/s, A100 ~1.5 TB/s, 4090 ~1 TB/s), and BF16/FP16 compute (H100 ≈200 TFLOPS). It covers model topology (dense vs MoE; “thinking”/instruct/hybrid behaviors), and practical optimizations: quantization (Q6+/Q7/FP8 typically negligible accuracy loss; Q5 small hit; Q4 noticeable; Q3+ unacceptable), layer offloading to CPU for hybrid inference (useful for MoE), multi-GPU tradeoffs (NVLink vs PCIe), and framework choices. The takeaway: match model size to VRAM and bandwidth, prefer quantized weights and CPU/GPU hybrid strategies when needed, and use community leaderboards (e.g., lmarena.ai) and quantized releases (e.g., Unsloth) to pick deployable models.