Scaling your models to Zero with Fly.io (xeiaso.net)

Fly.io published a practical walkthrough for running a private Ollama LLM server on cloud GPUs that automatically “scales to zero” when idle. The guide shows how to attach your machine to Fly’s private network with WireGuard (fly wireguard create → import config → verify api.internal), launch an app (fly launch), allocate a private IPv6 flycast address (fly ips allocate-v6 --private), and configure the VM for GPUs and persistent storage. Key config items include selecting VM sizes (a100-40gb, a100-80gb, or l40s — A100s concentrated in ORD, L40s in SEA), adding a build section to pull the Ollama Docker image, a 100 GB mount for downloaded models, and an HTTP service block that enables automatic stopping with a minimum of zero. Deploy with fly deploy and point your Ollama client at your-app.flycast (set OLLAMA_HOST) to run heavy models like Nous Hermes Mixtral in the cloud instead of locally. This is significant because it makes private, cost-effective LLM hosting accessible to developers: models can run on powerful GPUs on-demand, be woken by other internal apps, and shut down automatically to avoid idle GPU costs. Important trade-offs to note include model load times on cold starts, private-network-only access unless exposed, and region-driven GPU availability — all practical considerations when integrating cloud-hosted LLMs into development and production workflows.