Show HN: Docker Model Runner Integrates vLLM for High-Throughput Inference (github.com)

Docker Model Runner (DMR) is an open-source tool from Docker that streamlines pulling, running and serving local LLMs and other AI models via Docker or any OCI-compliant registry. Aimed at developers, it bundles a Go backend (model-runner) and a CLI (model-cli) that can run locally or inside Docker Desktop/Engine, enabling quick model lifecycle commands (list, pull, run, delete) and chat-style completions. Significance: it reduces friction for experimenting with local models (including llama.cpp-based servers), offers a Docker-native UX (optional Docker CLI plugin), and exposes a simple REST API so existing tooling and CI/CD can integrate easily. Technically, model-runner is written in Go (requires Go 1.24+), provides a Makefile and Docker image that includes a llama.cpp server binary, and supports configurable server version/variant via LLAMA_SERVER_VERSION and LLAMA_SERVER_VARIANT. It supports persistent model storage by mounting a host models directory, TCP access (examples use ports 13434 and 8080), and exposes an OpenAI-style chat completions endpoint (/engines/llama.cpp/v1/chat/completions) plus Prometheus-format metrics (/metrics). GPU acceleration is possible but requires CGO build deps (Xcode tools, gcc, MinGW/Visual Studio). There’s experimental Helm/Kubernetes support and monitoring hooks, making DMR useful both for local dev and as a building block for on-prem or edge model deployments.