Deploy LLMs at the edge: Alki OSS toolchain (github.com)

Alki is an open-source toolchain for deploying and managing LLMs at the edge that automates Hugging Face → GGUF conversion, quantization, packaging and fleet orchestration. With a single CLI flow (validate, pack, image, publish), Alki converts HF models to GGUF (direct Q8_0 conversion supported) or accepts pre-converted GGUFs, applies quantization, benchmarks throughput/memory, and emits production-ready bundles (containers, systemd units, k8s manifests, SBOMs). Bundles run on the llama.cpp runtime (broad CPU/GPU support) and expose an OpenAI-compatible API via llama-server, enabling local inference, monitoring, and A/B rollout across hundreds of edge devices without cloud dependency. For engineers, the technical win is turnkey edge readiness: quantization profiles (Q8_0, with Q4_K_M and Q5_K_M in development) shrink model sizes dramatically (Q4_K_M ~75% smaller) while keeping toolchain outputs portable across Docker, k3s, and systemd. Conversion requires optional PyTorch dependencies (~2GB) and pulls llama.cpp conversion tools (~150MB) on first use. Alki also provides performance benchmarking (tokens/sec, memory), SBOMs, manifests, and a bundle registry for efficient updates. Roadmap items include advanced quantizers, hardware optimization profiles, multi-runtime backends (Ollama, MLC-LLM, ONNX) and multi-modal support, making Alki a pragmatic choice for productionizing edge LLMs today and scaling toward heterogeneous runtimes.