Checkpoint-engine: A middleware to update model weights in LLM inference engines (github.com)

Checkpoint-engine is a new middleware designed to efficiently update model weights during large language model (LLM) inference, a crucial process for reinforcement learning and continual model improvement. The system supports in-place weight updates across thousands of GPUs with remarkable speed—for example, updating a 1-trillion-parameter Kimi-K2 model across a large GPU cluster takes roughly 20 seconds. This capability addresses a significant bottleneck in deploying and fine-tuning massive LLMs, enabling faster iteration and deployment in production environments. Technically, checkpoint-engine features two main weight update strategies: Broadcast and Peer-to-Peer (P2P). The Broadcast method excels when many inference instances update weights synchronously, leveraging a highly optimized three-stage pipeline (H2D transfer, inter-worker broadcast via CUDA IPC, and selective reload). P2P is designed for dynamic scaling scenarios where new inference instances join on-the-fly without disrupting existing workloads, transferring weights from CPU-to-GPU across nodes via RDMA with mooncake-transfer-engine support. The system smartly manages GPU memory and communication pipelines to maximize throughput, falling back to serial execution if needed. Tested extensively on diverse models and hardware setups (including BF16 and FP8 precision formats), checkpoint-engine integrates tightly with vLLM and offers easy installation and deployment, making it a practical tool for large-scale distributed inference. Its modular architecture and open design hint at broad applicability beyond vLLM, promising faster, more flexible LLM weight management in production AI/ML systems.