Explorations of RDMA in LLM Systems (le.qun.ch)

A small team built and open-sourced an RDMA communication library optimized for LLM systems that embraces Unordered Reliable Datagram (URD) semantics and runs on both AWS EFA and NVIDIA ConnectX NICs. They applied it to three real problems—disaggregated KvCache transfer, RL post-training parameter updates, and MoE routing—and report that their MoE kernel decodes slightly faster than DeepEP on ConnectX‑7 and delivers the first actually-usable MoE performance on EFA. The work addresses production pain points with collective APIs (rigid participant worlds, blocking reconfiguration, enforced ordering, and worst-case sizing) by trading strict ordering for a simpler, more scalable model suited to applications that only need eventual delivery. Technically, the library leans on hardware reliability but embraces unordered delivery, exposes two-sided SEND/RECV plus one-sided WRITE_IMM with 32‑bit immediates, and uses an ImmCounter model (receivers count immediates to detect completion). It supports multi‑NIC aggregation, host‑proxy GPU RDMA (to integrate with CUDA Graphs), and optimized multi‑page/multi‑receiver APIs. Portability required reimplementing a verbs backend for ConnectX and careful SRD-vs-RC handling; performance tuning included parallelism, NVLink/RDMA sync, PTX and allocation optimizations. A notable caveat: DeepEP still wins on some prefill paths by sending fewer RDMA bytes via clever NVLink partial-reductions. Paper, code, and blog links are available for full details.