rdmatop: Cross-Provider Htop for RDMA Traffic

Researchers from the UCCL team have introduced rdmatop, a groundbreaking tool designed to monitor RDMA (Remote Direct Memory Access) traffic across a variety of network interface cards (NICs) in real-time. Unlike traditional tools like ibtop that are limited to InfiniBand, rdmatop offers a provider-agnostic view, functioning with devices from NVIDIA, AWS, Broadcom, and AMD. By utilizing RDMA netlink, rdmatop provides immediate visibility of throughput, packet transmission/reception rates, and process activity, making it easier to diagnose performance bottlenecks in multi-node large language model (LLM) training and inference.

This development is significant for the AI/ML community as it addresses a critical gap in monitoring capabilities. RDMA is increasingly used in multi-GPU setups, and being blind to network performance can hinder applications dramatically. rdmatop simplifies the troubleshooting process by delivering valuable insights at a glance, allowing users to quickly identify issues such as fallback to TCP sockets or imbalanced utilization of multiple NICs. The tool is easily deployable in various environments, including Kubernetes and Slurm clusters, providing researchers and practitioners with the ability to optimize their distributed training operations effectively.