The GPU Observability Gap: Why We Need eBPF on GPU Devices (eunomia.dev)

Researchers demonstrated a way to close the “GPU observability gap” by extending eBPF into GPU devices: bpftime’s CUDA/SYCL attachment lets eBPF programs run inside GPU kernels on NVIDIA and AMD hardware. That means the same in-kernel, dynamically programmable observability and control that transformed CPU-side tracing can now be applied to GPU workloads, enabling real-time profiling, debugging, and runtime extensions of GPU code without changing source or switching to synchronous execution. This matters because modern GPUs run thousands of threads in SIMT warps across complex memory hierarchies and asynchronous streams—behavior that CPU-side hooks (LD_PRELOAD, driver syscall tracing) only see as opaque API timings, while vendor tools (CUPTI, Nsight, ROCProfiler) are siloed, heavyweight, and hard to correlate with Linux kernel and userspace events. Running eBPF on the device promises fine-grained, warp- and memory-access visibility, dynamic instrumentation of kernels, and time-aligned correlation with host/thread/syscall events—critical for diagnosing warp divergence, memory stalls, SM underutilization, PCIe or RDMA interference, and intermittent tail-latency spikes in production LLM and HPC workloads. The approach preserves production-friendly programmability and control-plane integration, reducing the need for synchronous debugging or costly rebuilds, though it raises new considerations around instrumentation overhead and vendor integration.